#1. Why install Kubernetes on AWS EC2?

돈It would be great to save money by filling up EC2 with docker containers and running them efficiently. Even better if using efficient cores like Graviton 2~3!

#2. Let’s installing Kubernetes

To avoid the hassle of installation, I chose the Ubuntu OS and simply pasted commands to install it. Here’s the official website link to install kubeadm, kubelet, kubectl:

sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# If the folder `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Attempt initialization:

sudo kubeadm init

It doesn’t work. As mentioned on the official site, a CRI needs to be installed for any progress. Let’s install containerd CRI.

[init] Using Kubernetes version: v1.29.2
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR IsPrivilegedUser]: user is not running as root
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

#3. Understanding Concepts

Let’s host Nginx:

kubectl create deployment nginx --image nginx

Output:

error: failed to create deployment: Post "http://localhost:8080/apis/apps/v1/namespaces/default/deployments?fieldManager=kubectl-create&fieldValidation=Strict": dial tcp 127.0.0.1:8080: connect: connection refused

Environment settings like kubeadm are not configured, hence it didn’t work. This is just to confirm it doesn’t work.
Kubernetes is still unfamiliar to me…
I’ll have to dig into the official site again. Set up the control environment with Kubeadm.

Plan to use t4g.micro arm aarch for free until the end of 2024, but it has only 1GB RAM, which is not enough. Linux memory swap isn't supported, but versions after 1.28 group v2 might work. If not, I'll try again.

1. Setting up Kubernetes

Kubernetes (k8s) starts by setting up the Control panel and work node computers to build an integrated computing environment. In the built environment, executions like pods are managed using the API on the control panel.

For high availability settings, load-balance and auto-failover must be set on the control panel. Alternatively, using just one control panel without such settings or using AWS or Google’s EKS is also viable… To save money using only one EC2, k8s ingress (reverse proxy, API Gateway, whatever) is a luxury.

sudo kubeadm init --v=5

It didn’t work. Try again.

[preflight] Some fatal errors occurred:
	[ERROR Mem]: the system RAM (917 MB) is less than the minimum 1700 MB
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

There’s a way to use only kubelet, but what does that mean?
Kubeadm settings, it seems possible to run without a Kubernetes control panel,
I need a conceptual re-load.

2. Understanding Kubernetes Components

While installing kubeadm,
What are cgroups…

In Kubernetes, cgroups (Control Groups) are a Linux kernel feature used to limit and manage resources for containers. Cgroups control resources like CPU, memory, disk I/O, and network bandwidth that a container can use. 

Through cgroups, Kubernetes manages the amount of resources allocated to each container, limiting or adjusting resource usage to maintain cluster stability and performance.

For example, Kubernetes allows users to set CPU and memory resource limits for each pod or container. These limits are implemented through cgroups, which apply and manage these limits to ensure containers do not exceed them.

Thus, cgroups play a crucial role in limiting and managing resource use in containers within Kubernetes. This allows each container to use the necessary resources without impacting other containers and operate stably.

If not using the Kubernetes control panel, is there no need to install a cgroup driver? Why isn’t there talk of cgroups, only CNI, CRI installation, and kubelet handling…

The Korean translation might be why; it just says set an appropriate cgroup driver if you want to control the container runtime. So installing CRI and setting it up well means you can use kubelet.

Configuring a cgroup driver

Both the container runtime and the kubelet have a property called "cgroup driver", which is important for the management of cgroups on Linux machines.

Warning:
Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail.

See Configuring a cgroup driver for more details.

Set the cgroup well, or kubelet won’t run…

1) What is CNI…

Kubernetes CNI (Container Network Interface) is a standard interface for managing communication between containers and between containers and the external network within a Kubernetes cluster.

Kubernetes clusters are distributed systems where containers run on multiple hosts (nodes). These containers need to communicate with each other and the outside world. CNI makes this possible.

CNI manages and sets up the network in Kubernetes through network plugins. Each node must have these CNI plugins installed. These plugins allow the use of various networking solutions and provide different networking functions and features depending on the type of plugin.

Typically, CNI plugins perform the following tasks:

Network interface setup: Creates and configures a virtual network interface for containers.
IP address assignment: Assigns IP addresses to containers and manages network configurations.
Routing and forwarding setup: Configures routing and forwarding for communication between containers and with the outside.
Security and policy application: Applies network security and policies to limit or protect communication between containers.
Through CNI, Kubernetes clusters can flexibly configure the network using various networking solutions. For example, various CNI plugins like Flannel, Calico, Weave, etc., are available, and each can be chosen based on its characteristics and advantages/disadvantages.

Since there’s no plan to use the Kubernetes control panel, CNI might need to be installed directly.

2) What is CRI…

CRI (Container Runtime Interface) is a standardized interface between container orchestration systems like Kubernetes and container runtimes. CRI defines the API used by container orchestration systems to manage and execute containers.

Traditionally, Kubernetes directly depended on specific container runtimes like Docker to manage containers. However, this made it difficult for Kubernetes to integrate with various container runtimes and lacked flexibility when switching to different container runtimes.

With the introduction of CRI, Kubernetes standardized communication with container runtimes, reducing the complexity of changing or replacing runtimes. In other words, CRI provides a standard interface between container runtimes and Kubernetes, loosening the coupling between orchestration systems and container runtimes.

This allows Kubernetes to support various container runtimes, and developers and operators can choose or change container runtimes. Various container runtimes like Containerd, CRI-O, Docker implement CRI and can be integrated with Kubernetes.

In summary, CRI is a standardized interface between Kubernetes and container runtimes, playing a crucial role in integrating various runtimes and providing flexibility for Kubernetes.

Since there’s no plan to use the Kubernetes control panel, CRI might need to be installed directly.

3) Relationship between Docker and Containerd – Good reference material

Containerd is an open-source container runtime used to manage container execution. It is one of the core features of Docker, developed as a project separated from Docker. Containerd is widely used in container orchestration systems like Kubernetes to manage container execution.

Thus, the statement "use containerd to run that" means that when executing containers through Kubernetes or other container orchestration systems, containerd is used to manage and execute the containers.

Then, even if images are fetched from Docker, containerd can be used to execute the containers. This is because Docker is used to fetch and store images, but containerd can be used to manage and execute the containers.

In short, even if Docker images are fetched, containerd can be used to execute the containers when run through Kubernetes or other container orchestration systems. This provides the flexibility that the tools used to manage images and containers can be separated from the runtime used to execute the containers.

Ultimately, whether you use Containerd or not, whether you use Docker images or not, it doesn’t matter….

Does the talk of installing CNI, CRI mean that cgroups must also be installed?
Even if you set up CRI, you have to limit the execution of pods on the worker, right?

Ah, if you don’t want to control the resources of the work, you don’t need to install the cgroup driver?

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
...
cgroupDriver: systemd

From 1.28 onwards, the default cgroupDriver is systemd, which is the cgroup driver for the container runtime,

• containerd
• CRI-O

These two can be examples.

Just install that, and the cgroup-driver is configured…

3. CGROUP settings

1) Pre-setup

Forwarding IPv4 and letting iptables see bridged traffic

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

Verify that the br_netfilter, overlay modules are loaded by running the following commands:

lsmod | grep br_netfilter
lsmod | grep overlay

Verify that target – 확인할 대상 목록

• net.bridge.bridge-nf-call-iptables, 
• net.bridge.bridge-nf-call-ip6tables,
• net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables, 
• net.bridge.bridge-nf-call-ip6tables,
• net.ipv4.ip_forward 

system variables are set to 1 in your sysctlconfig by running the following command:

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

2) Install containerd – 링크

I’ll use the ubuntu apt, dnf installation options. It says the CNI plugin is not installed. Then, if installed with the official binary, it’s installed….

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# docker에서 깔라고 알려주는거라 다 깔라는데.... 난.... compose는 깔지 않을래....
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin -y

3) default containerd settings

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

Looks like the default config is applied. Let’s set it and check the content seen on the k8s official site.

disabled_plugins = []
...

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

The result of performing containerd config default | sudo tee /etc/containerd/config.toml is In the settings of /etc/containerd/config.toml, the SystemdCgroup setting is false.
It seems it should be changed to true to see if it works.

Related issues occur during the kubeadm process due to problems with CRI.
As recorded below, follow the bug content written.
- (Even if SystemdCgroup is not set to true, kubeadm init works, other options are important. bug solution link)

The systemd cgroup driver is recommended if you use cgroup v2.
If you use cgroup v2, it is recommended to use systemd.
But then what about me?

According to the reference link, it is also recommended from ubuntu 22.04+.

stat -fc %T /sys/fs/cgroup/

When I checked, it was cgroup2fs….. I have to do it….

Note:
If you installed containerd from a package (for example, RPM or .deb), you may find that the CRI integration plugin is disabled by default.

You need CRI support enabled to use containerd with Kubernetes. Make sure that cri is not included in thedisabled_plugins list within /etc/containerd/config.toml; if you made changes to that file, also restart containerd.

If you experience container crash loops after the initial cluster installation or after installing a CNI, the containerd configuration provided with the package might contain incompatible configuration parameters. Consider resetting the containerd configuration with containerd config default > /etc/containerd/config.toml as specified in getting-started.md and then set the configuration parameters specified above accordingly.

disabled_plugins had nothing, but it didn’t work. Change it and something runs…

4) Run the containerd service

sudo systemctl daemon-reload
# sudo systemctl start containerd
# start로 했다가 변경 사항이 반영되지 않아서, 안되더라 꼭 restart 를 하라
sudo systemctl stop containerd
sudo systemctl restart containerd
sudo systemctl enable --now containerd

5) Set up CNI

Why suddenly write to set up more CNI…
Directly called the running service of finally.

(1) Kubelet standalone service setup

It’s an example found elsewhere, but I tested it before registering the service.
It didn’t match my environment, and the options were said to be wrong.

/usr/bin/kubelet --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --pod-manifest-path=/etc/kubernetes/manifests

This should run and be maintained as a service for kubelet standalone to work…
It’s default in the kubelet option document, but do I need to declare it….

Moreover, it is said that it does not match my environment and the options are wrong.

--container-runtime-endpoint string     Default: "unix:///run/containerd/containerd.sock"

What should I do for kubelet standalone? I need to look up official documents or examples.

/etc/kubernetes/admin.conf is used for initial settings.
This seems to be created only if Kubeadm init succeeds…. Ah, there’s not much to check quickly….
For reference, use multiple EC2s to find out the actual operation trigger. The work is slow….

sudo kubeadm init

[init] Using Kubernetes version: v1.29.2
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR CRI]: container runtime is not running: output: time="2024-02-26T00:18:17Z" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/run/containerd/containerd.sock: connect: no such file or directory\""
, error: exit status 1
	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
	[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

First install CRI, then do Kubeadm init…
After reinstalling everything as directed for containerd and starting all services, the following occurs:

[init] Using Kubernetes version: v1.29.2
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR CRI]: container runtime is not running: output: time="2024-02-26T00:45:43Z" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

It seems connected, but there’s a problem… Maybe it needs to be clarified.

The shaded part below was an attempt, but it turned out to be meaningless. The issue during kubeadm initialization was that the option to use CRI in the containerd settings was crucial.

wget https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-arm64-v1.4.0.tgz
mkdir -p /opt/cni/bin
tar Cxzvf /opt/cni/bin cni-plugins-linux-arm64-v1.4.0.tgz

curl -L -o /opt/cni/bin/calico https://github.com/projectcalico/cni-plugin/releases/download/v3.14.0/calico-arm64
chmod 755 /opt/cni/bin/calico
curl -L -o /opt/cni/bin/calico-ipam https://github.com/projectcalico/cni-plugin/releases/download/v3.14.0/calico-ipam-arm64
chmod 755 /opt/cni/bin/calico-ipam

6) Solving CNI Installation Bug

In /etc/containerd/config.toml, the option
disabled_plugins = [] needs to be changed.

enabled_plugins = ["cri"]

That should do it. Now let’s just go back and run with only containerd and kubelet. Using 2G RAM isn’t free after all.

sudo systemctl daemon-reload
# sudo systemctl start containerd
# start로 했다가 변경 사항이 반영되지 않아서, 안되더라 꼭 restart 를 하라
sudo systemctl stop containerd
sudo systemctl restart containerd
sudo systemctl enable --now containerd

4. Run the kubelet service, deploy a test pod, and conduct tests.

1) 테스트 도구 설치

It is said that crictl is the most useful tool for debugging tests. Let’s download it using wget.

VERSION="v1.29.0" # check latest version in /releases page
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-arm64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-arm64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz

ubuntu@ip-10-0-145-244:~$ crictl
NAME:
   crictl - client for CRI

USAGE:
   crictl [global options] command [command options] [arguments...]

VERSION:
   v1.29.0

Setting up crictl

cat << "EOF" | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

2) Deploy a test pod

cat << EOF | sudo tee /etc/kubernetes/manifests/web.yaml
apiVersion: v1
kind: Pod
metadata:
  name: static-web
spec:
  containers:
  - name: web
    image: nginx
    ports:
    - name: web
      containerPort: 80
      hostPort: 8080
      protocol: TCP
EOF

3) Run kubelet

sudo /usr/bin/kubelet --container-runtime-endpoint=unix:///run/containerd/containerd.sock --pod-manifest-path=/etc/kubernetes/manifests

E0226 07:03:44.971905    6463 pod_workers.go:1298] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized" pod="default/static-web-ip-10-0-136-250" podUID="103f5f5baab872f8bf4f0d41fb78b05a"

4) The CNI plugin was not initialized. Have we finally reached the CNI plugin…

It turns out that the installation via cni-plugins-linux-arm64-v1.4.0.tgz had already been completed.
The plugins are well installed, and a network environment setup is required for the service space
Kubernetes will use, similar to setting up a VPC with CIDR.

– containerd CNI setup – https://github.com/containerd/containerd/blob/main/script/setup/install-cni

wget https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-arm64-v1.4.0.tgz
mkdir -p /opt/cni/bin
tar Cxzvf /opt/cni/bin cni-plugins-linux-arm64-v1.4.0.tgz

Look at the executions installed.

ubuntu@ip-10-0-136-250:~$ /opt/cni/bin/b
bandwidth  bridge
ubuntu@ip-10-0-136-250:~$ /opt/cni/bin/b
bandwidth  bridge
ubuntu@ip-10-0-136-250:~$ /opt/cni/bin/bandwidth
CNI bandwidth plugin v1.3.0
CNI protocol versions supported: 0.3.0, 0.3.1, 0.4.0, 1.0.0
ubuntu@ip-10-0-136-250:~$

It seems we might not need to install the plugin.

Let’s just install the 10-containerd-net.conflist…

sudo cat << EOF | sudo tee /etc/cni/net.d/10-containerd-net.conflist
{
  "cniVersion": "1.0.0",
  "name": "containerd-net",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "promiscMode": true,
      "ipam": {
        "type": "host-local",
        "ranges": [
          [{
            "subnet": "10.88.0.0/16"
          }],
          [{
            "subnet": "2001:4860:4860::/64"
          }]
        ],
        "routes": [
          { "dst": "0.0.0.0/0" },
          { "dst": "::/0" }
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}
EOF

Let’s check the status of the pods with crictl ps.

sudo systemctl daemon-reload
# sudo systemctl start containerd
# start로 했다가 변경 사항이 반영되지 않아서, 안되더라 꼭 restart 를 하라
sudo systemctl stop containerd
sudo systemctl restart containerd
sudo systemctl enable --now containerd

Check it in another tab.

sudo crictl pods

ubuntu@ip-10-0-136-250:~$ sudo crictl pods
POD ID              CREATED              STATE               NAME                         NAMESPACE           ATTEMPT             RUNTIME
9e0988cda0127       About a minute ago   Ready               static-web-ip-10-0-136-250   default             0                   (default)

PODID=$(sudo crictl pods | awk /static-web/{'print $1'})
PODIP=$(sudo crictl inspectp -o go-template --template '{{.status.network.ip}}' ${PODID})

curl $PODIP:80

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Finally, it works….
Last test, go go.

ROUTABLE_NODE_IP=$(ip -o r g 1 | awk '{print $7}')

curl $ROUTABLE_NODE_IP:8080

This also works well. Thankfully.

4. Running with kubelet only (kubelet standalone)

Finalizing the CNI settings in /etc/cni/net.d/10-containerd-net.conflist, based on the installed CNI plugin and CNI containerd, completes the CNI setup.

# /etc/containerd/config.toml
disabled_plugins = [] -> enabled_plugins = ["cri"]
SystemdCgroup = false

Simply placing the configured pod in the /etc/kubernetes/manifests/ folder will maintain the service.

In this section, set up WordPress and persistent volumes to complete the deployment of multiple WordPress hosts and assign them to an API gateway.

5. Building Kubelet Standalone Service

Ultimately, it failed. Refer to the following for more details:
Failed content blog: aws-ec2-kubernetes-wordpress-hosting

Fin

전체 과정 블로그 /aws-ec2-mariadb-환경-설정