Install k3s kubernetes cluster

Setting up a Kubernetes lab is often a necessity in my daily work. Testing various software, understanding how to run it, configure it, learning new features, and so on. It's part of the job.
Running directly on the cloud could raise costs and using local development tools is a choice I am always going with (when possible).
In my experience I've found that k3s is a suitable tool for the job:
- It's easy to install
- It's easy to configure
- Requires low resources
Since working on different machines and operating systems I've standardized the setup to work only on Ubuntu. Multipass is a tool that comes to the rescue. It allows me to create virtual machines from the terminal and makes my life easier.
Another neat feature is utilizing cloud-init for the provisioning of VM instances on creation.
So my recipe for the homemade Kubernetes laboratory is:
- K3s
- Multipass
- cloud-init
Architecture
The resulting Kubernetes cluster will look like the one in the picture below.

What you get in the end is a Kubernetes cluster accessible from the host operating system.
➜ ~ multipass list
Name State IPv4 Image
k3s Running 10.117.145.168 Ubuntu 22.04 LTS
172.17.0.1
10.42.0.0
10.42.0.1
➜ ~ hostname
qdnqn
➜ ~ curl 10.117.145.168
404 page not found
➜ ~ curl 10.117.145.168/kafdrop -IL
HTTP/1.1 302 Found
Content-Length: 0
Date: Sun, 22 Jan 2023 13:24:45 GMT
Location: http://10.117.145.168/kafdrop/
HTTP/1.1 406 Not Acceptable
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin,Accept,X-Requested-With,Content-Type,Access-Control-Request-Method,Access-Control-Request-Headers,Authorization
Access-Control-Allow-Methods: GET,POST,PUT,DELETE
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 3600
Content-Language: en-US
Content-Length: 10807
Content-Type: text/html;charset=UTF-8
Date: Sun, 22 Jan 2023 13:24:46 GMT
This is made possible by the svclb-traefik pod. If you are interested in the configuration of the Traefik on k3s check the blog post below.

As you can see in the picture below, the process is broken into a few steps:
- Launch multipass command for creating VM
- Provision the new VM using cloud-init
- Run commands on the first boot using cloud-init
- Run setup after the provision is completed using .bash_profile (running only once on the first boot)
- Done

Multipass
Launching a new machine using multipass is given in the /.start.sh
script. Let's look at the contents of it.
#!/bin/bash
cd "$(dirname "$0")"
echo "$(pwd) is current working directory."
multipass launch -n k3s --mem 4G --disk 40G --cpus 4 --cloud-init init-config.yaml --mount ./:/home/ubuntu/cloud-native-infrastructure
Changing the current working directory to the directory of the location of the script. After that multipass is starting the VM.
The cloud-init configuration is given in the --cloud-init init-config.yaml
. The file /init-config.yaml
holds all the information for the cloud-init provisioning process.
After the cloud-init provision process is done multipass will mount this repository in the home directory of the ubuntu user.
Let's review init-config.yaml
. For example, Run commands have a duty to install needed software for the VM.
runcmd:
- 'curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"'
- 'curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"'
- 'echo "$(cat kubectl.sha256) kubectl" | sha256sum --check'
- 'install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl'
- 'mkdir -p /etc/apt/keyrings'
- 'curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg'
- 'echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null'
- 'apt-get update'
- 'apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y'
- 'snap install go --classic'
- 'curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3'
- 'chmod 700 get_helm.sh'
- './get_helm.sh'
- 'wget https://go.dev/dl/go1.19.linux-amd64.tar.gz'
- 'sudo tar -xvf go1.19.linux-amd64.tar.gz'
- 'sudo mv go /usr/local'
- 'echo "$(hostname -I | cut -d" " -f1) k3s.local.registry.com" >> /etc/hosts'
- 'curl -sfL https://get.k3s.io | sh -'
- "echo 'eval `ssh-agent`' >> /etc/bash.bashrc"
- 'echo "sudo chown -R ubuntu /etc/rancher" >> /etc/bash.bashrc'
- 'echo "sudo chown ubuntu /var/run/docker.sock" >> /etc/bash.bashrc'
- 'echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/bash.bashrc'
- 'echo "alias k=kubectl" >> /etc/bash.bashrc'
- 'ssh-keyscan -t rsa github.com >> /etc/ssh/ssh_known_hosts'
- 'eval `ssh-agent`'
- 'cd /home/ubuntu/'
- 'chown -R ubuntu /home/ubuntu'
- 'touch /home/ubuntu/.cloud-init.done'
This is the part where you want to add software before the actual login. If you want to add any commands to run before the first login - add them here.
I am using this to install kubectl, helm, k3s, go, docker, change ownership in the VM to the ubuntu user (default user on k3s), make an alias for kubectl, and create a file when cloud-init is done: .cloud-init.done
.
The file .cloud-init.done
is used by ~/.bash_profile
to know when the cloud-init process of provisioning is done.
The ~/.bash_profile
is appended using the write_files method from cloud-init.
write_files:
- content: |
{
"insecure-registries" : ["k3s.local.registry.com:5000"]
}
path: /etc/docker/daemon.json
- content: |
mirrors:
k3s.local.registry.com:5000:
endpoint:
- "http://k3s.local.registry.com:5000"
path: /etc/rancher/k3s/registries.yaml
append: true
- content: |
#!/bin/bash
SP="/-\|"
if [[ ! -f "/home/ubuntu/.cloud-init.done" ]]; then
echo -n "Waiting for cloud init to finish "
fi
while [[ ! -f "/home/ubuntu/.cloud-init.done" ]];
do
printf "\b${SP:i++%${#SP}:1}"
sleep 1
done;
if [[ ! -d "/home/ubuntu/cloud-native-infrastructure" ]]; then
echo -n "Waiting for mount of directory "
fi
while [[ ! -d "/home/ubuntu/cloud-native-infrastructure" ]];
do
printf "\b${SP:i++%${#SP}:1}"
sleep 1
done;
eval `ssh-agent`
sudo chown ubuntu /var/run/docker.sock
cd /home/ubuntu/
cd /home/ubuntu/cloud-native-infrastructure
./scripts/setup-infrastructure.sh
export GOROOT=/usr/local/go
export PATH=$GOROOT/bin:$PATH
path: /home/ubuntu/.bash_profile
As can be seen when spawning a shell this shell script will also run and do the next things:
- Wait for the cloud-init provision process to complete using a semaphore file
.cloud-init.done
- Change ownership of the docker socket to the current user
- Run the script from the repository /scripts/setup-infrastructure.sh
- Add Go to the path
The next important part is the scripts/setup-infrastructure.sh
.
This script holds the next content.
#!/bin/bash
cd "$(dirname "$0")"
if [[ ! -f "/home/qdnqn/.run.once" ]]; then
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
while [[ $(kubectl get crd | grep ingressroutes.traefik.containo.us | wc -l) == 0 ]];
do
sleep 5
done;
docker run -d -p 5000:5000 --restart=always --name registry registry:2
kubectl create ns kafka
kubectl create ns kafdrop
kubectl apply -f resources/raw/yaml/setup/traefik-config-k3s.yaml
helm upgrade --install kafka ../charts/kafka --namespace kafka --values ../charts/kafka/values.yaml
helm upgrade --install kafka ../charts/kafdrop --namespace kafdrop --values ../charts/kafdrop/values.yaml
VM_IP=hostname -I | cut -d " " -f1
sed -i "s/{VM_IP}/${VM_IP}/g" resources/raw/yaml/setup/ingresses.yaml
kubectl apply -f resources/raw/yaml/setup/ingresses.yaml
touch /home/ubuntu/.run.once
fi
This script will run only once. Using the same semaphore file technique only this file is named .run.once
.
At the first spawn of the shell inside the VM using multipass shell k3s
, the setup-infrastructure script will start the docker registry inside the VM but outside of the Kubernetes cluster.
It will create needed namespaces and ingresses from the resources dir.
It will helm upgrade --install
needed charts.
At the end of the script, it will create .run.once
file in the home directory and this part will not be run at the subsequent spawns of the shell.
The process itself can take around 5-10 minutes to complete. Depends on your machine and internet speed but you will get a fully functional Kubernetes cluster.
After reading the post I suggest reviewing the architecture image of the process itself.
If you want to get the repository, navigate to the link below.
If you need any help or suggest improvements please leave a comment to start a conversation.