Expose Kubernetes service securely to the internet on k3s
Introduction
Kubernetes has revolutionized container orchestration, enabling developers to efficiently manage and scale their applications. Among the various Kubernetes distributions available, k3s has gained popularity due to its lightweight and easy-to-deploy nature.
To install single node k3s cluster is pretty straightforward using k3s provided install script is enough. If you need simple cluster for local development or local testing see the article below.
While deploying a k3s cluster within your local environment is straightforward, exposing it securely to the internet requires careful consideration.
This article will guide you through the process of installing k3s cluster and exposing your Kubernetes cluster workload to the internet via LoadBalancer service while maintaining Kubernetes API in a private network.
This pattern is robust in the security aspect because the Kubernetes API is still in a private network which restricts the attack path to the API of the cluster. Drawback is that cluster API is only accessible inside of the network.
Network
This installation will require that you have a virtual machine that already has a network interface that is exposed to the internet.
$ ifconfig
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet ********* netmask 255.255.255.0 broadcast *********
inet6 ********* prefixlen 64 scopeid 0x0<global>
inet6 ********* prefixlen 64 scopeid 0x20<link>
ether ********* txqueuelen 1000 (Ethernet)
RX packets 51536 bytes 44630374 (44.6 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 49946 bytes 45883625 (45.8 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
We are interested in the interface that has the public IP assigned to it. In this case, it is ens3. It can be any interface with the public IP.
We will record inet *********
IPv4 address for later use.
Kubernetes cluster install with k3s
TOKEN=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 64 ; echo '')
echo "Generated token is: $TOKEN"
echo "Be sure to save it!"
curl -Lo ./k3s https://github.com/k3s-io/k3s/releases/download/v1.26.5+k3s1/k3s
sudo chmod a+x k3s
sudo cp k3s /usr/local/bin/k3s
Install k3s
The snippet above will download the k3s tools for us. The cluster is not running yet.
Kubernetes cluster will be isolated on the separate interface. LoadBalancer services will only be exposed to the internet. Kubernetes cluster API will be only accessible from within the private network.
sudo ip link add vethk3s0 type veth peer name vethk3s
sudo ip addr add 10.8.5.1/24 dev vethk3s
sudo ip link set vethk3s up
Create a virtual ethernet interface
This will create a new network interface of virtual ethernet type.
sudo su
ufw enable
echo "1" > /proc/sys/net/ipv4/ip_forward
exit
Enable firewall and enable IP forwarding
This will enable the firewall to filter the traffic. IP forwarding is needed when Linux is acting as a router.
Allow 22/ssh and 80/http protocols in the firewall.
sudo ufw default reject incoming
sudo ufw allow 80/tcp
sudo ufw allow 22
Reject all incoming requests - allow only http and ssh.
Starting the master node:
sudo k3s server \
--kubelet-arg="cloud-provider=external" \
--cluster-init \
--disable traefik \
--token $TOKEN \
--etcd-arg '--client-cert-allowed-hostname 10.8.5.1' \
--etcd-arg '--initial-advertise-peer-urls=https://10.8.5.1:2380' \
--etcd-arg '--listen-peer-urls=https://127.0.0.1:2380,https://10.8.5.1:2380' \
--etcd-arg '--listen-metrics-urls=https://127.0.0.1:2381' \
--etcd-arg '--advertise-client-urls=http://10.8.5.1:2379' \
--etcd-arg '--listen-client-urls=https://127.0.0.1:2379,https://10.8.5.1:2379' \
--tls-san 10.8.5.1 \
--node-ip 10.8.5.1 \
--advertise-address 10.8.5.1 \
--bind-address 10.8.5.1 \
--flannel-iface=vethk3s \
--write-kubeconfig-mode 644
Run single-node k3s master node
After booting the cluster the master node is ready as it is shown below.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
qdnqn-0 Ready control-plane,etcd,master 4m23s v1.26.5+k3s1
Current node status.
Restarting node master
After the initial start of the master node if the node needs to be stopped and started again next starting of the cluster is shown below.
sudo k3s server \
--kubelet-arg="cloud-provider=external" \
--cluster-init \
--disable traefik \
--token $TOKEN \
--tls-san 10.8.5.1 \
--node-ip 10.8.5.1 \
--advertise-address 10.8.5.1 \
--bind-address 10.8.5.1 \
--flannel-iface=vethk3s \
--kubelet-arg='address=10.8.5.1' \
--write-kubeconfig-mode 644
Deploy Nginx to the cluster
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app.kubernetes.io/name: nginx
replicas: 1
template:
metadata:
labels:
app.kubernetes.io/name: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app.kubernetes.io/name: nginx
ports:
- protocol: TCP
port: 80
nodePort: 30272
targetPort: 80
type: LoadBalancer
Nginx and LoadBalancer service which will expose the Nginx to the internet
Applying the manifest and inspect the final state.
$ kubectl apply -f manifest.yaml
$ kubect get pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-579b58dcd-pt2c9 1/1 Running 0 6s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 14m
my-service LoadBalancer 10.43.249.240 10.8.5.1 80:30272/TCP 25s
Expose LoadBalancer to the internet
sudo iptables -A PREROUTING -t nat -i ens3 -p tcp --dport 80 -j DNAT --to-destination 10.8.5.1:30272
sudo iptables -A FORWARD -p tcp -d 10.8.5.1 --dport 30272 -j ACCEPT
sudo iptables -A POSTROUTING -t nat -s 10.8.5.1 -o ens3 -j MASQUERADE
NAT the requests from the public interface to the vethk3s on specific ports.
The snippet above will forward the requests from the public interface to the vethk3s interface.
This will preform NAT in the following scenario:
- Packet is coming to the 80 port on the public interface ens3
- NAT the packet to the 10.8.5.1 on the port 30272 (NodePort of the LoadBalancer service)
- Outgoing (from the cluster) packets will be reverted to the public interface ens3
Testing the outside connectivity
$ PUBLIC_IP="{YOUR PUBLIC IP HERE}"
$ curl http://{$PUBLIC_IP}/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
External machine ping
The Nginx responded from the Kubernetes cluster to our request. The next step should be to install an ingress controller to handle multiple routing on the application level.
You could use Traefik or the Nginx. Bind the LoadBalancer service to the ingress controller to the ingress controller and you can host multiple applications on your Kubernetes cluster which are accessible from the Internet.
This demo is a simple demonstration of how to expose the Kubernetes cluster workload to the internet.