Jun 19, 2023 6 min read

Expose Kubernetes service securely to the internet on k3s

Introduction

Kubernetes has revolutionized container orchestration, enabling developers to efficiently manage and scale their applications. Among the various Kubernetes distributions available, k3s has gained popularity due to its lightweight and easy-to-deploy nature.

To install single node k3s cluster is pretty straightforward using k3s provided install script is enough. If you need simple cluster for local development or local testing see the article below.

While deploying a k3s cluster within your local environment is straightforward, exposing it securely to the internet requires careful consideration.

This article will guide you through the process of installing k3s cluster and exposing your Kubernetes cluster workload to the internet via LoadBalancer service while maintaining Kubernetes API in a private network.

This pattern is robust in the security aspect because the Kubernetes API is still in a private network which restricts the attack path to the API of the cluster. Drawback is that cluster API is only accessible inside of the network.

Network

This installation will require that you have a virtual machine that already has a network interface that is exposed to the internet.

$ ifconfig
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet *********  netmask 255.255.255.0  broadcast *********
        inet6 *********  prefixlen 64  scopeid 0x0<global>
        inet6 *********  prefixlen 64  scopeid 0x20<link>
        ether *********  txqueuelen 1000  (Ethernet)
        RX packets 51536  bytes 44630374 (44.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 49946  bytes 45883625 (45.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

We are interested in the interface that has the public IP assigned to it. In this case, it is ens3. It can be any interface with the public IP.

We will record inet ********* IPv4 address for later use.

Kubernetes cluster install with k3s

TOKEN=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 64 ; echo '')
echo "Generated token is: $TOKEN"
echo "Be sure to save it!"
curl -Lo ./k3s https://github.com/k3s-io/k3s/releases/download/v1.26.5+k3s1/k3s
sudo chmod a+x k3s
sudo cp k3s /usr/local/bin/k3s

Install k3s

The snippet above will download the k3s tools for us. The cluster is not running yet.

Kubernetes cluster will be isolated on the separate interface. LoadBalancer services will only be exposed to the internet. Kubernetes cluster API will be only accessible from within the private network.

sudo ip link add vethk3s0 type veth peer name vethk3s
sudo ip addr add 10.8.5.1/24 dev vethk3s
sudo ip link set vethk3s up

Create a virtual ethernet interface

This will create a new network interface of virtual ethernet type.

sudo su
ufw enable
echo "1" > /proc/sys/net/ipv4/ip_forward
exit

Enable firewall and enable IP forwarding

This will enable the firewall to filter the traffic. IP forwarding is needed when Linux is acting as a router.

Allow 22/ssh and 80/http protocols in the firewall.

sudo ufw default reject incoming
sudo ufw allow 80/tcp
sudo ufw allow 22

Reject all incoming requests - allow only http and ssh.

Starting the master node:

sudo k3s server \
--kubelet-arg="cloud-provider=external" \
--cluster-init \
--disable traefik \
--token $TOKEN \
--etcd-arg '--client-cert-allowed-hostname 10.8.5.1' \
--etcd-arg '--initial-advertise-peer-urls=https://10.8.5.1:2380' \
--etcd-arg '--listen-peer-urls=https://127.0.0.1:2380,https://10.8.5.1:2380' \
--etcd-arg '--listen-metrics-urls=https://127.0.0.1:2381' \
--etcd-arg '--advertise-client-urls=http://10.8.5.1:2379' \
--etcd-arg '--listen-client-urls=https://127.0.0.1:2379,https://10.8.5.1:2379' \
--tls-san 10.8.5.1 \
--node-ip 10.8.5.1 \
--advertise-address 10.8.5.1 \
--bind-address 10.8.5.1 \
--flannel-iface=vethk3s \
--write-kubeconfig-mode 644

Run single-node k3s master node

After booting the cluster the master node is ready as it is shown below.

$ kubectl get nodes
NAME      STATUS   ROLES                       AGE     VERSION
qdnqn-0   Ready    control-plane,etcd,master   4m23s   v1.26.5+k3s1

Current node status.

Restarting node master

After the initial start of the master node if the node needs to be stopped and started again next starting of the cluster is shown below.

sudo k3s server \
--kubelet-arg="cloud-provider=external" \
--cluster-init \
--disable traefik \
--token $TOKEN \
--tls-san 10.8.5.1 \
--node-ip 10.8.5.1 \
--advertise-address 10.8.5.1 \
--bind-address 10.8.5.1 \
--flannel-iface=vethk3s \
--kubelet-arg='address=10.8.5.1' \
--write-kubeconfig-mode 644

Deploy Nginx to the cluster

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app.kubernetes.io/name: nginx
  ports:
    - protocol: TCP
      port: 80
      nodePort: 30272
      targetPort: 80
  type: LoadBalancer

Nginx and LoadBalancer service which will expose the Nginx to the internet

💡

The nodePort is defined to use the port 30272 which is in fact the port we configured in our NAT in the previous snippet.

Applying the manifest and inspect the final state.

$ kubectl apply -f manifest.yaml
$ kubect get pods
NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-579b58dcd-pt2c9   1/1     Running   0          6s
$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.43.0.1       <none>        443/TCP        14m
my-service   LoadBalancer   10.43.249.240   10.8.5.1      80:30272/TCP   25s

Expose LoadBalancer to the internet

sudo iptables -A PREROUTING -t nat -i ens3 -p tcp --dport 80 -j DNAT --to-destination 10.8.5.1:30272
sudo iptables -A FORWARD -p tcp -d 10.8.5.1 --dport 30272 -j ACCEPT
sudo iptables -A POSTROUTING -t nat -s 10.8.5.1 -o ens3 -j MASQUERADE

NAT the requests from the public interface to the vethk3s on specific ports.

The snippet above will forward the requests from the public interface to the vethk3s interface.

This will preform NAT in the following scenario:

Packet is coming to the 80 port on the public interface ens3
NAT the packet to the 10.8.5.1 on the port 30272 (NodePort of the LoadBalancer service)
Outgoing (from the cluster) packets will be reverted to the public interface ens3

Testing the outside connectivity

$ PUBLIC_IP="{YOUR PUBLIC IP HERE}" 
$ curl http://{$PUBLIC_IP}/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

External machine ping

The Nginx responded from the Kubernetes cluster to our request. The next step should be to install an ingress controller to handle multiple routing on the application level.

You could use Traefik or the Nginx. Bind the LoadBalancer service to the ingress controller to the ingress controller and you can host multiple applications on your Kubernetes cluster which are accessible from the Internet.

This demo is a simple demonstration of how to expose the Kubernetes cluster workload to the internet.