Kubernetes operators

Kubernetes is the current de facto standard for the deployment and running of applications that are suitable for modern cloud platforms. A declarative way of defining infrastructure state using YAML allows a super easy definition of the scheme for the deployment of the application. Deploying stateless applications is not a big deal. On the other hand — deploying distributed stateful applications, configuring, and operating them is a challenging task.

Kubernetes addressed this issue by allowing developers to extend it, using the Kubernetes operator. The operator reacts to the custom resource and reconciliate the state in the cluster with the state defined in the custom resource, by implementing logic embedded in the operator itself.

When designing/writing an application, intended to run on the Kubernetes, one should take into account capabilities provided by Kubernetes and take that information when designing software architecture. It can speed up implementation, make an application more reliable and the code can focus more on business logic itself.

Creating operator

There are multiple ways to create an operator. You could write one from scratch using Kubernetes client-go. It’s a tedious task and the learning curve is steep. As an alternative, multiple tools provide boilerplate code and speed up the writing of operators. Popular ones are Operatorsdk and Kubebuilder. The focus of the article will be on creating an operator using Kubebuilder. Let’s create an operator which will create a pod running a simple HTTP API and bind some data to the HTTP API.

Also, check out https://github.com/kubernetes-sigs/kubebuilder.

Kubebuilder example code

Kubebuilder provides CLI for creating and managing operator projects. To start a new project, one would only need to hit (Assuming Linux-based Terminal):

kubebuilder init --domain clientmgr.io --repo clientmgr.io/tutorial
kubebuilder create api --group qdnqn --version v1 --kind Client

Kubebuilder will create a directory structure and you can start developing the operator straightaway. The core components of an operator are found in the following locations:

• api/v1/clients_type.go
• controllers/client_controller.go

The first file, api/v1/clients_type.go, is holding the structure of the Custom resource. The second one is holding controller logic.

Creating custom resources in the api/v1/clients_type.go is described below.

// ClientSpec defines the desired state of Client
type ClientSpec struct {
  // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
  // Run "make" to regenerate code after modifying this   file
  ClientId            string `json:"clientId,omitempty"`
  ContainerImage      string `json:"containerImage,omitempty"`
  ContainerTag        string `json:"containerTag,omitempty"`
  ContainerEntrypoint string `json:"containerEntrypoint,omitempty"`
}

// ClientStatus defines the observed state of Client
type ClientStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Run "make" to regenerate code after modifying this file
  ClientStatus string `json:"clientStatus,omitempty"`
  LastPodName  string `json:"lastPodName,omitempty"`
}

Kubebuilder creates the boilerplate, the user only needs to add custom fields, like in the code block above (ClientId, ContainerImage, ContainerTag, and ContainerEntrypoint). By doing this, the custom resource is defined. To create Custom Resource Definition user can run make manifests and make install in the root directory of the project, to apply them to the k8s cluster.

Custom Resouces are created but how to implement custom logic? Let’s check controllers/client_controller.go.

The main code goes into the reconcile function. As seen in the gist below you can see that code is behaving like a state machine.

Checks are performed and action is triggered based on three states of the custom resource:

• Pending: Switch to Running state.
• Running: Create a pod and bind to the HTTP API or Switch to cleaning to rebind to another pod.
• Cleaning: Check if the pod has any active clients through HTTP API; If not delete the pod. Otherwise if needed bind the client to the new pod.

It is suggested to maintain reconcile function as clear as possible. In this example, code is constructed only for clear demonstration for readers. When maintaining operator code It would be best to export code and use functions in the reconcile loop to maintain readability.

The operator itself is bounded to Kubernetes and doesn't do anything by itself. It needs a, most often stateful, application to operate on. In this case we are operating on the custom HTTP API written in GO.

The gist is given below.

It's a simple HTTP API serving on some routes:

• GET on the /client/:id
• GET on the /hasClients
• POST on the /addClient

All routes are served on the 8080.

An interesting aspect of Kubernetes is Object ownership. The parent-child relationship helps when objects are deleted. When deletion occurs deletion can propagate from parent to child or not. This can be modified by the flag
--cascade=orphan. On the other way around when deleting a child, parent will persist.

In this case, an operator is acting also like a controller, something like Deployment or StatefulSet, so we will set ownership reference. Doing it this way when the custom resource is deleted it will propagate to the pods also. Great.

More reading at the https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/.

To make the operator do some things let’s apply custom resources.

In the gist, there are 3 clients defined: two pod-owners and one extra client who will bind to the API in the pod created from client-sample-2.

Let’s observe the behavior of the operator before/after applying the custom resources.

Below you can find a terminal session describing the behavior of the operator created.

An operator is deployed in the tutorial-system namespace (L3). We can see that the default namespace is empty before applying custom resources (L1). After clients have been applied to the cluster, pods are created automatically — Something similar would’ve been created using Deployment or StatefulSet (L10). What makes our operator distinct is the fact that It does understand applications running HTTP API.

Conclusion

In this article, a simple operator use case is described. Operators can be used to operate distributed applications eg. Hazelcast, DBS, Kafka, and so on. These applications indeed have their operators already available for use by end-users.

It takes a lot of effort to create reliable operators. The use case is simplified to demonstrate how one could create an operator using Kubebuilder. This operator is created only for demonstration and It could be improved a lot.

Using this design pattern to deploy applications to the Kubernetes should be justified. Not every application is eligible for deployment using operators. Trade-offs should be taken into consideration before starting on this journey.

Complete code for the operator and HTTP API can be found on the Github.

GitHub - qdnqn/kubebuilder-operator

Contribute to qdnqn/kubebuilder-operator development by creating an account on GitHub.

GitHubqdnqn

GitHub - qdnqn/gohttp-api

Contribute to qdnqn/gohttp-api development by creating an account on GitHub.

GitHubqdnqn