Kubernetes Controllers Part One

Posted on 2019-06-26 Edited on 2019-06-29 Disqus:

Deconstructing an existing controller

In order to get a deeper understanding into the design and logic of a Kubernetes controller, we will start by deconstructing the example that comes with the Kubernetes source code. We will examine the structure of the source code, the code generation that automates a lot of the boiler plate and then the actual controller (controller.go) code that manages the logic.

Use-case of sample-controller

A Kubernetes controller is designed to take the functionality that the kubernetes scheduler has and apply its mechanisms to additional use-cases. The “original” controller was an etcd controller which allowed kubernetes to become aware of etcd and its mechanisms so that kubernetes could suddenly create/scale/upgrade and destroy etcd clusters correctly.

The sample-controller is a simplistic controller that creates a resource called Foo that defines a number of additional resources and constraints.

Core components of a controller

A controller can leverage a number features of the client-go library in order to efficiently interact with the Kubernetes API, this section will cover a few of the concepts exposed from the client library that are used by the sample-controller

Informer

Without the concept of the informer a controller would have to repeatedly “poll” the Kubernetes API for state information, this can be an anti-pattern that can cause additional load on the Kubernetes API and create controller overhead as it processes additional reply data from a poll. In order to provide an efficient method of determining the state of a resource in Kubernetes the client-go library has the concept of an informer. The informer allows a client to specify a particular resource and the operations to be “informed” about.

Create an informer

This will set up the informer, and then start with a channel to can signify when it needs to shut down.

1 2	kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) kubeInformerFactory.Start(stopCh)

Add handlers for various events

This will add three even handlers for add/updates/delete events on the deployments resource within kubernetes

kubeInformerFactory.Apps().V1().Deployments().Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
	AddFunc: addDeploymentFunction(),
	UpdateFunc: updateDeploymentFunction(),
	DeleteFunc: deleteDeploymentFunction(),
})

Lister

A lister provides the capability to retrieve various objects (types) from the Kubernetes API, and is part of the informer that we created previously however instead of creating various callback functions the client invokes a “list” as part of a query. The Lister query typically will be invoked with two parameters, the namespace that the resource resides in and the name of the resource of a particular resource type.

deployment, err := c.deploymentsLister.Deployments(foo.Namespace).Get(deploymentName)
// If the resource doesn't exist, we'll create it
if errors.IsNotFound(err) {
	deployment, err = c.kubeclientset.AppsV1().Deployments(foo.Namespace).Create(newDeployment(foo))
}

Getting the source code

There are two components that will be needed in order to build our Kubernetes controllers, one being the code generation tooling and the second being the actual code that makes up the sample controller.

Kubernetes Code Generation tools

go get -u k8s.io/code-generator/...

Note Why the ...

From the command go help packages:

An import path is a pattern if it includes one or more “…” wildcards, each of which can match any string, including the empty string and strings containing slashes. Such a pattern expands to all package directories found in the GOPATH trees with names matching the patterns. As a special case, x/… matches x as well as x’s subdirectories. For example, net/… expands to net and packages in its subdirectories.

Kubernetes Sample Controller

go get k8s.io/sample-controller

How the source code is organised

There are a few areas that are key to how the structure of this all makes sense, in order to understand this we will walk the directory structure and identify some of the key files. To begin we should be in the directory with the sample-controller source code is located (should be $GOPATH/go/src/k8s.io/sample-controller).

main.go
controller.go
|-pkg/
|  |-apis/
|  |   |-samplecontroller/
|  |-generated/

main.go

This file has the main() function that starts the controller logic and does the following:

Set up the signal handlers (So a ctrl+c or SIG{X} will be handled by the controller)
Parse the command line flags
Build up two client sets, and start two informers that watch for the signal handler
Build the controller configuration
Start the newly configured controller, and again pass the signal handler

controller.go

This source code file contains all of the logic and functions that provide all of the kubernetes controller functionality.

Function flow

c := Controller

c.Run() -> c.runWorker() -> c.processNextWorkItem() -> c.syncHander()

The two main functions that are invoked are syncHandler() and handleObject.

The syncHandler with be invoked with a namespace and object name that is taken from the work queue, it will then attempt to find the object in the Kubernetes API and process it.

Split the key into a namespace/object name
Use a Lister to find the object in the API server
Parse the object and look for it’s deployment
Use a Lister to find the deployment object referred to in the Foo resource
If the deployment doesn’t exist create it
If the Foo resource has changed update the deployment

The handleObject is invoked through the informer and will add things to the work queue that will be processed eventually by the syncHandler function.

pkg/apis/samplecontroller

In this folder are a number of files that identify the controller name, the controller version and then all of the information that the controller needs in order to define itself or interact with other systems. All of these files are then processed by the code-generator tools in order to build out the correct source to work as expected with Kubernetes.

Test the code generation

The sample-controller repository comes with pre-generated code already in the packages folder. We can teat the code generation tools by moving the generated code to a different location and re-generating out code.

The following will move the generated folder to a backup folder:

mv pkg/apis/generated pkg/apis/generated-backup

The update-codegen.sh script is a hardcoded script that will regenerate all the code for the sample controller.

./hack/update-codegen.sh

Test that our code compiles correctly from out generated code.

go build

Custom Resource Definition (CRD)

This will update the Kubernetes API to inform it that a new resource of type Foo exists and is controlled by the controller samplecontroller.k8s.io version v1aplpha1.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: foos.samplecontroller.k8s.io
spec:
  group: samplecontroller.k8s.io
  version: v1alpha1
  names:
    kind: Foo
    plural: foos
  scope: Namespaced

Deployment

This deployment will update the expected state in the Kubernetes API, which will be processed by the controller.

apiVersion: samplecontroller.k8s.io/v1alpha1
kind: Foo
metadata:
  name: example-foo
spec:
  deploymentName: example-foo
  replicas: 1