thebsbdox

Adding a gateway to kube-vip

Wed, 14 Jan 2026 15:05:39 +0000

So …

That was over a year ago, and well I didn’t write a service mesh (in the traditional sense) although I did write some interesting bits and finally did some MutatingWebHook stuff. With my initial idea being to use sidecars to implement all of the magic proxy stuff. Although I largely ended up getting stuck with some issues around Kubernetes v1.33 and EphemeralContainers which has largely been fixed at this point. So given the Christmas break I decided to have another crack at it!

tl;dr where’s the code -> https://github.com/kube-vip/kube-gateway

kube-gateway (v0.0001)

The architecture for the kube-gateway (today) consists of:

The gateway watcher
The proxy

The gateway watcher

This currently is just a pod that has the permissions to watch for pod updates for things such as an IP address has been assigned or that an annotation has been applied. When the correct annotations are applied to a pod then the watcher will modify the pod.Spec, which I hear you cry 😿 is READ-ONLY! This is true (partially), but the read-only sections typicaly mean I can’t add a volume/configmap/secret and that can render it impossible to add things like secrets or sidecars to a pod (at run-time) impossible. However, it turns out that the pod.Spec.EphemeralContainer is read-write so this does provide us with an opportunity to add an additional container although we still can’t add any volumes as they are immutable.

So to get around this the watcher has this snippet of code:

// Create certificates and then a Kubernetes secret
i.c.createCertificate(pod.Name, pod.Status.PodIP)

// create a secret in kubernetes
err := i.c.loadSecret(pod.Name, i.clientset)
if err != nil {
	slog.Error(err)
}
ec.EnvFrom = append(ec.EnvFrom, v1.EnvFromSource{
	SecretRef: &v1.SecretEnvSource{
		LocalObjectReference: v1.LocalObjectReference{
			Name: secret, // map to the secret we created earlier
		},
		Optional: nil,
	},
})

The watcher itself has it’s own Certificate Authority certs and will use them to create new certificates for the pod (based upon it’s IP address), which are then loaded into a Kubernetes secret. Finally we can reference the secrets inside the EphermeralContainer as environment variables as this is allowed in the Kubernetes spec!

Then the annotations we apply are typically passed to the EphermeralContainer as additional environment variables that the kube-gateway uses!

Annotating your pod for mTLS 🔐

To enable mTLS between two pods we will need to apply a gateway to each pod (to handle the encryption and decryption on either side).
BONUS: If you want to offload the TLS encryption the kube-gateway supports kTLS (in-kernel TLS)!

To enable this then each pod will need the following annotation before enabling encryption:

kubectl annotate pod <pod name> kube-gateway.io/ktls="true"

This is done with the following:

This will apply the gateway to pod-01:

kubectl annotate pod pod-01 kube-gateway.io/encrypt="true"

This will then apply the other gateway to pod-02:

kubectl annotate pod pod-02 kube-gateway.io/encrypt="true"

Annotating your pod for AI 🤖

Annotate your LLM

This is highly experimental at the moment, but the feature set will expand as it is worked on 😀

As of today (15th January 2026) the implementation is expecting your LLM to be running within the cluster somewhere, this is a self imposed limitation of the kube-gateway (and will be changed soon).

A kube-gateway endpoint is required (today) in order to facilitate end-to-end connectivity, we will need to annotate our LLM with a gateway endpoint annotation:

kubectl annotate pod -n ollama ollama-bbbc54cbc-8mqt5 kube-gateway.io/endpoint="true"

Now anything that is interacting with it can be modified with our gateway annotations!

Annotate your AI workloads

Imagine we have some workloads created a long time ago (3 weeks in the world of AI 🙄), and these workloads are using an old/outdated/expensive model and although they workloads perform OK we want to improve them. With the kube-gateway we can modify this on the fly.

Depending on the workload type or language/code or SDK used the connection to the LLM, then a HTTP keep-alive may be used and if this is the case then the existing TCP session will be used for all API calls (and we wont be able to pass it to the gateway). So the netflush annotion will instruct the kube-gateway pod on startup to use some eBPF 🐝 magic to knock the TCP session to be recreated without disturbing the connection. (this annotation is optional and will need investigating, the python ollama SDK doesn’t require this for example).

kubectl annotate pod aiworkload kube-gateway.io/netflush="true"

The next annation for our aiworkload is to change the model:

kubectl annotate pod aiworkload kube-gateway.io/ai-model="gemma2:2b"

Now whenever there is a request our kube-gateway will modify the traffic transparently to use the model specified!

Finally enable the kube-gateway on the workload:

kubectl annotate pod aiworkload kube-gateway.io/ai="true"

eBPF, mTLS and AI oh my..

So under the covers what is actually occuring in order to facilitate this?

Well simply put there are three parts to it:

The watcher takes input and adds an ephemeral container to a specified workload
Some eBPF code is injected to ensure that traffic from the correct process has it’s original destination captured and is now sent to a new destination, which is our gateway!
Our gateway in the ephermal container recieves the connection, and will lookup the original destination. We can make changes to the TCP stream/data (such as in the AI use-case) or we can encrypt it and send it to the pod that now also has a gateway as a ephemeral container attached to decrypt the traffic and send it to the original application.

What’s next?

At this point we can easily sit in the middle of any traffic and transparently mutate it, which allows us a huge opportunity to add in all sorts of guardrails and monitoring opportunities.

mTLS

The main goal in the future would be to redirect to a per-node gateway, which should be doable and drastically reduce the need for a gateway per pod architecture.

AI workloads

Remove the need for a gateway endpoint, which means that only the “client” side would need to have a gateway attached and would mean the “client” can connect to anything transparently. Additionally extending the functionality of what we may want to mutate in the AI payload, such as limitating tokens or having rules about prompts etc.. I guess we shall see!

Either head to kube-gateway.io or the Github repository https://github.com/kube-vip/kube-gateway

Egress v2 in kube-vip

Thu, 17 Jul 2025 10:19:56 +0100

I’m not sure if it was my time working for Docker or just the general confusion around what various companies and startups are doing around container images, but I’ve always opted to go FROM scratch where possible. Several years ago we (me) wanted to implement egress control within Kube-vip, although up to that point I didn’t really want to be involved in the datapath I decided to go for it. My main concern about the datapath is that people get very upset when their packets arrive backwards or get lost 😱. However after a bit of research and seeing what the rest of the world was doing, I decided to opt for a slightly different (if not weird approach) which was to tie the service to a pod and map the external address 1:1 with a pod. There is a ton of detail about that in this blog post 👉 https://thebsdbox.co.uk/2024/11/03/Egress/.

iptables & nftables

In order to implement the egress source NAT I decided to largely duplicate the behaviour of kube-proxy, which is to SNAT the address of the pod to the address of the load balancer service and job done 🎉. However in order to do that we needed to start adding iptables binaries into the kube-vip images. This now suddenly means adding in other dependancies, and as we all know more dependancies means a larger vector for attacking. Under most circumstances a lot of people wouldn’t mind and/or care but unfortunately kube-vip serves as the control plane HA VIP and is in a critical position, so exploiting it would be a big issue. So now we introduce two kube-vip images, one from scratch and another with all the iptables and dependancies.

tech-debt = tech-debt + 1;

The state of iptables & nftables

What, you want some details.. fine

So we’ve been using iptables for years and it largely works, it has its issues with scale but so does everything at one point or another. One of the main issues has been programming rules or lower level abstractions, given we’re ultimately programming the kernel can we get rid of all the binaries. Well the answer to that is:

The answer unfortunately is: No.

Additionally:

We don’t guarantee a stable interface

… awesome, and it has been this way for years and years.

When is iptables not iptables

When it is nftables, of course 🤷🏼‍♂️. It turns out that for the last few years all of our legacy iptables code hasn’t been churning out iptables rules, you’ve been living a lie! Feel free to read this madness at your earliest convenience. But the tl;dr is that in most places the iptables binary is actually a new program that will take your iptables syntax and pump out nftables rules.

So why is no-one writing nftables rules?

Well because you didn’t have to, if the iptables rules are being on the-fly being written as nftables rules then just leave it to that. But additionally the syntax is dramatically different to iptables rules, which means that it comes with it’s own learning curve. This and a few buggy releases of the userland tooling have unfortunately slowed it’s adoption. However, recently there has been quite a few things happening in the nftables space! Namely Kubernetes 1.31 introduced the capability to use kube-proxy in an nftables mode, which is detailed here.

The (finally) growing ecosystem for nftables

The Kubernetes project has finally started to embrace nftables and has opted to write a wrapper around nftables, which is called knftables. This is a wrapper around the nftables binaries, which unfortunately still requires bringing along both the nftables binaires, the libraries and any other dependancies. But what is there if you’re a slightly mad purist who demands scratch.

A pure Go package for manipulating nftables 🎉

I’ve been watching https://github.com/google/nftables for quite some time, actually since November 2023. The thing that was missing from this was the capability to implement SNAT rules in this pure Go implementation. Luckily this feature was implemented not too long after, but then life gets in the way and you’re too busy climbing a mountain 🧗🏻 or doing something else 🚵🏻.

However I finally managed to find some time several weeks ago to finally investigate what the code would look like in order to finally rip out all the external dependancies and program the kernel directly!

Egress v2 finally 🥹!

We’re calling this Egress v2 as it’s a complete re-write of how egress is handled within kube-vip, although to make things as simple as possible we’re just adding an additional annotation to the service that will be used to control egress :-)

kube-vip.io/egress: true - will enable the service to be used as the egress for the pod it is selecting
kube-vip.io/egress-internal: true - will use the internal/Egress v2 mechanism for configuring the source NATing

This move also simplifies the rules that need writing within the kernel, before we would have to write multiple rules to ensure that we didn’t egress when a pod would try and talk to another pod or a service (this would break everything). Additionally added the capability add additional subnets that shouldn’t have the egress address changed or to only egress when sending traffic to a specific destination port. All of this functionality created additional rules that where hard to order or could be accidentally left behind when cleaning rules (on service deletion).

Egress v2 tables and chains

To make it super easy to ensure that kube-vip orders and cleans its egress configuration correctly it will do the following:

Create a table for IPv4 table ip kube_vip_v4 and IPv6 table ip6 kube_vip_v6
Each service creates it’s own chain chain kube_vip_snat_v4_abc the chain contains the IP version (v4/v6) and more importantly the UUID of the service abc in reality would be much longer
In each chain is the rule for a particular service

Example rule: ip saddr { 10.0.0.1 } ip daddr != 10.0.0.0-10.0.255.255 ip daddr != 10.96.0.0-10.96.255.255 snat to 11.0.0.2
The above rule is typical of the rules created for egress, where we ensure that we don’t egress for the podCIDR or the serviceCIDR. We can see that the POD IP 10.0.0.1 is source NAT’d to 11.0.0.2 when not going to those networks.

When a service is deleted we retrieve it’s UUID along with the IP address version and simply delete the chain, and everything is completely tidy.

Additional details If kube-vip is forcefully restarted then it won’t garbage collect old rules. However we can ensure kube-vip is restarted with the environment variable egress_clean which will clean all rules from the kube-vip tables ensuring a clean slate as kube-vip re-participates in being part of hosting load balancing services.

The TL;DR

We’re going to be removing all external dependancies from kube-vip, so no more programs or other tooling to manage the networking (Great for image size and security)! Kube-vip can now speak directly to the kernel and configure your egress rules, in a much better way than before! 🤩

Coming soon!

All of the code has been written and is ready to merge at this time, and we’re hoping to get kube-vip v1.0 in to your hands soon! 🎉

Continuing building your service mesh

Mon, 02 Dec 2024 11:54:19 +0000

In a previous post I detailed (in my head at least) the “shopping list” of bits needed to implement a simple service mesh, you can read that here https://thebsdbox.co.uk/2024/11/30/Building-your-own-service-mesh/. Whilst that post covered some of the theoretical bits and the eBPF 🐝 magic, my aim is to wrap up all of the other pieces needed here.

The proxy

In this “build your own” service mesh, the proxy will live next to the application that we care about.

OMG a sidecar 😱

How that sidecar gets there can be an interesting discussion, so lets look at the choices.

You maintain some yaml, basically your deployments will need to have the sidecar added to to them. This has to be before you deploy as you can’t add a sidecar to an existing pod.
But what about ephemeral containers! (I hear you ask), well they’re pretty good and yes you can add them to an existing pod. BUT, if you need to mount files (like 🔐 certificates), then Volumes need adding to the pod.spec and you can’t do that. AH HA! I hear you think, use secrets and environment variables! that way it doesn’t modify the main body of the pod.spec, just the pod.spec.ephemeralcontainer[x]. Great idea, but it doesn’t work and there has been an issue open about it for nearly (checks notes) 2 years.
Ye olde sidecar injector! The defacto method of modifying a pod.spec before it’s actually committed to the Kubernetes API. (I won’t go into detail as people have written about this for some time)

So regardless of how the proxy gets there, it needs to be there. There will be a proxy in every application that we care about. When one pod wants to talk to another pod it will actually be the proxies that are doing the talking!

What is in a proxy

Our eBPF code
Code that will create connections and TCP listeners for sending and receiving traffic
The required certificates in order for traffic to be encrypted

Proxy startup

On startup out proxy will determine it’s own pid and along with the pod CIDR range add that to an eBPF map, it will then attach our eBPF programs to the kernel. Once that has occurred it will start the proxy listener, this is where our proxy will receive all traffic forwarded by our eBPF program. It will then read the certificates (from the filesystem or environment variables) and once they’re loaded it will start another listener for incoming TLS connections! That’s it.

Proxy running lifecycle

The proxy is listening on it’s internal proxy port
A new connection is received on this port hijacked by eBPF 🐝, where we do a getsockopt syscall with the option SO_ORIGINAL_DST. This returns to us the original destination address and port that this connection was heading to before we hijacked it with eBPF.
We create a new outbound connection to the original destination address, however we substitute the port with that of the proxies TLS listening port. This initiates a new TLS connection between both proxies!
The source proxy will send to the destination proxy the port that we were originally accessing.
The destination proxy will connect to that port and begin receiving traffic from the source.
At this point we have full end-to-end connectivity from one application to another, without the applications realising that we’re in the middle!

Creating Certificates 📝

In order for the TLS to work the certificates will need to be created with the correct details, namely the IP addresses of the pods to ensure that the identification works correctly. This raising a chicken and egg scenario as ideally we require these details asap, however we only can be allocated this IP address once the pod has been created by the Kubernetes API. As we can’t modify the Volume section of a pod once it has been created we can refer to secrets as environment variables before they have been created.

We then write some code to using Kubernetes informers, there is excellent detail here. These informers will “inform” us when a pod has been both created and updated , we care more about the update as this is the operation where the pod.status.podIP will be populated with the address we care about. Once we have this we can create the required certificates and upload them as a secret to be used by the proxy container.

The final piece is the injector 💉

This is relatively straight forward, this piece of code will on startup register through the AdmissionController that certain resources (pods in our case) when created be sent to our code. Which will patch the pod.spec to now include our container as an initContainer, and be sent back to the Kubernetes API server to be scheduled.

In Summary

We “kind of” have the makings of a service mesh at this point (in my mind at least), we transparently move traffic from the application through our proxy, where we can apply what we wish. In this Proof of concept we newly mint certificates and then establish end to end mTLS, where traffic is encrypted between source and destination. Although that doesn’t mean we have to end there 😀

What next..

All the source code for this experiment is available at https://github.com/thebsdbox/smesh so feel free to go and have a look around. It isn’t the tidiest, but it does work 😂

Building your own service mesh

Sat, 30 Nov 2024 16:55:22 +0000

I saw a few mentions about “service mesh” and mTLS amongst other things during the KubeCon US week and given some of the messing around i’d been doing with eBPF recently I asked myself “how hard could it be to write one from scratch”?

The service mesh shopping list

There are a bunch of components that we will need to implement in order for us to implement the “service mesh” type behaviour. Most service meshes implement a heck of a lot more, we’re exploring the basics needed to implement it.

Traffic redirector 🚦

We need a way of taking traffic from an application and sending it elsewhere, typically to our proxy where we will potentially modify the traffic. The traffic needs to be redirected in a way where the application does’t need to know about it occurring, however we need to ensure that the traffic will reach its destination and traffic is returned in a way that makes sense to the application. In most circumstances this is handled by iptables rules that will change the source and destination of the packets as they navigate the kernel. As a pod initiates a connection to another pod within the cluster we will need to redirect it to our program, which we will call the proxy.

The Proxy

Our proxy will need to be listening somewhere that is accessible on the network and as outbound connections are created their destination will be modified to that of the proxy (we also need to keep a copy of that destination somewhere). At this point we will start receiving data from the source and it is here were we opportunity to potentially change the original traffic or parse the traffic and then make decisions based upon what we learn.

The Injector 💉

The injector is code that will modify the behaviour of Kubernetes so that when new workloads are scheduled an additional container could be added, or something could run before the workload starts that will write iptables/nftables rules into the kernel.

Certificates 📝

If we are wanting to use mTLS between pods then we will need to create certificates, these certs will need things like the pod IPs or pod hostnames etc. in order for the certificates to work. Given that we wont know these details until the pod starts we will need to capture this information by watching Kubernetes and creating the certificates when we see a pod being created.

Lets get started 🐝

If I can’t control the traffic then I can’t do anything, so first things first, I’m going to use eBPF in order to manipulate the traffic and make sure that it is sent to where I need it to go. Why eBPF? well because!

So lets walk this through…

There are a bunch of methods for manipulating traffic XDP, TC, sockets etc.. so what’s the choice?

XDP? Nope, no egress and if we’re wanting to capture traffic being initiated out to somewhere else, then that’s egress.
TC? It has egress, BUT it’s already gone through the kernel, iptables, sockets etc.. changing the traffic to send back into the kernel is a bit of a pain.
Sockets, seems like the best option for what we’re aiming for.

The eBPF 🐝 magic 🪄

Our eBPF code is going to manipulate the L3 & L4 behaviour of packets as they traverse the kernel and in some-cases user-land (i.e. the proxy).

The life of our packet is the following!

For this walkthrough:

pod-01 is 10.0.0.10
pod-02 is 10.10.0.20

Our eBPF program is started and is passed the CIDR range of pods in our Kubernetes cluster and the pid of the proxy, this is done through an eBPF map.
The application within the pod (pod-01) is wanting to create an outbound connection connect(), in this case to pod-02. This would typically be a high internal port 32305 (for example) attempting to connect outbound.
The eBPF program will change the destination from 10.10.0.20 to the proxy that is listening on localhost, so 10.10.0.20:<port> would become 127.0.0.1:18000.
We also stuff the original destination address and port into a map, which uses the socket “cookie” as it’s key.
The proxy on 127.0.0.1:18000 will receive all the TCP magic from the application that started the connection and once the socket has been established we hook in with eBPF.
Here we will add to another map the source port 32305 and the unique socket “cookie”.
The proxy has an established connection from the application, however it needs to know the original destination, we do this through calling a syscall getsockopt with a specific option SO_ORIGINAL_DST. This is captured by eBPF, which it does a look up on the src port 32305 to find the cookie, it then uses the cookie to look up in another map to return the original destination 10.10.0.20:<port>.
The proxy can now establish a connection outbound to the destination pod or another proxy (this will be covered later).
As traffic is read() from the proxy it is then forwarded to the internal connection and the application in pod-01 processes it as if there was no proxy in the middle.

Why do we pass the `pid` of the proxy into the eBPF program? (I hear you ask)

Well, we would end up in a loop if the proxy has it’s out bound connections looped back to itself. So if we see a connection from the proxy then we don’t redirect it.

Abridged logs

$ kubectl logs  pod-01 -c smesh-proxy
[2024/12/02T10:17:58.618] [application] [INFO] [main.go:66,main] Starting the SMESH 🐝
[2024/12/02T10:17:58.618] [application] [INFO] [main.go:94,main] detected Kernel 6.8.x
[2024/12/02T10:17:58.682] [application] [INFO] [connection.go:23,startInternalListener] internal proxy [pid: 7] 127.0.0.1:18000
[2024/12/02T10:17:58.682] [application] [INFO] [connection.go:33,startExternalListener] external proxy [pid: 7] 0.0.0.0:18001
[2024/12/02T10:17:58.682] [application] [INFO] [connection.go:62,startExternalTLSListener] external TLS proxy [pid: 7] 0.0.0.0:18443

< Proxy is up and running>
< We receieve a forwarded connection from eBPF 🐝 >

[2024/12/02T10:18:14.080] [application] [INFO] [connection.go:75,start] internal proxy connection from 127.0.0.1:33804 -> 127.0.0.1:18000

< We've looked up the connection through eBPF to find the original destination >
< The proxy connects to pod-02 (on it's local proxy port, where it takes care of forwarding to the application in the same pod) and we can now start sending traffic from pod-01 through the proxy >

[2024/12/02T10:18:14.087] [application] [INFO] [connection.go:156,internalProxy] Connected to remote endpoint 10.10.0.20:18443, original dest 10.10.0.20:9000

< The application in pod-02 has established a new connection in the opposite direction >

[2024/12/02T10:18:16.081] [application] [INFO] [connection.go:95,startTLS] external TLS proxy connection from 10.10.0.20:47292 -> 10.0.0.10:18443

Summary

This post steps through the bits needed in order to form a service mesh and how we use eBPF in order to redirect traffic to another process listening within the same pod. We know that this is achievable, but we now need to understand how to architect these pieces and get traffic across to the other pod! (which i’ll cover in the next post)

UPDATE: That post is now available here

Egress

Sun, 03 Nov 2024 16:14:21 +0000

This is the third time I’ve tried to write this, a random panic on my Hackintosh wiped out half a blog post and the other attempt read like garbage 🫠. So here we go! I recently added Egress IPv6 to kube-vip, but it isn’t something that I really advertised as functionality in kube-vip, so I though it best to actually write about it. The concept of Egress within Kubernetes is a bit of an interesting one because it also exposes a bit of confusion when it comes to Kubernetes networking. Namely what actually comes out of the box with a Kubernetes cluster.

Kubernetes Networking

When we think about a simple Kubernetes cluster (and deploying it), then we need to consider all of the additional components that are actually required in order for the cluster to actually work. These are commonly thought of as the CxI plugins, where the x is usually R for the runtime, N for the networking plugin and finally S for storage. Without a CRI, there is simply no functionality to stop and start containers effectively making your cluster pointless and additionally without a CNI your containers wont have any networking capabilities! (well it turns out that’s not strictly true).

The CNI in it’s most basic form is largely doing:

Pod networking with IPAM
Multi node networking, allowing pods to speak to one another as if they were on a big flat network (often an overlay)
Network Policy
Ingress typically though hostPort

Above is a gross over simplification of what CNIs accomplish and most now do far more than just basic networking, however there is a huge amount of networking functionality that isn’t explained. Who or what creates a Kubernetes service and manages the forwarding to a pod that is selected, and additionally when a pod wishes to send traffic externally to the cluster who or what is happening.

Kube-Proxy

So if you do a kubectl get pods -A on a cluster you may encounter kube-proxy pods, which typically run on all nodes as a daemonset and are largely there to manipulate the masses of iptables rules required in order to make the internals of a Kubernetes cluster work. Whenever a service is created kube-proxy will create rules that will watch for ingress traffic to that service address, additionally it also keeps track of the endpoints that match that particular service. These rules, regardless of the technology (currently iptables but soon nftables in upstream) will modify traffic that is destined for a service IP and change that destination to one of the endpoint IP addresses. There is a great overview (in epic detail) available at learnk8s.

Where things get interesting is egress traffic, if you had a pod that had busybox+curl and you were using that the curl https://google.com it will typically just work, but the thing about the pod network is that it isn’t meant to be directly accessible! So how does return traffic get to the pod?

So the answer is pretty simple, and it is pretty much the same logic that applies to most household internet connections. This logic is the idea of NAT (Network Address Translation), but simply put like the pod in a Kubernetes cluster your home laptop isn’t directly accessible on the internet however the ADSL/Fibre router is. When an internal device (or pod) wants to connect to the outside world the traffic will traverse the either a physical router or routing logic in the kernel and the destination address is changed to the address of the router. So in the diagram below google.com will receive a connection from the external address which it can communicate with, when it receives traffic from this connection it will then change the address back to the internal address and traffic is then passed back internally.

┌────────────────────────────────────┐                                                 
│┌────────────────┐                  │                                                 
││Pod-01 10.0.0.3 ┼─────────┐        │            ┌───────────────────────────┐
│└────────────────┘         │        │            │ google.com                │
│                           │        │            └───────────────────────────┘
└───────────────────────────▼────────┘                          ▲              
                 Node-01 192.168.0.22───────────────────────────┘

So this is fine, effectively each node in the cluster will take care of handling the traffic originating from pods by making it look like it’s originating from the node where the pod resides and in most cases this works. However if your cluster has 1000 nodes and you have a pod that wants to access something external protected by a firewall, well that firewall will now need 1000 rules allowing access because who knows where that pod will be scheduled to run in the future. Or perhaps one of those modern app transformation programs actually works (I’ve been part of a myriad of failed attempts to create these programs) and you’ve finally migrated an application that has some static IP requirements, well again there is no way to have anything static in this scenario.

Egress Magic 🪄

The most common solution to implement “controlled” Egress within a Kubernetes cluster is the concept of an Egress gateway and in most cases this is another small pod (or pods) that sit within the cluster that traffic is forwarded to. This gateway will then do the same manipulation however it will typically have a series of IP addresses that it can use dependant on who the originating traffic is coming from. So in theory we could tie our legacy application to its original IP address and the gateway will then take care of manipulating the traffic to make it appear as though this original IP address is still in use.

Node-01 192.168.0.22                                      
┌──────────────────────────────────────────┐                                    
│┌────────────────┐                        │                                    
││Pod-01 10.0.0.3 │                        │                                    
│└───────┬────────┘                        │                                    
│        │                                 │                                    
└────────┼─────────────────────────────────┘                                    
         │                                                                      
         │                                         ┌───────────────────────────┐
         │                      ┌─────────────────►│ Google.com                │
┌────────┼──────────────────────┼──────────┐       └───────────────────────────┘
│        │              ┌───────┼────────┐ │                                    
│        └─────────────►│Egress Gateway  │ │                                    
│                       └────────────────┘ │                                    
│                                          │                                    
└──────────────────────────────────────────┘                                    
                      Node-02 192.168.0.23

Egress with kube-vip 🐙

A few years ago I popped into the #sig-networking channel (excellent place to discuss ideas btw) to discuss egress within Kubernetes as I wanted to implement it within kube-vip. There are a few projects that have implemented through different mechanisms in the past, but largely if you wanted to implement it with a stable architecture the obvious solution was using a gateway (as discussed above). After fiddling around with the rules that kube-proxy implements I decided that perhaps there was an alternative solution!

The kube-vip project is everyones (some peoples) goto project when wanting to deploy a load balancer solution within a Kubernetes cluster, and it is pretty devastatingly simple. Apply a load balancer address to a node, and then when traffic is sent to it the traffic then makes its way to a pod via kube-proxy (easy)!

┌─────────┐                                                                 
│Client🧑‍💻 ┼────────────────────► 192.168.0.100───────►Node-01 192.168.0.22  
└─────────┘                     ┌──────▲──────────────────┬────────────────┐
                                │      │                  │                │
                                │┌─────┼──────┐         ┌─▼──────────────┐ │
                                ││kube-vip🐙  │         │Pod-01 10.0.0.3 │ │
                                │└────────────┘         └────────────────┘ │
                                └──────────────────────────────────────────┘

The same, but backwards

For Egress could we potentially combine some of the behaviour of both kube-proxy and kube-vip to produce a stable Egress solution? It turns out yes, we can and again it is devastatingly simple. To accomplish this we effectively overload the behaviour of a Kubernetes service by using it to define not only the ingress (as a loadbalancer) but also the egress with additional iptables rules. This creates a 1:1 relationship between a service and a pod and uses the service IP as the pods ingress and egress address.

┌────────────┐                                                                    
│Server      │◄──────────────────────┐                                            
└────────────┘                       │192.168.0.100           Node-01 192.168.0.22
                                      ┌──▲────▲──────────────────────────────────┐
                                      │  │    │               ┌──────────┐       │
                                      │  │    └───────────────┤Kernel 🧠 │◄─┐    │
                                      │  │                    │          │  │    │
                                      │  │        ┌──────────►│(iptables)│  │    │
                                      │  │        │           └──────────┘  │    │
                                      │┌─┼────────┼─┐         ┌─────────────┼──┐ │
                                      ││kube-vip 🐙 │         │Pod-01 10.0.0.3 │ │
                                      │└────────────┘         └────────────────┘ │
                                      └──────────────────────────────────────────┘

If we look the above diagram we’ve created a service that selects Pod-01 and the following will happen:

The service load balancer is attached to the node where Pod-01 currently resides
The Kernel is updated so that traffic leaving from Pod-01 should be updated so that it looks like it is coming from the load balancer, so effectively 10.0.0.3 is re-written to become 192.168.0.100.
As the process sends traffic or initiates connections to the Server they are now all coming from the externally facing load balancer address.

In the event Pod-01 is killed or crashes then kube-vip is notified through a Kubernetes watcher, where the event of a pod delete will result in the egress rules being cleaned up. When the Pod reappears on another node, then the egress rules will be re-applied and the traffic will continue to be appearing to come from the same IP address, so we get a level of High Availability for free.

Where is the eBPF 🐝

I’ve wanted to re-implement this functionality in eBPF for quite a long time, however it simply isn’t as easy as I hoped.

“But isn’t there an egress hook for TC?”

Indeed there is, however this hook is the last step in the chain before the packet goes back onto the nic and leaves the machine. What this effectively means is that all of the netfilter (iptables & kube-proxy) magic will have already modified the packet before our eBPF program can see it.

So for now the plan is to migrate kube-vip to use googles nftables GO library and we’ll see what happens next.

Thanks

Lars Ekman
Antonio Ojea
The other folks that listened to my daft ideas in #SIG-NETWORK

Highly Available workloads across Mulitple Kubernetes Clusters

Sat, 11 May 2024 09:19:58 +0100

Our Journey begins in 2003 were I somehow blagged a Unix role, largely due to experience with FreeBSD and sparse access to a few Unix systems over the years. This first role consisted of four days and twelve hours per shift, where we were expected to watch for alerts and fix a myriad of different systems ranging from:

Every alert in HP Openview (which you can see in the screenshot below) would lead down a rabbit hole of navigating various multi-homed jump boxes in order to track down the host that had generated the alert.

Once we finally logged onto the machine we were often presented with a obscure version of Unix, a completely random shell or a user-land set of tooling that was incomprehensible.. it was ace.

Typically in this environment the architecture was 1:1, as in one server would host one big application. Although there were some applications that had strict demands and in some cases penalties in the event of downtime, it was these applications that would often make use of various clustering technologies of the time in order to provide high availability. Given the infrastructure was a complete hodgepodge of varying systems it would stand to reason that the clustering software would follow suit, this meant that we were presented with systems such as:

High Availability

As mentioned above in this place of work some applications simply weren’t allowed to fail as there would be a penalty (typically per minute of downtime charges) so for these applications a highly available solution is required. These solutions are to keep an application as available to end users as possible, so in the event an application crashes then it’s the clustering softwares job to restart it. Although you could create your own with a bash script:

#!/bin/bash
while true
do
  echo "Starting program"
  /usr/bin/exciting_program
  sleep 1
done

So restarting the program to ensure availability is one thing, but what about things such as OS upgrades or hardware failures? In those use-cases then the application will cease to run as the system itself will be unavailable. This is where multiple systems are clustered together in order to provide a highly available solution to any downtime, when a system becomes unavailable then it’s the clustering softwares role to make a decision about how and where to restart the application. A lot of my time was fighting with Sun Cluster and how it would implement high availability 🫠

Implementing High Availability (back in the day)

In these systems there were a number of pre-requisites in order for high availability to work

Shared storage

If the application had persistent data, and they pretty much all were based upon Oracle databases back then then this underlying storage needed to be shared. This is so that in the event of a failure the storage can be mounted on the node that is selected to take over the workload.

Re-architecting the application

This doesn’t technically mean re-writing the application, it means writing various startup and shutdown scripts with logic in them in order to ensure that the clustering software can successfully complete them without them ending in a failed state.

Networking

If the application was moving around during a failover then external programs or end users still needed to access it with the same address/hostname etc.. so in the event of a failover a virtual IP (VIP) and hostname will typically be added to the node where the application is failing over to.

Quorum

In order for a node to become the chosen one in the cluster a quorum device should be present in order to ensure that a decision can be made about who will takeover the running of applications, and in the case of a network connection failure between nodes that a “split brain” scenario can’t occur.

Split Brain will occur when cluster nodes can no longer communicate, leading them to believe that they’re the only nodes in the cluster and thus voting for themselves in order to become the leader. This would lead to multiple nodes all accessing the same data or advertising conflicting IP addresses, and generally causing chaos.

Did HA work?

Sometimes

One evening I was asked to lead a large planned change to upgrade the hardware on a mighty Sun e10K that was being used to host arguably one of the most important applications within the entire company. The plan was relatively straight forward:

Log into node1, check it was primary and running the application.
Log into node2, ensure its health (hardware/disk space etc..)
Initiate a failover on node1
Watch and ensure services came up on node2 and validate the application was healthy and externally accessible
Upgrade hardware on node1
Check node1 is healthy again (after upgrade)
Initiate a failover on node2, back to node1
Realise something is wrong
Panic
Really panic
Consider fleeing to a far flung country

So what went wrong?

The application failover from node1->node2 went fine, we saw the application shutdown followed by the shared storage being unmounted and finally the network information removed from node1. Then on node2 we witnessed the storage being mounted, the networking details being applied followed by the application being restarted. We even had the applications/database people log in and watch all the logs to ensure that everything came up correctly.

When we failed back things went very wrong, the first application/storage/network all moved back however the second application stopped and everything just hung. Eventually the process excited with an error about the storage being half remounted. The app/database people jumped onto the second node to see what was happening with the first application whilst we tried to work out what was happening. Eventually we tried to bring everything back to node2 where everything was last running successfully and again the application stopped and the process timed out about the storage 🤯

At this point we had a broken application spread across two machines trying to head in opposite directions but stuck in a failed state, at this point various incident teams were being woken up and various people started prodding and poking things to fix it.. this went on for a few hours before we worked out what was going on. (Spoiler it was step 4)

So this change was happening in the middle of the night, meaning that ideally no-one should really be using it or noticing it not working for the “momentary” downtime. One of the applications team was had opened a terminal and had changed directory to where the application logs where (on the shared storage) in order to watch and make sure the application came up correctly. However, this person then went to watch TV or get a nap (who knows) leaving their session logged on and living within the directory on the shared storage. When it came to failing the application back the system refused to unmount the shared storage as something was still accessing the filesystem 🤬 .. even better when we tried to bring the other half of the application back it failed because someone was looking at the database logs when it attempted to unmount the shared storage for that 🫠

I think it was this somewhat stressful experience that weirdly made me want to learn more about Sun cluster and other HA solutions, and here we are today :-D

High Availability in Kubernetes

Much like that script I embedded above, HA is endlessly performed by Kubernetes typically referred to as the “reconciliation loop”. The reconciliation loops role is largely to compare expected state and actual state and reconcile the difference, so expected state if 3 pods and there is only 1 then schedule 2 more etc. Additionally within the Kubernetes cluster (actually it comes from etcd but 🤷🏼‍♂️) is the concept of leader election, which allows things running within the cluster to use this mechanism to elect a leader amongst all participants. This mechanism allows you to have multiple copies of an application running and with a bit of simple logic ensure that only the active/leader instance is the one that is actually doing the processing or accepting connections etc.

High Availability across Kubernetes

Perfecting Protocol Parsing (Probably) with eBPF

Mon, 15 Jan 2024 20:18:40 +0000

I recently had a little bit of time to kill and decided to see if I could actually do some parsing of other protocols with eBPF. The previous post that I created was about http and whilst it’s an important application protocol to be able to read and potentially manipulate, it feels like there was only so much that could be done. Webpages are highly dynamic and can contain large amounts of data, which are qualities that aren’t always the best to try and parse with eBPF.

So my next attempt was to see how difficult it would be in order to parse something a bit spicier 🌶️! So I recently wrote a basic parsed for BGP messages, which originally was designed to just parse the first bit of data to understand the different message types and give some insight into what was occurring when BGP peers are sending info back and forth, it evolved over the weekend a little bit and now understands peering information and before I decided to write this can now manipulate the data between peers (without the BGP software being aware).

Code is available here

So to begin we will need to do what we always do when we have some network data (the socket buffer skb) in eBPF, which is to check it’s HTTP->TCP/UDP and strip off the headers once we are looking at the correct traffic. This is covered in the previous two eBPF posts, and is in all of the example code so I wont ~~duplicate~~ triplicate the code here. With all of these headers removed (I say removed, we just move the pointer (bit like the needle on a record player) past them so we’re now left with the data portion remaining). With our raw data remaining we now need to convert this into a format that matches the protocol itself, so lets start there!

Protocols

A lot of these protocols are pretty old, and are detailed in documents called Request for Comments or an rfc. These documents put together by experts in the field largely define the architecture of a protocol and a good example, which I used in order to parse HTTP is this one and you can see that this was originally authored in 1999.

So lets get to the crux of it, if you’ve been working with JSON/YAML/XML etc. or anything else that is obviously structured then abandon hope all yee who enter 😂 Almost every protocol has it’s own unique way of how it structures data, some are cleaner than others. To begin with BGP seemed pretty straight forward…

To begin with we’ll need to use the rfc document for the BGP standards, quickly reading through this we can understand that every BGP message starts with the same “fixed size” header:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           Marker                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Length               |      Type     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

(the diagrams in rfcs are a tad confusing, however the descriptions are a bit clearer)

Simply put the marker should be 16 octets! (aka 16 bytes), the length should be 2 octets (2 bytes or 16 bits) and the type is 1 octet (1 byte or 8 bits) and with this information we can create a structure to put the raw data in that will allow us to shape it into the BGP Message header.

struct bgp_message {
	__u8 marker[16];
	__u16 length;
	__u8 type;
};

The joys of padding

If we look at our struct above we can see marker is 16 bytes, length is 2 and type is 1 giving us a grand total of (drum roll 🥁) … 19 bytes. So why oh why, when we do a sizeof(bgp_message) do we end up with 20 bytes 🤯 This was specifically an issue with the BGP keep alive messages that consist of just a BGP message (Type set to 4), where I would attempt to read the BGP message header (expecting it to be 19 bytes) and the compiler was trying to read 20, which was obviously 1 byte too many causing the load_bytes function to fail.

So after some annoying failed attempts to copy 20 bytes our of 19, I realised that my bpg_message struct is probably being padded, this is process of adding some additional data making it more efficient for the CPU to load and store the data. More detail is available here, in most cases it’s not a problem, however we need everything to align perfectly so set packing per byte we can add the following #pragma pack(1) (which effectively disables packing). Now our struct is the correct size and we have will be able to retrieve data from the skb without causing any errors.

Getting data from the `skb`

So we should have a variable that points to the location in the skb where the data lives, this after the frame/IP/TCP headers have ended in my code it’s usually poffset. We will create a variable called bgpm that will now populate with the bytes from the skb using the bpf_skb_load_bytes function.

struct bgp_message bgpm;
int ret = bpf_skb_load_bytes(skb, poffset, &bgpm, sizeof(bgpm));
if (ret != 0) {
	bpf_printk("error %d",ret); // if we can't load the data print the error message
	return 0;
}

(We can see that the sizeof(bgpm), with the padding enabled was causing this to fail as there were only 19 bytes left in the skb and we were trying to load 20 🙄)

Once we have the header, we need to move our poffset so that we point to whatever exists after the header

1	poffset += sizeof(bgpm); // remove header

Understanding application data

We have successfully parsed the header, so we can now use this information to start to understand what the additional data remaining is and with BGP the message type and the length of the remaining data are key. The bgp.type will be one of the following values:

1 - OPEN
2 - UPDATE
3 - NOTIFICATION
4 - KEEPALIVE

Where the bgp.length will represent how much data exists (including the header), so to determine how much “remaining” data is left we would remaining = bgpm.len - sizeof(bgpm) given a KEEPALIVE message is just the header, this should return 0. However other message types often come with additional data!

So lets parse the header, and we’ll look at the UPDATE message in further detail!

#define BGP_OPEN		    1
#define BGP_UPDATE		    2
#define BGP_NOTIFICATION	3
#define BGP_KEEPALIVE		4

...

switch (bgpm.type)
{
	case BGP_OPEN:
	case BGP_UPDATE:
	// Parse the UPDATE data :-)
	case BGP_NOTIFICATION:
	case BGP_KEEPALIVE:
	default:
}

(As every message comes through the kernel we parse the header and then process the remainder of the data)

The UPDATE message is (personally) pretty bonkers:

+-----------------------------------------------------+
|   Withdrawn Routes Length (2 octets)                |
+-----------------------------------------------------+
|   Withdrawn Routes (variable)                       |
+-----------------------------------------------------+
|   Total Path Attribute Length (2 octets)            |
+-----------------------------------------------------+
|   Path Attributes (variable)                        |
+-----------------------------------------------------+
|   Network Layer Reachability Information (variable) |
+-----------------------------------------------------+

As mentioned, if you’ve been writing/parsing JSON or higher level data structures then arrays etc. are pretty simplistic. With these older structures we will need to do various bits of logic to determine how many pieces of information are marked as variable.

The Withdrawn Routes is straight forward enough, the Total Path Attributes is mind boggling…

Without screaming into the void too much, we’re given {x} amount of bytes as the Path Attributes and we would need to do the following:

Read the first 3 bytes to get the flags/type/len.
Then dependant on the type read another random sized number of bytes, as each path attribute contains a different amount of data
We can load that len into the another specific path attribute struct and read that particular data
Move the data pointer forward the size of the Path Attribute “header” and the length of the remaining data len.
Once we’ve done all that move the poffset the size of the Total Path Attributes Length so we can read the NLRI data
Sip a large glass of whisky

Modify the BGP data

So whilst everything detailed above is great for gaining insight into what is happening from a BGP perspective, perhaps we may want to impose some changes to the BGP data as it’s flowing through! For this example we will change the AS number of a new route as it’s being pushed out to a ToR switch. In order to do this we will need to look for the Path Attribute with the type of 2 known as the AS_PATH detailed here.

struct bgp_path_as {
	__u8 type;
	__u8 lenth;
	__u32 as; 
};

(Here is the format defined as a C struct)

At this point we’ve gone through through each of the Path Attributes found type 2/AS_PATH and pulled it from the skb, and we want to change it to a different AS number.

bgp_as.as = bpf_htonl(65002);
ret = bpf_skb_store_bytes(skb, pathOffset, &bgp_as, sizeof(bgp_as), BPF_F_RECOMPUTE_CSUM);
if (ret != 0) {
    bpf_printk("error %d",ret);
    return 0;
}

*(NOTE: pathOffset points to after the path header of the AS_PATH entry exists)

Here we can use the bpf_skb_store_bytes to write an updated bpg_as that has our changed AS number, this helper also has the flag BPF_F_RECOMPUTE_CSUM that takes care of fixing any checksum changes due to the changed underlying data.

NOTE: You should notice that where we’re assigning the new AS 65002 we’re wrapping it with the function bpf_htonl, which is effectively changing a host to network long. Simply put numbers that are used for networking use a different “endian” (the number are stored backwards), you can read more about that here.

The user land BGP program that is peering to the ToR is blissfully unaware that the route it is advertising is using a different AS number 😂 at this point.

Outro

The RFC docs are a great way to begin to understand what this seemingly opaque block of data that proceeds the various headers when processing network data with eBPF. The lack of unbounded loops and some other freely expressible way of manipulating data mean that extra thought has to be given when looking and parsing application data. But with a thoughtful approach I don’t see why most protocols can’t be processed by eBPF, today we need to bind programs to TC (Traffic Control) but once XDP has egress support we can offload so much application processing that the network layer will become incredibly powerful. I’m exciting to parse more protocols :-) (DNS next).

Application traffic with eBPF

Fri, 08 Dec 2023 18:01:41 +0000

In a previous post I talked a little bit about building up the knowledge with eBPF to start to understand a little bit more about what is going in and out of a network adapter. Basically taking your ethernet frame and stripping off the headers (Ethernet + IP Header + TCP/UDP Header) you are finally left with what remains within the packet from an application or data sense.

All of the code lives within the “learning eBPF” repository, specifically the eBPF code is here. The plan for this post is to step through the bits that I think are useful or could be important…

Note This code did do some Ingress/Egress packet modification so uses some eBPF helpers that requires 6.1+ of the Linux Kernel to work.

The maps!

Presumably you’ve come across these before? If not never fear!! Simply put an eBPF map is the mechanism for communicating between user-land and the in-kernel eBPF program. What is exceptionally cool (in my mind at least) is that these maps use keys and values.. so I don’t have to loop around data comparing and looking for what matches whatver it is i’m looking for, I pass a key and if something matches I get the corresponding data :D

Below is the map that I will use, which is called url_map the key is 20 characters long (a bounded “string” some might say), and the value that is assigned to that key is a struct that i’ve defined above.

// Defines a different URL associated with a key
struct url_path {
  __u8 path_len;
  __u8 path[max_path_len]; // This should be a char but code generation between here and Go..
};

// Defines my URL map
struct {
  __uint(type, BPF_MAP_TYPE_HASH);
  __uint(max_entries, 1024);
  __type(key, char[max_path_len]);
  __type(value, struct url_path);
}
url_map SEC(".maps");

The eBPF programs!

There are two eBPF programs defined in the code tc_egress and tc_ingress, bonus points if you can guess how they are attached! For this post, we will only concern ourselves with the tc_ingress program.

So as we would see in the myriad of examples that already exist we need to go through the header identification dance.

Do the sanity checks, and cast the data to the type of ethhdr (Ethernet header)
Find the protocol within the ethernet frame by reading the h_proto within the ethernet header (also called Ethertype).
Cast the data after the ethernet header as a iphdr (IP header)
Find the protocol within the IP Header, we also will need to determine the size of the IP header (turns out they can be different sizes! ¯\_(ツ)_/¯)
To determine the size of the header we times it’s value by four, why I hear you ask! Well this value is multiplied by 32bits do determine the size of the header, so if the value was 6 then the header would be 192 bits (or 24 bytes). So to simply determine the IP header in bytes we can multiply this value by 4!
Cast the data *after the IP Header as a tcphdr (TCP Header)
Like step (5) we will need to determine the size of the TCP Header (it again can be dynamic) and it’s the same step here, we simply need to multiply the value doff by four to determine the header size in bytes.
With all of this calculated we can now infer that the data lives at the end of the Ethernet Header size, the IP Header size and the TCP Header size.
Finally we can determine how big the application data is by taking the tot_len (total length) from the IP Header and subtracting the IP and TCP Header sizes.

Application Data !!

In order to read this data we will need a few things that were mentioned above!

First, we will need the data offset (where the data starts) and that is found after the Ethernet header + the IP Header size (once calculated) and the TCP Header (again, once calculated). We will also need a buffer in order to store the data we will be reading from the socket buffer.

// A data buffer to store our application data
char pdata[60];

// Calculate the offset to where our data actually lives
poffset = ETH_HLEN + ip_hlen + tcp_hlen;


// Load data from the socket buffer, poffset starts at the end of the TCP Header
int ret = bpf_skb_load_bytes(skb, poffset, pdata, 60);
if (ret != 0) {
   return 0;
}

We use the bpf_skb_load_bytes to read the a set amount of data (60 bytes) into our buffer (pdata) from the socket buffer (skb) starting from the offset where we know the data is (poffset)!

At this point we have 60 bytes of data, should be enough for us to write some code to understand it.

HTTP Data :-)

Lets look at what happens when we try a HTTP request!

 ~ curl code/test -vvv
*   Trying 192.168.0.22:80...
* Connected to code (192.168.0.22) port 80 (#0)
> GET /test HTTP/1.1
> Host: code
> User-Agent: curl/7.87.0
> Accept: */*

...

I’m using curl to request the URL /test from the host code (code is my development VM, that runs code-server). We can see the data that is sent to the server (each line begins with > to determine the direction of communication). The first line of data in a HTTP request is typically a verb followed by the resource we would like to interact with and this request ends with the HTTP specification and a carriage return as defined in the HTTP standards. So we can see the line that we care about is GET /test (we/I don’t really care about the HTTP specification at this point :D).

Find the HTTP method

The first step is to read the first three characters of pdata and make find if pdata[0] == G, pdata[1] == E and pdata[2] == T this will effectively allow us to find if this is both a HTTP request in the first place and specifically if it is a HTTP request!

Once we’ve validated those first 3 bytes we will want to read the more data starting from the 4 byte (three bytes for the request and one for the space between)!

char path[max_path_len];
memset(&path, 0, sizeof(path));

int path_len = 0;

// Find the request URI (starts at offset 4), ends with a space
for (int i = 4; i < sizeof(pdata) ; i++)
{
    if (pdata[i] != ' ') {
        path[i-4] = pdata[i];
    } else {
        path[i-4] = '\0';
        path_len = i-4;
        break;
    }
}

The above function will read through the rest of the HTTP data (from the 4th byte) until it encounters a space, leaving us with the URL we are trying to GET! We can validate this with a debug print statement:

1	bpf_printk("<- incoming path [%s], length [%d]", path, path_len);

Which will look like the following in your logs:

1	<idle>-0 [001] dNs3. 2252901.017812: bpf_trace_printk: <- incoming path [/test], length [5]

Acting on the HTTP application request

The above explanations detail what and how we’re reading the data, but if we want to “dynamically” look up the HTTP requests we will need to make use of eBPF maps.

In our GO userland code we do the following:

path := flag.String("path", "", "The URL Path to watch for")
flag.Parse()

// ... 

// Create a uint8 array
var urlPath [20]uint8
// copy our bytes into the uint8 array (we can cast)
copy(urlPath[:], *path)

// place our urlPath as the key
err = objs.UrlMap.Put(urlPath,
  bpfUrlPath{
    Path:    urlPath,
    PathLen: uint8(len(urlPath)),
  })
if err != nil {
  panic(err)
}

As we can see in the code above our Go program when started will read from the flag -path and that will be used as a key in our eBPF map, the value can be ignored for now.

struct url_path *found_path = bpf_map_lookup_elem(&url_map, path);
if (found_path > 0) {
    bpf_printk("Looks like we've found your path [%s]", path);
    // perhaps do more, block traffic or redirect?
}

In our eBPF program we will do a map lookup on the HTTP request, if that request as a char array exists as a key then we can operate on it!

Starting our Go program now sudo ./http -interface ens160 -path /test will yield the following:

INFO[0000] Starting 🐝 the eBPF HTTP watcher, on interface [ens160] for path [/test] 
INFO[0000] Loaded TC QDisc                              
INFO[0000] Press Ctrl-C to exit and remove the program  
          <idle>-0       [001] d.s3. 2252901.015575: bpf_trace_printk: <- 0.0.0.0:56345 -> 0.0.0.0:80
          <idle>-0       [001] D.s3. 2252901.015642: bpf_trace_printk: -> 192.168.0.22:80 -> 192.168.0.180:56345
          <idle>-0       [001] d.s3. 2252901.017552: bpf_trace_printk: <- 0.0.0.0:56345 -> 0.0.0.0:80
          <idle>-0       [001] d.s3. 2252901.017793: bpf_trace_printk: <- 0.0.0.0:56345 -> 0.0.0.0:80
          <idle>-0       [001] dNs3. 2252901.017812: bpf_trace_printk: <- incoming path [/test], length [5]
          <idle>-0       [001] dNs3. 2252901.017814: bpf_trace_printk: Looks like we've found your path [/test]

Conclusion

Parsing HTTP isn’t too bad as it is a relatively simple protocol, it uses easy verbs and simple methods for structure with spaces and carriage returns to differentiate. This methodology would potentially work OK with other protocols like DNS, POP3 or SMTP. When things are encrypted we would need someway of decrypting before we can parse the data (that’s beyond me…). However, I hope that this sparks some ideas into playing more with eBPF and attempting to parse and operate on applications with eBPF!

eBPF adventures in networking

Sat, 18 Nov 2023 17:50:16 +0000

I’ve been wanting to write some hopefully useful posts around eBPF for sometime, although usually by the time I’ve come up with something I though may be useful someone has already beaten me to the punch. Given that I’ve been focussing in networking one way or another for a while, this has largely been the area that I’ve focussed on, although I did manage to put something together for the recent eBPF summit 2023 that I thought was quite fun. As mentioned there are a lot of people that are starting to write eBPF content, so I’ll potentially refer to their posts instead of duplicating content.

`XDP` vs `TC`, or even `sysprobes`

I’ll start with a few acronyms or even technologies in the Linux Kernel that you may or may not have come across. But basically from my perspective at least these are your main options for modifying a running system to interact with networking data.

XDP

There already exists a lot of information about the eXpress Data Plane, so I’ll not delve into too much detail. The tl;dr is that an XDP eBPF program that hooks into XDP will have access to the an incoming network frame before it is processed by the kernel itself. In some cases the eBPF program will be loaded into the NIC driver itself, which will effectively offload the program to the NIC itself.

PROs

The best performance
Excellent for use cases such as firewalls, DDos protection or load balancing
Sees incoming traffic before anything else can make any modifications

CONs

Ingress only, any traffic that you see with an XDP program is only incoming and there is currently no way of seeing traffic that is outbound
Uses the XDP data structure, which is a little different the SKB that is the default for most socket programming.

TC (or Traffic Control)

The Traffic Control is an integral part of the kernel networking structure, largely comprising of the capability of adding things such as qdiscs and filters to an interface. The qdisc largely focuses on providing a TBD and a filter can then be attached to this qdisc, often a filter will actually be an eBPF program under the covers.

A common workflow is:

Create a qdisc or replace an existing one that concerns itself with either ingress or egress. The qdisc is attached to an interface.
Load your eBPF program
Create a filter, that attaches itself to either ingress or egress now exposed through the qdisc on an interface. That filters has the eBPF program attached too it, meaning all traffic either incoming or outgoing will now run through a program (if connected)
Profit 💰

PROs

Provides hooks for ingress and egress
Uses the traditional SKB data structure

CONs

It’s slightly more complicated to attach a TC program to either their ingress or egress queues. The user will need to make use of qdiscs in order to do this, some eBPF SDKs don’t support TC program usage natively.
The traffic a TC eBPF program sees may have already been modified by an earlier XDP program or even the kernel itself.

Syscalls

This might seem a little weird compared to the other two, which are specifically designed in order to handle networking. Whereas an alternative is to attach some eBPF code to a syscall within the kernel, specifically calls such as tcp4_connect() / tcp6_connect(). This is a little bit further down the stack as at this point an incoming packet has already been through a lot of the kernel logic and the eBPF introspection point is as the traffic is about to interact with an application itself.

Programming a network!

So at this point we (hopefully) realise that we’ve a number of different entry points that will allow us to inject our code on the “conveyor belt” that a packet will traverse starting from the NIC all the way to the application (and back, in the case of egress).

Recap

At the beginning of our so called “conveyor belt” we can attach our XDP program and get the raw untouched network data. In the middle of the “conveyor belt” our TC program will become part of the path through the kernel and receive potentially modified network data. At the end of the conveyor belt we can attach code to functions that the application will call in order to get the network data just before it is ingested by the running application.

Data representation

Depending where you attach your program determines two main things, one the relative level of potential modification of traffic and how the traffic is represented.

The XDP struct

I’d write about it but DataDog already have done, you can read that here.

The SKB (Socket buffer)

The SKB is a data type that has existed within the kernel long before eBPF was added to the kernel, and it already comes with a number of helper functions that make interacting with an SKB object a little easier. For more deep dive into SKB you can read this -> http://vger.kernel.org/~davem/skb_data.html

Parsing the data

Regardless of which struct you interact with, they share some commonality and that is largely that there are two variables that are identical across both data types.

These are:

*data, which is a pointer to the data received by the eBPF program
data_len, which is an integer that specified how much data there is (to help make sure you never access *data more than data_len (obvious really 🤓))

So that all seems simple enough, but wait… what is actually in *data?? (Well that is for you to discover)

Well we do that through continually “casting” the *data and moving along it to strip off the various headers in order to understand and find the underlying data!

casting?

You can skip this if you like, but this is a quick (and terrible) example of how we typically take some raw data and turn it into something that makes sense. At the moment *data will just be a stream of random data that won’t make any sense and we will need to effectively add “formatting” too it so that we can understand what it looks like.

Consider the following random line of data Bobby0004500100.503 Harvard Drive90210 some of it makes sense to the raw eye but some of it is unclear.

Imagine the data structure called “person”:

Name: string
Age: number
Balance: float
Street: string
ZipCode: number

If we were to “cast” our random data to the “person” structure above it would suddenly become:

Name: Bobby
Age: 45
Balance: 100.50
Street: 3 Harvard Drive
ZipCode: 90210

Now all of a sudden I’m able to both understand and access the underlying variables in the structure as they now make sense, I.e. person->Name and find out that this particular object of type person has the name variable “Bobby”!

This is exactly what we will do to our *data !

What’s in the data?

So the first step is to determine if the data starts with an Ethernet frame! Pretty much all of the data that travels around starts with an Ethernet frame, which is pretty simplistic but it’s role is to have a source and destination hardware address (regardless of virtualisation/containerisation/cabled network or WiFi). So our first step is to cast our *data to the type ETHHDR, if this is successful we will now be able to understand the variables that make up the Ethernet header data type. These would include the source and destination MAC addresses, but also more importantly the what the contents of the remaining data is. Again, in most circumstances the contents of the *data after the Ethernet header is typically an IP header, but we will validate be checking the Ethernet frames TBD variable.

Once we validate that the next set of data is the IP Header we will need to cast the data after the Ethernet Header to the type IPHDR. Once we do this we will have access to the IP specific data such as source IP (saddr) or destination address (daddr), again importantly the IP header contains a variable that details what the data is after the end of the IP Header. This is usually a TCP header or UDP header, but there are other alternatives such as sctp etc..

Once we’ve looked inside the IPHeader and determined that the data type is TCP (could be UDP or something else), we will cast the data after both the Ethernet header and the IP header to the type TCP header! (Almost there). With access to the contends to the TCP header we have the TCP specific data, such as source port or destination port, the checksum to ensure validity of the data amongst other useful variables.

We now have almost everything, however the TCP header can be variable length so we will need to determine this by looking at the tcl_len variable, which we need to times by 4. We now have everything we need to get to the final data!

So, the *data points to the beginning of the data! We have determined that there is an Ethernet header followed by a IP header and finally a TCP header, which means *data + Ethernet header + IP header + TCP header = Actual application data !

What can we do with this information ?

As we parse through the various headers, we effectively unlock more and more information at different layers of the OSI model!

[layer 2] The Ethernet Header provides us with the source and destination hardware addresses, we could use this information to potentially stop frames being processed from source MAC addresses that we know to be dangerous.

[layer 3] The IP Header contains the source and destination IP addresses, again we can act like a firewall by having an eBPF program drop all traffic for a specific IP address. Alternatively we could have logic that will potentially redirect traffic based upon the IP addresses of we could even implement load balancing logic at this layer that will redirect to an underlying set of other IP addresses

[layer 4] The TCP or UDP Headers define the destination port numbers, which we can use to determine what the application protocol is (I.e. port 80 typically means that the remaining *data is likely to be HTTP data). More often than not we would perform actions such as load balancing at this layer, based upon the destination (I.e. balance across multiple other load balancer addresses)

[layer 7] As mentioned the data at the end of the collection of various headers is the actual application data, which we can also parse (as long as we know the format). So for instance if an external web browser were to try and access /index.html on my machine with an eBPF program attached, I’d parse all the way to TCP to determine that it was port 80 and then the application data should be in the HTTP format. I could validate this by looking at the first three characters of application data (after all the headers), with some pseudo code like below:

ApplicationData = EthernetHDR + IPHDR + TCPHDR // Add all headers lengths together to find the data
If ( data[ApplicationData] = "G" && data[ApplicationData+1] = "E" && data[ApplicationData+2] = "T" ) {
	// It's a HTTP GET request 
	// do something exciting
}

Wrap up

Now we “kind of” understand the logic we should probably look at implementing some code to do all this .. that’s for another day though.

Finding and fiddling with Slacks APIs

Wed, 18 Oct 2023 11:41:53 +0100

It’s starting to feel a little bit as though the noose is starting to tighten a lot in the IT industry at the moment, whether it be Open Source projects spuriously changing their licenses and pulling the rug from their users to companies that expanded too far and too quickly during the pandemic suddenly contracting. This tightening of the belt has also impacted a bunch of tooling that people have come to depend on as part of their day to day life, or their workflow.

Slack isn’t slack when it comes to the community usage

So! Whats this rambling collection of words about?

Like all of the wonderful platforms as a service out there ~~IRC for money~~ Slack has the various tiers, each unlocking more and more functionality whilst ramping up the various costs associated. A lot of communities (especially in the Open Source world) have been built on the free and open communication that takes place on a Slack instance/workspace devoted to a project or community. But the problem i’m trying to address in this post is the hoops people need to jump though in order to join these communities hosted on Slack.

When the community first starts out, it makes perfect sense that could be a need for simple restraints and/or approval process for joining a community. This is usually handled through an invitation process, if you want to join the community then you need to somehow signal that intent allowing the owner(s) of the community to then invite you to join. However once you start to hit any sort of scale then this simply becomes an unmanageable task, ultimately impacting the growth and health of a hopefully growing community and it is at this point where you would potentially want to open the floodgates as the community (hopefully) exponentially grows.

Note On the side of security, we’re moving to a reactive instead of proactive approach for maintaining community membership (again coming with it’s own challenges).

So what are your options?

The legitimate option

You’re main options without trying to automate anything are to simply select the + Add coworkers button and create an invite link, this can be set to never expire and is meant to work for 400 people (YMMV). It doesn’t really look like there are many options available to automate this procedure, you can open Slack in a web browser with “developer mode” enabled to capture the API endpoint but i’ve been unable to determine what token can be used.

The endpoint for this API call is https://<workspace>.slack.com/api/users.admin.createSharedInvite, which should be posted with a FORM with fields such as token, set_active however no token I could produce would seem to get this to work (even taken the token used in the browser).

Example code:

token := os.Getenv("SLACK_TOKEN")
instance := os.Getenv("SLACK_INSTANCE")

slack_url := fmt.Sprintf("https://%s/api/users.admin.createSharedInvite", instance)

values := url.Values{
	"expiration":  {"99999"},
	"token":       {token},
	"set_active":  {"true"},
	"max_signups": {"200"},
}
client := &http.Client{}

req, err := http.NewRequest("POST", slack_url, strings.NewReader(values.Encode()))
if err != nil {
	fmt.Println(err)
	return
}
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")

resp, err := client.Do(req)
if err != nil {
	fmt.Println(err)
	return
}
if err != nil {
	log.Fatal(err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
	log.Fatal(err)
}
fmt.Println(string(body[:]))

The “other” option

With trying to replicate the Slack client behaviour not working, we’re left to trying less legitimate routes to make it easier for people to join the Slack workspace. So there are a number of Open Source projects that are quite old and unmaintained that make use of an old API (undocumented) endpoint /api/users.admin.invite (not to be confused with /api/admin.users.invite, api docs). This endpoint works with a legacy token that you’re no longer able to create (as of 2020).. it turns out invoking the correct sequence will allow you to create a token that can still use this endpoint however!

Create an App https://api.slack.com/apps
Create the App from scratch, give it a name (any will do)
Scroll down and make a copy the Client ID and the Client Secret
Scroll further up and find the Permissions box
In the Redirect URLs box we will need to enter a bogus URL (make sure it doesn’t actually resolve) such as https://0.0.0.0/, ensure you click Save URLs
Under the User Token Scopes, select the Add an OAuth Scope and add admin
Finally scroll back up to OAuth Tokens for Your Workspace and select Install to Workspace.
Select the Allow button to create our first token!

At this point we will have a User OAuth Token, however this token even with it’s permissions wont work with the users.admin.invite endpoint 🙄

To generate our actual token we need to do some additional oauth steps!

Build our specific url! https://<workspace>.slack.com/oauth?client_id=<Client ID>&scope=admin%2Cclient
Replace the <workspace> with the name of your particular workspace, and change <Client ID> to the ID from your application.
Paste this URL into a new tab on the logged in browser, you’ll be redirected to a webpage that won’t work (remember the 0.0.0.0). However you should see the new URL in the address bar! (Including a code= part of the URL). Grab this code for the next part!
Build our new URL! https://<workspace>.slack.com/api/oauth.access?client_id=<Client ID>&client_secret=<Client Secret>&code=<code>
Either open this in a web browser again, or in curl (remember to put the URL in quotes ")
The output from this will be some json {"ok":true,"access_token":"xoxp-..., the access_token is what we’re after !!
Verify that you can invite people with your new token!

1
2
3

curl -X POST 'https://<workspace>.slack.com/api/users.admin.invite' \
--data 'email=dan@thebsdbox.co.uk&token=<token>&set_active=true' \
--compressed

A few seconds later…

1	{"ok":true}

Just to test, you can run the invite command again and if it’s all working as expected you should receive an error!

1	{"ok":false,"error":"already_in_team_invited_user"}

In conclusion

That seems like a lot of effort, but in the end it’s now possible again to build a simple web form allowing people to register themselves to join your Slack workspace!

The Hackintoshes over the years

Sat, 31 Dec 2022 10:46:07 +0000

Here we are again.. do I bother.. I guess I did

The Why?

As much as I love a nice shiny laptop, i’ve been using a steadily evolving desktop machine for well over a decade. In fact I don’t think i’ve ever been without a workstation since I was first introduced to a computer:

Amstrad CPC 464
Amstrad 1512
Intel DX4 (100Mhz)
P200 (With MMX), had a snappy advert back in the day
P3 came in a cool cartridge
P4 something or other ..
Intel core quad something
i7 Haswell (todays machine)

In hindsight I realise that my current set up is actually pretty aged, but either I haven’t noticed or don’t care enough to upgrade. I also feel like i’ve been through a lot of Operating systems through my time:

MS-DOS 2/3/6.22
Gem.. not sure this counts
Windows 3.11
Windows 95 .. on 1,000,000 floppy disks
Windows 98/ME, I don’t want to talk about it
Windows 2000, the pinacle of Windows
Windows XP
BeOS, some random gave me a CD in a bookshop at UNI
Slackware, such a miserable experience I still don’t run a Linux desktop

At some point at University I started a part time job as a Macintosh repair/support person, which I used to hold the belief was my worst work experience. However, this exposed me to the world of Apple hardware and the symbiotic operating system. Well.. System7, MacOS 8 and MacOS 9 were pretty grim but some of the hardware 🤩. I remember one day someone bringing in a 20th Anniversary Mac The product was a complete failure, but look at it !

The G5 tower, such a clean yet imposing piece of metal.. pop the side off and it looked like the inside of a Dyson vacuum cleaner. I also got to work with the Apple rack mount hardware too, arguably the last time there would be anything aesthetically pleasing in a data centre.

Anyway… I was lucky enough to be given a Powerbook G4, it had been repaired and after 6 months the person had never collected it or replied to calls/emails ¯\_(ツ)_/¯ and thus began my switch over to OSX (then MacOS X and now MacOS, clearly naming is hard).

Hackintosh

With a shiny Powerbook in hand I proceeded to move all my daily content (music/photos) and social platform(s).. largely just last.fm, myspace and facebook at that point. My poor old workstation started to be turned on less and less as I started to find it slow and clunky when compared. In the mid 2000s Apple did the unthinkable (or maybe the thinkable, IBM were taking forever for a laptop G5 chip) and pivoted the Apple hardware away from PowerPC to Intel hardware, HOWEVER the OS whilst x86 is still tied to Apple specific hardware.

I honestly have no recollection where I found it (probably a dodgy torrent site), but I came across a OS X 10.4 DVD Image that was hacked to install on non Apple hardware and that was it!

Some looking around on archive.org has revealed that it was JaS OS Hackintosh OS images..

For the next couple of OS X releases all of the underlying work was largely taken care of and things “just worked”.. here I was with the Apple experience but without the additional cost!

Apple Airport on the shelf there, a fantastic Access point back in the day

The Hackintosh’s endured a lot of my terrible skillz at WoW

Additional complexity

At some point the overhead of producing these packaged OS installers clearly wasn’t worth it (presumably Apple were making things additionally difficult) and suddenly a lot of the overhead was past down to the users. At this point Hackintosh users suddenly had to care about things like EFI/Bootloaders/Kexts and a raft of additional tooling 🤯.

To get a working system the following steps usually were required:

Get a USB stick
Create an EFI partition
Install a bootloader (clover was the one, back in the day) and kexts for your system
Get a MacOS X installer and write it to the USB stick
Boot from the USB stick and find that (Display/USB/Network/Disk etc.. didn’t work) go back to step 3
Install MacOS X to internal disk
Boot from USB Stick into Mac OSX on internal disk
Transfer working /EFI to internal disk and install any missing drivers/kernel modules
Hope Apple didn’t release anything new for a while…

Additionally in the middle of all this Nvidia had a falling out with Apple, meaning that Apple switched to AMD and immediately stopped any new drivers for Nvidia cards :-(

This was pretty much par for the course until MacOS Big Sur was released at which point a new boot loader was required (for reasons I’m not bothered to understand at this point)… Whilst this new bootloader allowed for installing these new MacOS releases, it incurs a significant overhead.. largely the managing of a LOT of XML. If by some chance you manage to incant a working /EFI configuration then lucky you, but it’s a slow and steady process involving many many many reboots and kernel panics. Luckily there is a pretty good community (it’s always about a strong community) in the hackintosh space both on reddit and in other places such as TonyMacx86. The community is pretty helpful at inspecting peoples XML and pointing out irregularities and mistakes and getting people on their way.

With a working /EFI I managed to move to OpenCore and finally stand up BigSur and then later on Monteray!

Today

So here we are.. to today (or the day I wrote this)..

Anyway, for “reasons” the /EFI partition is FAT (old Microsoft standard) which isn’t the most reliable during power loss… Inevitably I had a powercut a few days ago and my /EFI partition was wiped, which lost the entire contents of my finely tuned and artisanal XML config. At this point I couldn’t decide if I had the energy to go through this process again. But given the xmas break and prospect of having to speak with the in-laws I decided this might be an opportunity..

The process has improved since I followed the steps for BigSur, in fact the guide for OpenCore is fantastic and the additional drivers for USB/Audio just work™️ etc..

Get the OpenCore bootloader
Add in the ACPI tables for your architecture
Add in drivers that are needed
Fight with XML
Fight with XML more
Fight with XML even more
Profit?

Nope

Yep

My trusty workstation is now back up and running on Ventura (although it doesn’t really seem like an OS for workstation use … )

The Future

The move to Intel allowed hackintosh’s to become a thing, however the recent move to ARM from Apple will probably be the final nail in the coffin in the community. It’s been a good run of running MacOS on my hardware, but it does leave me in a quandary of where to go next when I can no longer run a Hackintosh.

KatNav - A Kubernetes Sat Nav

Wed, 28 Jul 2021 19:32:51 +0100

Create the project with `kubebuilder`

1
2
3

kubebuilder init --domain fnnrn.me --owner "Dan Finneran"

kubebuilder create api --group katnav --kind Directions --version v1

Edit the Resources

The path is api/v1/directions_types.go

Edit the Directions Specification `DirectionsSpec`

The example is the usual Foo etc..

1 2	// Foo is an example field of Directions. Edit directions_types.go to remove/update Foo string `json:"foo,omitempty"`

We will change this to what we need for the directions to actually work:

// Source is where the beginning of our journey is
Source string `json:"source"`

// Destination is the end of our journey
Destination string `json:"destination"`

Edit the Directions Status `DirectionsStatus`

The Status reflects what has happened to our resource, and the expected status will be the directions.

// Directions is a list of directions to our destination
Directions string `json:"directions"`

// Routesummary gives a simple overview of the route
RouteSummary string `json:"routeSummary"`

// StartLocation is the start from the directions API
StartLocation string `json:"startLocation"`

// EndLocation is the start from the directions API
EndLocation string `json:"endLocation"`

// Distance is the total distance of the journey
Distance string `json:"distance"`

// Duration is the amount of time the journey will take
Duration string `json:"duration"`

// Error captures an error message if the route isn't possible
Error string `json:"error,omitempty"`

Configure Google Account

To get an API key:

Visit Google APIs Console and log in with a Google Account.
Select one of your existing projects, or create a new project.
Enable the API(s) you want to use. The Go Client for Google Maps Services accesses the following APIs:

Directions API

Create a new Server key from the credentials menu.

Billing may need to be enabled so do be careful with usage!

Apply API key to secret

This will create our secret (called katnav) within the “default” namespace:

1	kubectl create secret generic katnav --from-literal=directionsKey=<API TOKEN>

Add our code to the controller

The code that we’re going to add to the controller will do a few things:

Connect to the Google Maps API
Watch for directions resources being created within the Kubernetes API
Update these resources (the status) with the directions and additional journey information

Add the Google Maps client

Add "googlemaps.github.io/maps" to the imports and a pointer to the client in the DirectionsReconciler struct:

// DirectionsReconciler reconciles a Directions object
type DirectionsReconciler struct {
	client.Client
	Scheme  *runtime.Scheme
	mClient *maps.Client
}

Initialise the Google Maps client

At the beginning of the SetupWithManager() function we will add in code that will do the following:

Create our own Kubernetes client
Retrieve our secret in the default namespace
Create our maps client and update out pointer to the client so we can use it in the reconcile() function later:


// So we have a Kubernetes cluster in r.Client, however we can't use it until the caches
// start otherwise it will just return an error. So in order to get things ready we will
// use our own client in order to get the key and set up the Google Maps client in advance
config, err := restClient.InClusterConfig()
if err != nil {
	kubeConfig :=
		cmdClient.NewDefaultClientConfigLoadingRules().GetDefaultFilename()
	config, err = cmdClient.BuildConfigFromFlags("", kubeConfig)
	if err != nil {
		return err
	}
}
// create the clientset
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
	panic(err.Error())
}

secret, err := clientset.CoreV1().Secrets("default").Get(context.TODO(), "katnav", v1.GetOptions{})
if err != nil {
	return err
}

token := string(secret.Data["directionsKey"])
if token == "" {
	return fmt.Errorf("no Token found within API Key")
}
r.mClient, err = maps.NewClient(maps.WithAPIKey(token))
if err != nil {
	return err
}

Reconcile !

Below is our full Reconcile() function that does the following:

Retrieves our resource that needs reconciling
Gets the source/destination from the spec
Creates a Google Maps Directions request to the API
Updates the status of the resource with the direction information (that we format, such as removing the HTML tags etc..)
Apply the updated directions Object to the Kubernetes API!

func (r *DirectionsReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	// Set the log to an acutal value so we can create log messages
	log := log.FromContext(ctx)
	log.Info("Reconciling direction resources")

	var directions katnavv1.Directions
	if err := r.Get(ctx, req.NamespacedName, &directions); err != nil {
		if errors.IsNotFound(err) {
			// object not found, could have been deleted after
			// reconcile request, hence don't requeue
			return ctrl.Result{}, nil
		}
		log.Error(err, "unable to fetch Directions object")
		// we'll ignore not-found errors, since they can't be fixed by an immediate
		// requeue (we'll need to wait for a new notification), and we can get them
		// on deleted requests.
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}
	// your logic here
	log.Info("Determining journey", "Source", directions.Spec.Source, "Destination", directions.Spec.Destination)

	request := &maps.DirectionsRequest{
		Origin:      directions.Spec.Source,
		Destination: directions.Spec.Destination,
		Mode:        maps.TravelModeDriving,
	}

	route, _, err := r.mClient.Directions(context.Background(), request)
	if err != nil {
		log.Error(err, "unable to fetch Directions")
	}
	var directionsString string
	if len(route) == 0 {
		return ctrl.Result{}, nil
	}

	log.Info("New Route", "Summary", route[0].Summary)
	for x := range route[0].Legs {
		for y := range route[0].Legs[x].Steps {
			stripped := strip.StripTags(route[0].Legs[x].Steps[y].HTMLInstructions)
			directionsString += stripped + "\n"
		}
		directions.Status.Distance = route[0].Legs[x].Distance.HumanReadable
		directions.Status.Duration = fmt.Sprintf("Total Minutes: %f", route[0].Legs[x].Duration.Minutes())
		directions.Status.StartLocation = route[0].Legs[x].StartAddress
		directions.Status.EndLocation = route[0].Legs[x].EndAddress

	}

	directions.Status.RouteSummary = route[0].Summary
	directions.Status.Directions = directionsString

	err = r.Client.Status().Update(context.TODO(), &directions, &client.UpdateOptions{})
	if err != nil {
		log.Error(err, "unable to update journey")
	}
	return ctrl.Result{}, nil
}

Using !

`Sheffield.yaml`

This is our manifest for directions from York -> Sheffield!

apiVersion: katnav.fnnrn.me/v1
kind: Directions
metadata:
  name: directions-to-sheffield
spec:
  source: York, uk
  destination: Sheffield, uk

We can apply this to Kubernetes with:

1	kubectl apply -f ./sheffield.yaml

We should see the following logs from the Controller:

1
2
3

2021-07-28T15:56:47.901+0100    INFO    controller-runtime.manager.controller.directions        New Route       {"reconciler group": "katnav.fnnrn.me", "reconciler kind": "Directions", "name": "directions-to-sheffield", "namespace": "default", "Summary": "M1"}
2021-07-28T15:56:47.906+0100    INFO    controller-runtime.manager.controller.directions        Reconciling direction resources {"reconciler group": "katnav.fnnrn.me", "reconciler kind": "Directions", "name": "directions-to-sheffield", "namespace": "default"}
2021-07-28T15:56:47.906+0100    INFO    controller-runtime.manager.controller.directions        Determining journey     {"reconciler group": "katnav.fnnrn.me", "reconciler kind": "Directions", "name": "directions-to-sheffield", "namespace": "default", "Source": "York, uk", "Destination": "Sheffield, uk"}

Looking at our Directions 👀

If we examine the newly created directions object we should see that it’s status has been updated accordingly !

$ kubectl get directions directions-to-sheffield -o yaml
apiVersion: katnav.fnnrn.me/v1
kind: Directions
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"katnav.fnnrn.me/v1","kind":"Directions","metadata":{"annotations":{},"name":"directions-to-sheffield","namespace":"default"},"spec":{"destination":"Sheffield, uk","source":"York, uk"}}
  creationTimestamp: "2021-07-28T14:34:56Z"
  generation: 2
  name: directions-to-sheffield
  namespace: default
  resourceVersion: "6986253"
  uid: c4077b0b-8f2f-4fc3-82c0-2d690a24d98a
spec:
  destination: Sheffield, uk
  source: York, uk
status:
  directions: |
    Head southwest on Lendal Bridge/Station Rd/A1036 toward Rougier St/B1227Continue to follow A1036
    Keep left to continue toward Station Rd/A1036
    Continue onto Station Rd/A1036Continue to follow A1036
    Turn right onto Blossom St/A1036Continue to follow A1036
    At the roundabout, take the 2nd exit onto Tadcaster Rd Dringhouses/A1036
    Take the ramp to Leeds
    Merge onto A64
    Merge onto A1(M) via the ramp to Leeds/M1/Manchester/M62
    Keep right at the fork to continue on M1
    At junction 34, take the A6109 exit to Sheffield(E)/Rotherham(C)/Meadowhall
    At Meadowhall Roundabout, take the 4th exit onto Meadowhall Rd/A6109
    Keep right to stay on Meadowhall Rd/A6109Continue to follow A6109
    At the roundabout, take the 1st exit onto Brightside Ln/A6109Continue to follow A6109
    Slight right onto Savile St/A6109
    Turn right onto Derek Dooley Way/A61Continue to follow A61
    Slight left onto Corporation St/A61
    Slight left onto Corporation St/B6539
    At the roundabout, take the 2nd exit onto W Bar Green/B6539Continue to follow B6539
    At the roundabout, take the 3rd exit onto Broad Ln/B6539Continue to follow B6539
    At the roundabout, take the 1st exit onto Upper Hanover St
    Continue onto Hanover Way
    At the roundabout, take the 1st exit onto Moore St
    Continue onto Charter Row
    Continue onto Furnival Gate
    Furnival Gate turns left and becomes Pinstone St
    Turn right onto Burgess St
    Burgess St turns right and becomes Barker's Pool
    Barker's Pool turns left and becomes Leopold St
  distance: 93.8 km
  duration: 'Total Minutes: 77.816667'
  endLocation: Sheffield, UK
  routeSummary: M1
  startLocation: York, UK

The "perfect" virtual Tinkerbell environment on Equinix Metal

Fri, 02 Apr 2021 17:45:02 +0100

pre-requisites

This is a rough shopping list of skills/accounts that will be a benefit for this guide.

Equinix Metal portal account
GO experience (basic)
iptables usage (basic)
qemu usage (basic)

Our Tinkerbell server considerations

Some “finger in the air” mathematics are generally required when selecting an appropriately sized physical host on Equinix metal. But if we take a quick look at the expected requirements:

CONTAINER ID        NAME                   CPU %               MEM USAGE / LIMIT    MEM %               NET I/O             BLOCK I/O           PIDS
582ce8ba4cdf        deploy_tink-cli_1      0.00%               832KiB / 7.79GiB     0.01%               0B / 0B             13.1MB / 0B         1
4aeb11684865        deploy_tink-server_1   0.00%               7.113MiB / 7.79GiB   0.09%               75.7MB / 76.6MB     4.94MB / 0B         11
24c7c5961a2b        deploy_boots_1         0.00%               8.652MiB / 7.79GiB   0.11%               0B / 0B             5.66MB / 0B         7
37d2e840bd54        deploy_registry_1      0.02%               8.344MiB / 7.79GiB   0.10%               0B / 0B             1.32MB / 0B         11
a70a54baff08        deploy_hegel_1         0.00%               2.984MiB / 7.79GiB   0.04%               0B / 0B             246kB / 0B          5
33d953c9b057        deploy_db_1            13.66%              6.363MiB / 7.79GiB   0.08%               24.7MB / 22.2MB     1.5MB / 1.34GB      8
2fa8a6064036        deploy_nginx_1         0.00%               2.867MiB / 7.79GiB   0.04%               341kB / 0B          0B / 0B             3

We can see that the components for the Tinkerbell stack are particularly light, with this in mind we can be very confident that we can have all of our userland components (tinkerbell/docker/bash etc..) within 1GB of ram and leave all remaining memory for the virtual machines.

That brings us onto the next part, which is how big should the virtual machines be?

In memory OS (OSIE)

Every machine that is booted by Tinkerbell will be passed the in-memory Operating System called OSIE which is an alpine based Linux OS that ultimately will run the workflows. As this is in-memory we will need to account for a few things (before we even install our Operating System through a workflow.

OSIE kernel
OSIE RAM Disk (Includes Alpine userland and the docker engine)
Action images (at rest)
Action containers (running)

The OSIE Ram Disk whilst it looks like a normal filesystem is actually held in the memory of the host itself so immediately will withhold that memory from other usage.

The Action image will be pulled locally from a repository and again written to disk, however the disk that these images are written to is a ram disk, so these images will again withhold available memory.

Finally, these images when ran (Action containers) will have binaries in them that will require available memory in order to run.

The majority of this memory usage from the as seen from above is for the in-memory filesystem in order to host the userland tools and the images listed in the workflow. From testing we’ve normally seen that >2GB is required, however if your workflow consists of large action images then this will need adjusting accordingly.

With all this in consideration, it is quite possible to run Tinkerbell on Equinix Metals smallest offering the t1.small.x86, however if you’re looking at deploying multiple machines with tinkerbell then ideally a machine with 32GB of ram will comfortably allow a comfortable amount of headroom.

Recomended instances/OS

Check the inventory of your desired facility, but the recommended instances are below:

c1.small.x86
c3.small.x86
x1.small.x86

For speed of deployment and modernity of the Operating System, either ubuntu 18.04 or ubuntu 20.04 are recommended.

Deploying Tinkerbell on Equinix Metal

In this example I’ll be deploying a c3.small.x86 in the Amsterdamn faclity ams6 with ubuntu 20.04. Once our machine is up and running, we’ll need to install our required packages for running tinkerbell and our virtual machines.

Update the packages

1	apt-get update -y

Install git (to clone the sandbox)

1	apt-get install -y git

Install required dependancies

apt-get install -y 	apt-transport-https \
	ca-certificates \
	curl \
	gnupg-agent \
	software-properties-common \
	qemu-kvm \
	libguestfs-tools \
	libosinfo-bin

Grab shack (qemu wrapper)

1
2
3

wget https://github.com/plunder-app/shack/releases/download/v0.0.0/shack-0.0.0.tar.gz; \
tar -xvzf shack-0.0.0.tar.gz; \
mv shack /usr/local/bin

Create our internal tinkerbell network (not needed)

1	sudo ip link add tinkerbell type bridge

Create shack configuration

1	shack example > shack.yaml

Edit and apply configuration

Change the bridgeName: from plunder to tinkerbell, then run shack network create. This will create a new interface on our tinkerbell bridge

Run shack network create

Test virtual machine creation

shack vm start --id f0cb3c -v
<...>
shack VM configuration
Network Device:	plndrVM-f0cb3c
VM MAC Address:	c0:ff:ee:f0:cb:3c
VM UUID:	f0cb3c
VNC Port:	6671

We can also examine that this has worked, by examining ip addr:

11: plunder: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 2a:27:61:44:d2:07 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global plunder
       valid_lft forever preferred_lft forever
    inet6 fe80::bcc7:caff:fe63:8016/64 scope link 
       valid_lft forever preferred_lft forever
12: plndrVM-f0cb3c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master plunder state UP group default qlen 1000
    link/ether 2a:27:61:44:d2:07 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2827:61ff:fe44:d207/64 scope link 
       valid_lft forever preferred_lft forever

Connect to the VNC port with a client (the random port generated in this example is 6671).. it will be exposed on the public address of our equinix metal host.

Kill the VM:

1	shack vm stop --id f0cb3c -d

Install sandbox dependencies

Docker

1
2
3

curl -fsSL https://download.docker.com/linux/ubuntu/gpg |sudo apt-key add - \ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
sudo apt-get update \
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

Docker compose

sudo curl -L \
	"https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" \
	-o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Clone the sandbox

1 2	git clone https://github.com/tinkerbell/sandbox.git cd sandbox

Configure the sandbox

1 2	./generate-envrc.sh plunder > .env ./setup.sh

Start Tinkerbell

# Add Nginx address to Tinkerbell
sudo ip addr add 192.168.1.2/24 dev plunder
cd deploy
source ../.env; docker-compose up -d

At this point we now have a server with available resource, we can create virtual machines and tinkerbell is listening on the correct internal network!

Create a workflow (debian example)

Clone the debian repository

1
2
3

cd $HOME
git clone https://github.com/fransvanberckel/debian-workflow
cd debian-workflow/debian

Build the debian content

1
2
3

./verify_json_tweaks.sh 
# The JSON syntax is valid
./build_and_push_images.sh

Edit configuration

Modify the create_tink_workflow.sh so that the mac address is c0:ff:ee:f0:cb:3c, this is the mac address we will be using as part of our demonstration.

For using VNC, modify the facility.facility_code from "onprem" to "onprem console=ttys0 vha=normal". This will ensure all output is printed to the VNC window that we connect to.

Create the workflow

Here we will be asked for some password credentials for our new machine:

1	./create_tink_workflow.sh

Start our virtual host to install on!

shack vm start --id f0cb3c -v
<...>
shack VM configuration
Network Device:	plndrVM-f0cb3c
VM MAC Address:	c0:ff:ee:f0:cb:3c
VM UUID:	f0cb3c
VNC Port:	6671

We can now watch the install on the VNC port 6671

Troubleshooting

1	http://192.168.1.1/undionly.pxe could not be found

If a machine boots and has this error it means that it’s workflow has been completed, in order to boot this server

1	could not configure /dev/net/tun (plndrVM-f0cb3c): Device or resource busy

This means that an old qemu session left an old adapter, we can remove it with the command below:

ip link delete plndrVM-f0cb3c

1	Is another process using the image [f0cb3c.qcow2]?

We’ve left an old disk image laying around, we can remove this with rm

Putting a VIP in your Kubernetes Clusters

Fri, 01 Jan 2021 10:46:26 +0000

In this post i’ve a bunch of things I want to cover all about Type:LoadBalancer (or in most cases a VIP (Virtual IP address). In most Kubernetes environments a user will fire in some yaml defining a LoadBalancer service or do a kubectl expose and “magic” will occur. As far as the end-user is concerned their new service will have a brand new IP address attached to it and when an end-user hits that IP address their traffic will hit a pod that is attached to that service. But what’s actually occuring in most cases? Who can provide this address? How does it all hang together?

That’s what we’re going to cover in this post.

The venerable Tim Hockin covers this in a bunch of slidedecks here -> https://speakerdeck.com/thockin/bringing-traffic-into-your-kubernetes-cluster, so feel free to start here and then i’ll cover some bits in more detail or from a different angle :-)

Kubernetes Services

A lot of this is already mentioned above, but put simply a service is a method of providing access to a pod or number of pods either externally or internally. A common example is exposing a web server front end, where we may have a deployment of 10 nginx pods.. we need to allow end users to access these 10 pods. Within Kubernetes we can define a service that is attached to this deployment, and thanks to the logic within Kubernetes we don’t need to concern ourselves too much with these pods.. we can scale up/scale down.. kill pods etc.. as long as we’re coming through the service it will always have an upto date list of the pods underneath it.

Types of service

ClusterIP

This is an internal only service.. typically used for internal communication between two services such as some middlewhere level connectivity.

NodePort

A NodePort is a port created on every node in the cluster. An external user can connect to a node address on this port to access the service. If we were to use a NodePort with nginx then we’d be given a high port (usually 30000+) that will route traffic to the nginx ports e.g. worker0xx:36123 --> [nginx-pod0x:80]

LoadBalancer

The LoadBalancer service is used to allow external access into a service, it usually requires something external (Cloud Controller Manager) to inform the Kubernetes API what external address traffic should be accepted on.

ExternalName

This allows a service to be exposed on an external name to point to something else. The main use-case is being able to define a service name and having it point to an existing external service…

The Kubernetes Load Balancer service

So how does all this hang together…

-> also i’ll be walking through how you can implement this yourself NOT discussing how the big cloud providers do it :-)

If you take a “fresh” Kubernetes cluster and create a simple nginx deployment and try to expose it as a LoadBalancer you’ll find that it doesn’t work (or sits in pending).

1	kubectl expose deployment hello-world --type=LoadBalancer --name=my-service`

[some time later]

1 2	NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-service LoadBalancer 10.3.245.137 <pending> 8080/TCP 54s

Why??? Well the Type:LoadBalancer isn’t the responsibility of kubernetes, it effectively doesn’t come out of the box… as we can see the internal IP (CLUSTER-IP) has been created but no external address exists. This EXTERNAL-IP is typically a piece of information that is infrastructure specific, in the cloud we need addresses to come from their IP address management systems and on-premises who knows how addresses are managed ¯\_(ツ)_/¯

Cloud Controller Manager

The name can be slightly confusing as anyone can write a CCM and their roles aren’t necessarily cloud specific, i presume it’s more that their main use-case is to extend a kubernetes cluster to be aware of a cloud providers functionality.

I’ve covered Cloud Controllers before, but to save the dear reader a few mouse-clicks i’ll cover it briefly again… The role of a CCM is usually to plug into the Kubernetes API and watch and act upon certain resources/objects. For objects of type node then the CCM can speak with the infrastructure to verify that the nodes are deployed correctly and enable them to handle workloads. The objects of type LoadBalancer would require the CCM to speak with an API to request an IP address that is available to be an EXTERNAL-IP, alternatively a CCM may be configured with an IP address range it can use in order to configure the EXTERNAL-IP.

LoadBalancer watcher

Within the CCM it will have code that will be watching the Kubernetes API, specifically for watching Kubernetes services… When one has the spec.type = LoadBalancer then the CCM will act!

Before the CCM has chance to act, this is what the service will look like:

kubectl get svc my-service -o yaml

(some stuff omitted…)

- apiVersion: v1
  kind: Service
  metadata:
    creationTimestamp: "2021-01-02T17:46:16Z"
    name: my-service
    namespace: default
    resourceVersion: "5281940"
    selfLink: /api/v1/namespaces/default/services/hello-world
    uid: 7c14dbbd-f9fd-4279-8524-7418ff62d08f
  spec:
    clusterIP: 10.107.2.227
    externalTrafficPolicy: Cluster
    ports:
    - nodePort: 30497
      port: 80
      protocol: TCP
      targetPort: 80
    selector:
      app: hello-world
    sessionAffinity: None
    type: LoadBalancer
  status:
    loadBalancer: {}

We can see that the status is blank and the spec.loadBalancerIP doesn’t exist.. well, whilst we’ve been reading this the CCM has acted.. if we’re in the cloud it’s done some API calls or if we’re running our own it may have looked at it’s pool of addresses and found a free address. The CCM will take this address and modify the object by updating the spec of the service.

kubectl get svc my-service -o yaml

(more stuff omitted…)

- apiVersion: v1
 kind: Service
[....]
 spec:
   clusterIP: 10.107.2.227
   externalTrafficPolicy: Cluster
   loadBalancerIP: 147.75.100.235
   ports:
   - nodePort: 30497
     port: 80
     protocol: TCP
     targetPort: 80
   selector:
     app: hello-world
   sessionAffinity: None
   type: LoadBalancer
 status:
   loadBalancer: {}

… and that is it.

That is all the CCM needs to do although but your service will still be <pending> as the status is still blank :-(

Two other things need to happen at this point!

Kubernetes Services proxy

What is this “proxy” I speak of! … well, if you’ve noticed the kube-proxy pod on your cluster and wondered why it’s there then read on!

There is plenty more detail => here, but i’ll break it down mainly for LoadBalancers.

When we expose a service within Kubernetes, then the API server will instruct the kube-proxy to inject a bunch of rules into iptables/ipvs that will effectively capture traffic and ensure that this traffic goes to the pods defined under the service. By default any service we create will have its clusterIP and a port written into these rules so that regardless which node inside the cluster we try to access this clusterIP:port then the services proxy will handle this traffic and distribute it accordingly. The clusterIP are virtual IP addresses that are managed by the Kubernetes API, however with LoadBalancers the traffic is external and we don’t know what the IP address will be ¯\_(ツ)_/¯!

Well, as shown above the CCM will eventually modify the spec.loadBalancerIP with an address from the environment.. once this spec is updated then the API will instruct kube-proxy to ensure that any traffic for this externalIP:port is also captured and proxied to the pods underneath the service.

We can see that these rules now exist by looking at the output for iptables-save, all traffic for the address of the LoadBalancer is now forwarded on…

1 2	root@k8s:~# iptables-save \| grep 147 -A KUBE-SERVICES -d 147.75.100.235/32 -p tcp -m comment --comment "default/hello-world loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-2BRRD7ARGUSQ7MRP

The final piece of the puzzle is getting traffic to the machines themselves …

External traffic !

So between a CCM and the Kubernetes service proxy, we have been given an external IP address that we have for our service and the Kubernetes service proxy will ensure that any traffic in the cluster for that external IP address is distributed to the various pods. We now need to get traffic to the nodes themselves…

Hypothetically, if we had a second network adapter in one of the nodes then we could configure this network adapter with the externalIP and as long as we can route traffic to that IP address then the kubernetes service proxy will capture and distribute that traffic. Unfortunately that is a very manual operation, so what options/technologies could we adopt in order to manage this?

We would usually need a final running piece of software that “watches” a service, and once the service is updated with a spec.loadBalancerIP from the CCM we know that it’s good to advertise to the outside world! Also once we’re exposing this to the outside world we can modify the status of the service with the address that we’re exposing on so that clients and end-users know that they can now use this address!

status:
  loadBalancer:
    ingress:
    - ip: 147.75.100.235

There are two main technologies that we can use to tell an existing environment about our new loadBalancer address, and when someone accesses this address where to then send the traffic!

ARP

ARP (Address resolution Protocol) is a Layer 2 protocol, who has the main task of working out which hardware (MAC address) an IP address belongs to. When an IP address first appears on the network it will typically broadcast to the network that it exists, it will also broadcast the MAC address of the adapter that it’s bound to. This informs the network of this IP <==> MAC binding, meaning that on a simple network when packets need to go to an IP address the switching infrastructure knows which machine to send the traffic.

We can see this mapping by using the arp command:

$ arp -a
_gateway (192.168.0.1) at b4:fb:e4:cc:d3:80 [ether] on ens160
? (10.0.73.65) at e6:ca:eb:b8:a0:f3 [ether] on cali43893380087
? (192.168.0.170) at f4:fe:fb:54:89:16 [ether] on ens160
? (10.0.73.67) at 52:5c:1b:5f:e1:50 [ether] on cali0e915999b8d
? (192.168.0.44) at 00:50:56:a5:13:11 [ether] on ens160
? (192.168.0.45) at 00:50:56:a5:c1:86 [ether] on ens160
? (192.168.0.40) at 00:50:56:a5:5f:1d [ether] on ens160

Load Balancing NOTE: In order for this to work with out Kubernetes cluster, we would need to select a single node that would become in charge of hosting this externalIP and using ARP to inform the network that traffic for this address should be sent to that machine. This is because of two reasons:

If another machine broadcasts an ARP update then existing connections will be disrupted
Multiple machines can’t have the same IP address exposed on the same network

The flow of operation is:

A leader is selected
spec.loadBalancerIP is updated to 147.75.100.235
147.75.100.235 is added as an additional address to interface ens160
ARP broadcasts that traffic to 147.75.100.235 is available at 00:50:56:a5:4f:05
Update the service status that the service is being advertised

At this point the externalIP address is known to the larger network, and any traffic will be sent to the node that is elected leader. Once the traffic is captured by the rules in the kernel, the traffic is then sent to the pods that are part of the service.

BGP

BGP (Border Gateway Protocol) is a Layer 3 protocol, the main task is to update routers with the path to new routes (that new route can be a single address or a range of addresses). For our use-cases in a Kubernetes cluster we would use BGP to announce to the routers in the infrastructure that traffic for a our spec.loadBalancerIP should be sent to one or more machines.

Load Balancing NOTE: One additional benefit over ARP is that multiple nodes can advertise the same address, this actually provides both HA and load-balancing across all nodes advertising in the cluster. In order to do this, we can bind the externalIP to the localhost adapter (so it’s not present on the actual network) and leave it to the kernel to allow traffic from the routers into the kernel and allow it to be proxied by kube-proxy.

The flow of operation is:

spec.loadBalancerIP is updated to 147.75.100.235
147.75.100.235 is added as an additional address to interface lo
A BGP peer advertisement updates the other peers (routers) that the loadBalancerIP is available by routing to the machines address on the network (all machines can/will do this)
Update the service status that the service is being advertised

Any client/end-user that tries to access 147.75.100.235 will have their traffic go through the router, where a route will exist to send that traffic to one of the nodes in the cluster, where it will be passed to the kubernetes service proxy.

Overview

At this point we can see there are a number of key pieces amd technologies that can all be harnesed to put together a load-balancing solution for Kubernetes. However the CCM is arguably the most important, as the CCM has the role of being “aware” of the topology or architecture of the infrastructure. It may need to prep nodes with configuration details (BGP configuration settings etc.) and speak with other systems to request valid addresses that can be used for the loadBalancer addresses.

Kube-Vip and BGP loadbalancers with Unifi

Sun, 22 Nov 2020 16:06:56 +0000

Since starting at ~~Packet~~ Equinix Metal i’ve had to spend a bit of time getting my head around a technology called BGP. This is widely used at Equinix Metal, the main use case is to allow an Elastic IP address to route traffic externally (i.e. from the internet) into one or more physical machines in Equinix’s facilities. An external address will usually be routed into one of the Equinix datacenters, we can configure physical infrastructure to use this external IP and communicate to the networking equipment in this facility (using BGP) that traffic should be routed to these physical servers.

I’ve since been working to add BGP functionality to a Kubernetes load-balancing solution that i’ve been working on in my spare time kube-vip.io. This typically works in exactly the same method as described as above, where an EIP is advertised to the outside world and the BGP functionality will route external traffic to worker nodes and, where the service traffic is then handled by the service “mesh” (usually iptables rules capturing service traffic) within the cluster.

I’m hoping to take this functionality further and decided it would be nice to try and emulate the Equinix Metal environment as much as possible, i.e. having BGP at home. There are two methods that I could go down in order to have this sort of functionality within my house:

Create a local linux router, with it’s own network and use bgpd so I can advertise routes
Utilise my Unifi router, which after some light googling it turns out supports bgp

Given that:
a. It’s Sunday
b. I’m Lazy
c. The instructions for setting up a local linux router looked like a pita I opted to just use my USG.
d. The USG will enable me to advertise addresses on my existing network and the USG will route traffic the advertising hosts without me doing anything (exacerbating the laziness here)

Configuring the Unifi Security Gateway

These instructions are for the Unifi Security Gateway, however I suspect that the security gateway that is part of the “dream machine” should support the same level of functionality.

Before we begin we’ll need to ensure that we understand the network topology of the network so that we can configure for bgp to function as expected.

Device	Address
gateway	192.168.0.1
…
k8sworker01	192.168.0.70
k8sworker02	192.168.0.71
k8sworker03	192.168.0.72

In the above table we can see a subset of hosts on my network, the first host that is important is the gateway address on my network (which is the address of the USG). All hosts on my network have this set as the default gateway, which means that when a machine needs to access an IP address that isn’t 192.168.0.1-254 then it will send the traffic to the gateway for it to be routed to that specific address.

Your gateway can be found by doing something like the following:

1 2	$ ip route \| grep default default via 192.168.0.1 dev ens160 proto static

The other addresses that are worth noting are the three worker hosts in my Kubernetes cluster that will advertise their bgp routes to the gateway.

NOTE:
All of the next steps require ssh access into the appliance in order to enable and configure the USG so that bgp is enabled on the network.

If you don’t know your ssh password for your gateway then help is at hand.. because for some bizarre reason you can get the ssh username/password straight from the web UI.. Navigate to Network Settings -> Device Authentication and here you’ll find the ssh username and password.

NOTE 2:
Whilst it is possible to enable bgp functionality through the cli, the actual web client isn’t aware of the bgp configuration. This will result in the bgp configuration being wiped when you do things like modify firewall rules or port forwarding in the web UI.

Enabling BGP

To begin with we’ll need to ssh to the Unifi USG with the credentials that can be found from the web portal.

Below is the configuration that we’ll add, i’ll break it down below:

configure
set protocols bgp 64501 parameters router-id 192.168.0.1
set protocols bgp 64501 neighbor 192.168.0.70 remote-as 64500
set protocols bgp 64501 neighbor 192.168.0.71 remote-as 64500
set protocols bgp 64501 neighbor 192.168.0.72 remote-as 64500
commit
save
exit

configure - will enable the configuration mode for the USG.

set protocols bgp 64501 parameters router-id 192.168.0.1 - will enable bgp on the USG with the router-id as it’s IP address 192.168.0.1 the AS number 64501 is used to determine our particular bgp network

set protocols bgp 64501 neighbor x.x.x.x remote-as 64500 - will allow our bgp instance to take advertised routes from x.x.x.x with the AS identifier 64500

commit | save - will both enable and then save the configuration

This is it.. the USG is now configured to take routes from our Kubernetes workers!!!

Configuring Kube-Vip

I’ve recently started on a release of kube-vip that will provide additional functionality in order to automate some of the bgp configuration, namely the capability to determine the host IP address for the bgp server-id for advertising from a pod.

We can follow the guide for setting up kube-vip with bgp with some additional modifications show below!

Ensure that the version of the kube-vip image is 0.2.2 or higher

env:
  - name: vip_interface
	value: "lo"
  - name: vip_configmap
	value: "plndr" 
  - name: bgp_enable
	value: "true"
  - name: vip_loglevel
	value: "5"
  - name: bgp_routerinterface
	value: "ens160"
  - name: bgp_as
	value: "64500"
  - name: bgp_peeraddress
	value: "192.168.0.1"
  - name: bgp_peeras
	value: "64501"

The bgp_routerinterface will autodetect the host IP address of the interface specified and this will become our server-id for each pods bgp peering configuration. The bgp_peer<as|address> is the remote configuration for our USG as specified above.

Testing it out !

Create a easy deployment, we can use the nginx demo for this:

1	kubectl apply -f https://k8s.io/examples/application/deployment.yaml

Given that the USG is our default gateway we can actually advertise any address we like as a type:LoadBalancer and when we try and access if from inside the network the USG will route us back to the the Kubernetes servers!

For example we can create a load-balancer with the address 10.0.0.1

1	kubectl expose deployment nginx-deployment --port=80 --type=LoadBalancer --name=nginx-bgp --load-balancer-ip=10.0.0.1

Given the home network is in the 192.168.0.0/24 range anything that is outside of it will need to go through our USG router, where we’ve advertised (through BGP) that traffic needs to come to any of the Kubernetes workers.

We can see this below on the USG with show ip bgp

admin@Gateway:~$ show ip bgp
BGP table version is 0, local router ID is 192.168.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
			  r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
* i10.0.0.1/32      192.168.0.45                  100      0 i
*>i                 192.168.0.44                  100      0 i
* i                 192.168.0.46                  100      0 i

Fixing Plunders (OS) Image problem

Thu, 04 Jun 2020 08:57:43 +0100

This is a post about:

Operating Systems
The Linux Boot process
Installation methods
Doing weird hacky things 😃

In this post I’m going to cover the options that are typically available to an end user when looking to install an Operating System (mainly Linux, but I may touch on others 🤷) , we will touch on their pros/cons and look at what the alternatives are. We will then discuss concepts of immutable Operating System deployments and how we may go about doing so, including the existing eco-system. Finally we’ll look at what is required (using Go) to implement an Image based Operating System creation and deployment, and how they’ve been implemented into plunder.

Getting an Operating System deployed

Most Operating Systems are deployed in relatively the same manner:

A machine boots and reads from installation media (presented locally or over the network)
The target disks are prepared, typically partitions may be created or HA technologies such as disk mirroring and then finally these partitions are “formatted” so that they contain a file system.
Either a minimal set of packages or a custom selection of packages will be installed to the new file system. Most Operating Systems or distributions have their own concept of “packages” but ultimately under the covers the package contains binaries and required libraries for an application along with some logic that dictates where the files should be written too along with some versioning information that the package manager can use.
There may be some final customisation such as setting users, network configuration etc..
A Boot loader is written to the target disk so that when the machine is next powered on it can boot the newly provisioned Operating System.

The order of the steps may differ but pretty much all of the major Operating Systems (Linux, Windows, MacOS) follow the same pattern to deploy on target hardware.

Options for automating a deployment

There are usually two trains of thought when thinking about deploying an Operating System, which are scripted which will go through the steps listed above but no user interaction is required or image based which takes a copy of a deployment and uses that as a “rubber stamp” for other installs.

Scripted

Operating Systems were originally designed to be ran on hardware of a predetermined configuration, which meant that there was no need for customising of the installation. However as time passed a few things happened that suddenly required Operating Systems to become more flexible:

Usage of computers sky rocketed
The number of hardware vendors producing compute parts increased
A lot of traditional types of work became digitised.

All of these factors suddenly required an Operating System to support more and more types of hardware and it’s required configuration(s), furthermore end-users required the capability to tailor an Operating System to behave as needed. To provide this functionality Operating System vendors built rudimentary user interfaces that would ask questions or provide the capability for a user installing the OS to set various configuration options. This worked for a period of time but as more and more computer systems were deployed this became an administrative nightmare, as it was impossible to automate multiple installations as they required interaction to proceed and finally the need for humans to interact brings about the possibility of human error (Pesky humans) during the deployment.

In order for large scale IT system installations to take place then operations needed a method for unattended installations, where installations can happen without any involvement. The technique for this to work was to modify the Operating System installation code so that it could take a file that can answer all of the questions that would have previously required user input in order to progress the installation. These technologies are all named in a way that reflects that:

preeseed
answer file(s)
kickstart
jumpstart

Once a member of the operations team has “designed” the set of responses for their chosen Operating System then this single configuration can be re-used as many times as required. This removes the human element from accidentally entering the wrong data or clicking the wrong button during an Operating System installation and ensures that the installation is standardised and “documented”.

However, one thing to consider is that although a scripted installation is a repeatable procedure that requires no human interaction it is not always 100% reliable. This installation still runs through a lot of steps, such as every package has to be installed along with the prerequisite package management along with configuring devices and other things that happen during that initial installation phase. Whilst this may work perfectly on the initial machine undiscovered errors can appear when moving this installation method to different hardware. There have been numerous issues caused by packages relying on sleep during an installation step, the problem here usually is due to this package being developed on a laptop and then moved to much larger hardware. Suddenly this sleep is no longer in step with the behaviour of the hardware as the task completes much quicker than it had on slower hardware. This has typically led to numerous installation failures and can be thought of as a race-condition.

Image

Creating an image of an existing Operating System has existed for a long time, we can see it referenced in this 1960s IBM manual for their mainframes.

Shoutout for Norton Ghost !!

I have covered image based deployments, but my (google) research show ghost (1998) pre-dates Preseed/DebianInstaller (2004) and Kickstart(RHEL in 2000) and even Sun Microsystems with jumpstart (earliest mention I can find is Solaris 2.8 which is 1999).

Anatomy of a disk

We can usually think of a disk being like a long strip of paper starting at position 0 and ending with the length of the strip of paper (or its capacity). The positions are vitally important as they’re used by a computer when it starts in order to find things on disk.

The Boot sector is the first place a machine will attempt to boot from once it has completed its hardware initialisation and hardware checks the code in this location will be used in order to instruct the computer where to look for the rest of the boot sequence and code. In the majority of examples a computer will boot the first phase from this boot sector and then be told where to look for the subsequent phases and more than likely the remaining code will live within a partition.

A partition defines some ring-fenced capacity on an underlying device that can be then presented to the underlying hardware as usable storage. Partitions will then be “formatted” so that they have a structure that understands concepts such as folders/directories and files, along with additional functionality such as permissions.

Now that we know the makeup of a disk we can see that there are lots of different things that we may need to be aware of, such as type of boot loader, size or number of partitions, type of file systems and then the files and packages that need installing within those partitions.

We can safely move away from all of this by taking a full copy of the disk! Starting from position 0 we can read every byte until we’ve reached the end of the disk (EOF) and we have a full copy of everything from boot loaders to partitions and the underlying files.

The steps for creating and using a machine image are usually:

Install an Operating System once (correctly)
Create an image of this deployed Operating System
Deploy the “golden image” to all other hosts

Concepts for managing OS images

The concept for reading/writing OS images consists of a number of basic actions:

Reading or Writing to a storage device e.g. /dev/sda
A source image (either copied from a deployed system or generated elsewhere)
A destination for the date from the cloned system, when reading from a deployed system we need to store a copy of the data somewhere

In relation to the final point, there have been numerous options to capture the contents of an existing system over time:

Mainframes could clone the system to tape
Earlier “modern” systems would/could clone to a series of Floppy disks and later on CD-ROM media
Stream the contents of a storage device over the network to a remote listening server.

Reading (cloning) data from a storage device

When capturing from an existing system, especially when this OS image will be used to deploy else where, there are a few things that need to be considered about the state of the source system.

Firstly it is ideal that it is as “clean” as possible, which typically means that the OS is deployed and any required POST-configuration and nothing else. If a system has been up for time or has been used then the filesystem could already be dirty through excess logs or files generated that will clutter the source image and end up being deployed through to all destination servers. We may want to consider removing or zeroing the contents of swap partitions or files as well to ensure nothing is taken from the source and passed to destination machines that doesn’t need to be copied.

Secondly that the disk isn’t actually being used when we want to clone or image its contents, whilst technically possible if the Operating System is busy reading and writing whilst we attempt to copy the underlying storage we will end up with files that could be half-written to or ultimately corrupt. We can boot from alternative media (USB/CD-ROM/Alternate Disk/Network) in order to leave our source disk un-used, which will allow us to copy it’s contents in a completely stable state.

Once we’ve started up our cloning tooling we only need to read all date from the underlying storage device to a secondary data store. We need to be aware that even if we’ve only installed a small amount of data as part of the Operating System our cloned copy will be the exact size of the underling storage device. e.g. a 100GB disk that is installed with a basic Ubuntu 18.04 package set (900MB) will still result in a 100GB OS Image.

Writing an OS Image to a destination device

As with reading the contents of an underlying storage device we can’t (or I certainly wouldn’t recommend) trying to use the same storage device at the same time as writing too it. If you had an existing system running an Operating System such as Win10 and we used some tooling to re-write our ubuntu disk image to the same storage then well …

To circumvent this issue we would need to start our destination machine from another medium such as another storage device leaving our destination device alone until we’re ready to write our destination disk image to this storage.

Finally we also need to consider the sizes of the disk image and the destination storage:

If our destination storage is smaller than the disk image then our writing of the image will fail. However, the system may start afterwards as the main contents of the Operating System has fit on the storage device. This can lead to an unstable state as the filesystem and partitions have logical boundaries that exist well beyond the capacity of the destination disk.
On the other hand if we write a 10GB OS image to a 100GB physical storage device then the remaining 90GB is left unused without some additional steps, typically involving first growing the partition to occupy the remaining 90GB of space and then growing the filesystem so it is aware of this additional capacity.

Writing your own OS imaging software

This section will detail the design around software I wrote to accomplish all of the things I’ve discussed above. All of the code samples are written in Go and the source code for the project can be found on the BOOTy repository.

It will be in three parts:

First, creating a boot environment that allows us to safely read/write from underlying storage
Second, Reading and Writing from underlying storage
Finally, post steps.

A solution for a clean environment

To ensure we can make a clean read or write to a storage device we need to ensure that it’s not being used at the time, so we will need to boot from a different environment that has our tooling in it.

Simplest option

We can simply use a live-CD image that allow us to boot a fully fledged Operating System, from here we can use some tooling like dd to take a copy of contents of the storage. The problems with this solution are that we can’t really automate this procedure and we will need a writeable location for the contents of the source disk.

Writing a custom Operating System

Phase one: LinuxKit

My initial plan was to do this with LinuxKit, where the code to handle the reading and writing of the underlying storage would live within a container and I would then use LinuxKit to bake this into a custom Linux distribution that I could then start on remote hosts.

Start remote host and point to our LinuxKit OS
Start Linux Kernel
Kernel boots, finds all hardware (including our storage device(s)
LinuxKit starts /init which in turn will start our application container
The container will then interact with the underlying storage device (e.g. write an image to the disk)

This solution was quick and easy to write, however a few issues:

Ensuring console output of what is transpiring
Stable rebooting or restarting of the host
Dropping to a console if a write fails

So whilst it did work, I decided I could probably do something a little smaller

Phase two: `initramfs`

This design uses two pieces of technology, mainly a Linux Kernel and an initramfs which is where the second stage of Linux Booting occurs, if we use the same chronology as above this design would look like the following:

Start remote host and point to our OS
Start Linux Kernel
Kernel boots, finds all hardware (including our storage device(s)
Kernel starts /init within our initramfs , this init is our code to manage underlying storage.

At this point our init is the only process started by the kernel and all work is performed within our init binary. Within our initramfs we can add in any additional binaries, such as a shell (busybox) or Logical Volume Manager tooling to manipulate disks (more on this later).

In order for this tooling to work our custom init will need to “setup” the environment so that various things exist within the Linux filesystem, including special device nodes such as /dev/tty or /dev/random which are required for things like writing console output or generating UUIDs for filesystems etc..

Below is the example code that we will need to generate our environment:

package main

import "syscall"

func main() {

	// Mount /dev
  err := syscall.Mount("devtmpfs", "/dev", "devtmpfs", syscall.MS_MGC_VAL, 0)

	// Create /dev/random
	err := syscall.Mknod("/dev/random", syscall.S_IFCHR, makedev(1, 3))

}

We will need to mount various additional file systems in order for additional functionality to work as expected, mainly systems such as /proc(proc) and /sys (sysfs).

Once our environment is up and running, we will need to use dhcp in order to get an address on the network. There are numerous libraries available that provide DHCP client functionality and we can leverage one to run in a go routine to ensure we have an address and renew it if we get close to our lease expiring before we finish our work.

At this point we would do our disk image work, this will be covered next !

With an init we need to handle failure carefully, if our init process just ends either successfully 0 or with failure -1 then our kernel will simply panic. This means that we need to ensure we handle all errors carefully and in the event something fails ensure it is captured on screen (with a delay long enough for the error to be read) and then issue a command to either start a shell or reboot the host.

On error and we want to drop to a shell then the following code will ensure that the interrupt is passed from our init process to its children.

// TTY hack to support ctrl+c
cmd := exec.Command("/usr/bin/setsid", "cttyhack", "/bin/sh")
cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr

err := cmd.Start()
if err != nil {
	log.Errorf("Shell error [%v]", err)
 }
err = cmd.Wait()
if err != nil {
	log.Errorf("Shell error [%v]", err)
}

Finally instead of simply ending our init process we will need to issue a reboot or shutdown as simply ending init will result in a kernel panic and a hung system.

Note: Before doing this ensure that disk operations are complete and that anything mounted or open has been closed and unmounted, as this syscall will immediately reboot.

err := syscall.Reboot(syscall.LINUX_REBOOT_CMD_RESTART)
if err != nil {
	log.Errorf("reboot off failed: %v", err)
	Shell()
}

// We can opt to panic by doing the following
os.Exit(1)

We now have all of the required bits in place:

Our custom /init starts up
The relevant paths are created or mounted
Any devices nodes are created allowing us to interact with the system as a whole
We use a DHCP Client to retrieve an address on the network
In the event of an error we can start a shell process allowing us to examine further as to what the error may be
Finally we will issue a reboot syscall when we’ve finished

Managing images, and storage devices

With the above /init in place we can now boot a system that has all the underlying devices ready, is present on the network and in event of errors can either restart or drop to a point where we can look into debugging.

We are in a position where we can put together some code that will allow reading the storage and sending it across the network to a remote server OR pull that same data and write it to the underlying storage.

Identifying the server and behaviour

In order to know what action a server should take when it is booted from our custom init we need to use a unique identifier, luckily we have one that is built into every machine its MAC/Hardware address.

// Get the mac address of the interface ifname (in our case "eth0")
mac, err := net.InterfaceByName(ifname)
if err != nil {
	return "", err
}

// Convert into something that RFC 3986 compliant
return strings.Replace(mac, ":", "-", -1)

The above piece of code will find the MAC address of an interface passed as ifname and it will then convert that into a string that is a compliant URI string 00:11:22:33:44:55 -> 00-11-22-33-44-55 . We can now use this to build a url that we can request to find out what action we should be performing, so once init is ready it will build this configuration and perform a GET on http://<serverURL>/<MAC address>.

You may be wondering where we specify the <serverURL> address, well we can hard code this into our init alternatively we can pass this on boot as a flag to the kernel MYURL=http://192.168.0.1, this will appear as a environment variable within our newly started Operating System.

1	serverURL := os.Getenv("MYURL")

Reading and Writing

We can now call back to a server that will can inform the init process if it should be reading from the storage device and creating an OS image or pulling an image and writing it to a device.

Writing an image to disk

This is arguably the easier task as we can use a number of pre-existing features within Go to make this very straight forward. Once we’ve been told from the server that we’re writing an image, we should also be given a URL that points to an OS image location. We can pass this imageURL to our writing function and use an io.Copy() to write/stream this image directly to the underlying storage.

// New request to GET the actual image
req, err := http.NewRequest(http.MethodGet, imageURL, nil)
if err != nil {
	return err
}

resp, err := http.DefaultClient.Do(req)
if err != nil {
	return err
}
defer resp.Body.Close()

// Open the underlying storage
diskIn, err := os.OpenFile("/dev/sda", os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
	return err
}

 // Read from resp.Body and Write to the underlying Storage 
count, err := io.Copy(diskIn, resp.Body)
if err != nil {
	return fmt.Errorf("Error writing %d bytes to disk [%s] -> %v", count, destinationDevice, err)
}

Reading from Disk to an Image

The main issue we hit with reading from a disk to a remote location is around the large amounts of data, the underlying storage could easily be many many GBs of data that would need transmitting to a remote server. In order to send such large amounts of data over the HTTP protocol we can use a multipart writer that will break up the file and rebuild it on the server side. To do this we create a multipart writer and then as we read chunks of data from /dev/sda we send them as multiple parts over the network.

body, writer := io.Pipe()

 // New request to POST the image (contents of the disk)
req, err := http.NewRequest(http.MethodPost, uri, body)
if err != nil {
	return nil, err
}

 // Create our multipart Writer 
mwriter := multipart.NewWriter(writer)
req.Header.Add("Content-Type", mwriter.FormDataContentType())

errchan := make(chan error)

// Go routine for the copy operation
go func() {
	defer close(errchan)
	defer writer.Close()
	defer mwriter.Close()

	// imageKey is the key that the client will look for and
	// key is the what the file should be called, we can set this to teh MAC address of the host
	w, err := mwriter.CreateFormFile("imageKey", key)
	if err != nil {
		errchan <- err
		return
	}

     // Open the underlying storage
	diskIn, err := os.Open("/dev/sda")
	if err != nil {
		errchan <- err
		return
	}

     // Copy from the disk into the mulitpart writer (which inturn sends data over the network)
	if written, err := io.Copy(w, diskIn); err != nil {
		errchan <- fmt.Errorf("error copying %s (%d bytes written): %v", path, written, err)
		return
	}
}

resp, err := client.Do(req)
merr := <-errchan

if err != nil || merr != nil {
	return resp, fmt.Errorf("http error: %v, multipart error: %v", err, merr)
}

Shrinking Images

Using disk images can be incredibly wasteful when thinking about network traffic, a 1GB Operating System on a 100GB disk requires sending 100GB of data (even when most of the disk is probably zeros). To save a lot of space we can expand our read/write functions so that we pass our data through a compressor (or expander in the other direction), we can do this very easily by modifying the above code examples.

// Reading raw data from Disk, compressing it and sending it over the network
	
		zipWriter := gzip.NewWriter(w)
			if err != nil {
				errchan <- fmt.Errorf("[ERROR] New gzip reader: %s", err)
				return
			}

			// run an io.Copy on the disk into the zipWriter
			if written, err := io.Copy(zipWriter, diskIn); err != nil {
				errchan <- fmt.Errorf("error copying %s (%d bytes written): %v", path, written, err)
				return
			}

			// Ensure we close our zipWriter (otherwise we will get "unexpected EOF")
			err = zipWriter.Close()



// Expanding compressed data and writing it to Disk

		// Create a gzip reader that takes the compressed data over the network
		zipOUT, err := gzip.NewReader(resp.Body)
		if err != nil {
			fmt.Println("[ERROR] New gzip reader:", err)
		}
		defer zipOUT.Close()

      // Read uncompressed data from gzip Reader and Write to the underlying Storage 
		count, err := io.Copy(fileOut, zipOUT)
		if err != nil {
			return fmt.Errorf("Error writing %d bytes to disk [%s] -> %v", count, destinationDevice, err)
		}

We can see that we simply “man-in-the-middle” a compression solution that slots straight into the existing workflow that was shown earlier. The results of adding in compression are clear to see when taking an Ubuntu 18.04 standard install on any sized disk 4GB / 100GB the compressed OS Image is always around 900MB.

Tidying up

We could simply stop here and we have all of the components in place to both create and write Operating System images to various hardware over the network, however in a lot of cases we may want to perform some POST configuration once we’ve deployed an image.

Grow the storage

To perform this we (in most circumstances) require some external tooling, if our base Operating System image used technologies like LVM etc.. then we’ll need additional tooling to interact with them. So within our initramfs we may want to create a static build of LVM2 so that we can use this tooling without requiring a large amount of additional libraries within our RAM disk. One other technique we can use is to make use of tooling that may exist within the Operating System Image that we’ve just written to disk, below is a sample workflow:

Write Operating System image to /dev/sda
Exec out to /usr/sbin/partprobe within out ram-disk to scan /dev/sda and find our newly written disk contents
Exec out to /sbin/lvm to enable any logical volumes
TODO: look at BLKRRPART to replace partprobe

At this point our /init will have an update /dev that will have any partitions (/dev/sda1) or logical volumes (/dev/ubuntu-vg/root) present at which point we can act on these partitions to grow them and their filesystems.

Some cheeky `chroot` manuveurs

As mentioned we can use a workflow that means we can use the tooling that exists within our newly deployed Operating System instead of filling our ramdisk with additional tooling and the associated dependencies.

The following workflow will grow a 900MB image written to a 20GB disk that uses LVM:

Mount the root volume inside the ram-disk /dev/ubuntu-vg/root -> /mnt
Use chroot to pretend we’re inside the newly deployed Operating System grow the underlying partition chroot /mnt /usr/bin/growpart /dev/sda 1
Again chroot and update LVM to see the newly grown disk chroot /mnt /sbin/pvresize /dev/sda1
Grow the root logical volume chroot /mnt /sbin/lvresize -l +100%FREE /dev/ubuntu-vg/root
Finally grow the filesystem within the logical volume chroot /mnt /sbin/resize2fs /dev/ubuntu-vg/root
Finally, unmount the logical volume ensuring that any writes are flushed!

Configure networking and other things

The above workflow for growing the storage mounts our newly provisioned disk image, once we’ve finished growing the disk/partitions and filesystem we have the opportunity to do additional post-deployment steps. For networking this could include writing a static networking configuration given too us from the provisioning server to the underlying filesystem before we unmount it and boot from it.

Overview

In this post we have detailed some of the deployment technologies that exist in order to provision Operating Systems and we can appreciate that there are a number of options available to us about which approach can work best. We’ve also stepped through various code snippets that detail some of the functionality that has recently been added into plndr to add the capability to create and use OS images to quickly and efficiently deploy Operating Systems to bare-metal (and virtualised) servers.

All of the source code samples came from BOOTy which is a project to build an initramfs that can perform a call back to a plndr server to find out it’s course of action.

Any questions, mistakes or corrections either add in the comments or hit me up on twitter -> thebsdbox

Ready, set, client-go

Mon, 30 Mar 2020 15:43:24 +0100

This post will detail a number of (I think at least) awesome use-cases for client-go. Some of these are use-cases that are similar or may almost be identical to the examples that already exist, but with some additional text that details what some of these terms actually mean and when it makes sense to use them.

Why Client-Go?

Kubernetes exposes everything through an API (all managed by the active API server) from the control-plane, the API is rest based and is the sole way of controlling the a cluster. This means that things like CI/CD pipelines, various dashboards and even kubectl will all use the API through an endpoint (network address) with the credentials (key) in order communicate with a cluster.

As this is just a standard REST over http(s) then there are a myriad of methods that can be used in order to communicate with the Kubernetes API. We can demonstrate this in a quick example:

Check health of Kubernetes API

1 2	$ curl -k https://control-plane01:6443/healthz ok

The above example is an endpoint that requires no authentication, however if we try to use another endpoint without the correct authentication we’ll receive something like the following:

$ curl -k https://control-plane01:6443/api
{
  “kind”: “Status”,
  “apiVersion”: “v1”,
  “metadata”: {
    
  },
  “status”: “Failure”,
  “message”: “forbidden: User \”system:anonymous\” cannot get path \”/api\””,
  “reason”: “Forbidden”,
  “details”: {
    
  },
  “code”: 403
}

Note: Under most circumstances the authentication a user would need to speak with a cluster will live within a $home/.kube/config file. Tooling like kubectl will automatically look for this file in order to communicate with the API endpoint.

I wouldn’t recommend interacting with the raw endpoints as shown above, they’re mainly shown as an example of what is possible. To make life far simpler for developers to interact with the Kubernetes API then there are a number of wrappers/SDKs that provide:

Control and management of both the SDK <-> API versions
Language specific objects and methods to provide sane interfaces to Kubernetes objects
Helper functions for logging in and managing cluster access
(A plethora of additional features to make your life easier)

As mentioned this post will cover client-go, but there are numerous SDKs in various languages that are covered in varying levels of detail here.

Accessing a cluster, either In-Cluster or Outside cluster

This can be confusing for the first time an end-user will attempt to authenticate with a cluster using client-go.

Creating a Kubernetes cloud provider, doesn't required boiling the ocean

Wed, 18 Mar 2020 11:49:34 +0000

Whilst working on a number of kubernetes control-plane deployments and the fighting with kubeadm and components like haproxy, nginx and keepalived, I decided to try and create my own load-balancer. In the end this proved to be not too complicated to replicate both a Virtual IP and load-balancing over the backends (master nodes) with the standard Go packages. That project now is pretty stable and can easily be used to create a HA control-plane, all of that can be found on https://kube-vip.io. The next thing I wanted to try (and I’ve been considering learning about this for a while) was creating a load-balancer “within” Kubernetes, as in provide the capability and functionality to kubectl expose <...> --type=LoadBalancer. It turns out that in order to provide this functionality within Kubernetes you need to write a Cloud Provider that the Kubernetes Cloud Controller Manager can interface with.

This post will “chronicle” the process for doing that… :-D

Kubernetes Cloud Providers

We will start with the obvious question.

What are Cloud providers?

Out of the box Kubernetes can’t really do a great deal, it really needs a lot of components to sit on top of or to interface with in order for it to provide the capability to run workloads. For example even in a basic Kubernetes cluster there is a requirement to have a container runtime (CRI, Container Runtime Interface) in order to execute the container, then we would need a networking plugin (CNI, Container Network Interface) in order to provide networking within the cluster.

On the flip side, a typical cloud company (AWS, GCloud, Azure etc…) offers a plethora of cool features and functionality that it would be awesome to consume through the Kubernetes cluster:

Load Balancers
Cloud instances (VMs, in some placess bare-metal)
Areas/zones
Deep API integrations into the infrastructure

So how do we marry up these two platforms to share that functionality …

.. Kubernetes Cloud Providers ..

Using Cloud providers

In most circumstances, you won’t even know that you’re using a cloud provider (which I suppose is kind of the point) and only when you try to create an object that the cloud provide can create/manage/delete will it actually be invoked.

The most common use-case (and the one this post is focussing on) is the creation of a load balancer within Kubernetes and it’s “counterpart” being provided by the cloud vendor. In the case of the cloud vendor Amazon Web Services (AWS) then creating a services type: LoadBalancer will create an Elastic Load Balancer (ELB) that will then load balance traffic over the selected pods. All of this functionality from the Cloud Provider Interface abstracts away the underlying technology and regardless of where a cluster is running a LoadBalancer just becomes a LoadBalancer.

Creating a Cloud Provider!

So now onto the actual steps to creating your own cloud provider, this is all going to be written in Go and I’ll do my best to be as descriptive as possible.

Wait, what is the `cloud-controller-manager`?

In Kubernetes v1.6 the original design was that all the cloud providers would have their vendor specific code all live in the same place. This ultimately lead to a point where all Kubernetes clusters came with a large cloud-controller-manager that at startup would be told which vendor code path to run down.

These were originally called In Tree cloud providers and there has been a push over the last few years to move to Out of Tree providers. When deploying a Kubernetes cluster the only change is that instead of starting the cloud-controller-manager with a specific vendor path (e.g. vsphere or aws), the operator would deploy the vendor specific cloud-provider such as cloud-provider-aws.

A Note about the “why” of In Tree / Out of Tree

There has been a shift of stripping code and “vendor specific” functionality from the main Kubernetes source repositories and into their own repositories. The main reasons for this:

Removes a tight-coupling between external/vendor code and Kubernetes proper
Allowed these projects to move at a different release rate to the main project
Slims the Kubernetes code base and allows these things to become optional
Reduces the vulnerability footprint of vendor code within the Kubernetes project
The interfaces ensure ongoing compatibility for these Out Of Tree projects

So for someone to create their own cloud provider they will need to follow a standard that was set by the original cloud-controller-manager, this standard is exposed through method sets and interfaces which can be read about more here.

tl;dr simply put, the cloud-controller-manager sets a standard that means if I want to expose a Load Balancer service it needs to also expose a number of methods (with matching signature). We can further see in the LoadBalancer interface here all of the functions that my Loadbalancer must expose in order to work.

The interface

The interface for a cloud-provider can be viewed here, we can see that this interface provides a number of functions that will return the interface for a specific type of functionality.

The more common interfaces I’ve summarised below:

Instances controller - responsible for updating kubernetes nodes using cloud APIs and deleting kubernetes nodes that were deleted on your cloud.
LoadBalancers controller - responsible for load balancers on your cloud against services of type: LoadBalancer.
Routes controller - responsible for setting up network routes on your cloud

Example provider (code) `cloud-provider-thebsdbox`

This section will cover in Go code all of the basics for building a cloud-provider that will handle all of the services requests (that are type: LoadBalancer). When implementing your own, ensure you use correct paths and package names!

Our cloud-provider will use the following files:

pkg/thebsdbox/cloud.go
pkg/thebsdbox/loadbalancer.go
pkg/thebsdbox/disabled.go
main.go

The next sections will have the source code and then an overview at the end of each file.

`cloud.go`

package thebsdbox

import (
	“k8s.io/client-go/informers”
	“k8s.io/client-go/tools/clientcmd”
	cloudprovider “k8s.io/cloud-provider”
)

const ProviderName = “thebsbdox” 

// ThebsdboxCloudProvider - contains all of the interfaces for our cloud provider
type ThebsdboxCloudProvider struct {
	lb cloudprovider.LoadBalancer
}

var _ cloudprovider.Interface = &thebsdboxCloudProvider{}

func init() {
	cloudprovider.RegisterCloudProvider(ProviderName, newThebsdboxCloudProvider)
}

func newThebsdboxCloudProvider(io.Reader) (cloudprovider.Interface, error) {
	return &ThebsdboxCloudProvider{
		lb: newLoadBalancer(),
	}, nil
}

// Initialize - starts the clound-provider controller
func (t *ThebsdboxCloudProvider) Initialize(clientBuilder cloudprovider.ControllerClientBuilder, stop <-chan struct{}) {
	clientset := clientBuilder.ClientOrDie(“do-shared-informers”)
	sharedInformer := informers.NewSharedInformerFactory(clientset, 0)

	sharedInformer.Start(nil)
	sharedInformer.WaitForCacheSync(nil)

}

// ProviderName returns the cloud provider ID.
func (t *ThebsdboxCloudProvider) ProviderName() string {
	return ProviderName
}

// ENABLED Services

// LoadBalancer returns a loadbalancer interface. Also returns true if the interface is supported, false otherwise.
func (t *ThebsdboxCloudProvider) LoadBalancer() (cloudprovider.LoadBalancer, bool) {
	return t.lb, true
}

Our Cloud-Provider `ThebsdboxCloudProvider` struct{}

This struct{} contains our vendor specific implementations of functionality, such as load-balancers, instances etc..

1
2
3

type ThebsdboxCloudProvider struct {
	lb cloudprovider.LoadBalancer
}

We’re only defining a loadbalancer lb variable as part of our cloud-provider instances as this is the only functionality our provider will expose.

`init()`

This function will mean that before our cloud-provider actually starts (before the main() function is called) we will register our vendor specific cloud-provider. It will also ensure that our newly registered cloud-provider will be instantiated with the newThebsdboxCloudProvider function.

Instantiating our cloud-provider `newThebsdboxCloudProvider()`

When our cloud-provider actually has started (the main() function has been called) the cloud-provider controller will look at all registered providers, and it will find ours that we registered in the init() function. It will then call our instantiation function newLoadBalancer(), which will do any pre-tasks for setting up our load balancer, it will then assign it to lb.

Enabling our Load-Balancer `func (t *ThebsdboxCloudProvider) LoadBalancer() (cloudprovider.LoadBalancer, bool)`

This function is pretty much the crux of enabling the load balancer functionality and as part of the cloud-controller-manager spec defines what functionality our cloud-provider will expose. These functions will return one of two things:

Our instantiated functionality (in this case our loadbalancer object, returned as lb)
If this is enabled or not (true/false)

Everything that we’re not exposing from our cloud-provider will return false and can be seen in the disabled.go source.

`disabled.go`

All of these functions disable these bits of functionality within our cloud-provider

package thebsdbox

import cloudprovider “k8s.io/cloud-provider”

// Instances returns an instances interface. Also returns true if the interface is supported, false otherwise.
func (t *ThebsdboxCloudProvider) Instances() (cloudprovider.Instances, bool) {
	return nil, false
}

// Zones returns a zones interface. Also returns true if the interface is supported, false otherwise.
func (t *ThebsdboxCloudProvider) Zones() (cloudprovider.Zones, bool) {
	return nil, false
}

// Clusters returns a clusters interface.  Also returns true if the interface is supported, false otherwise.
func (t *ThebsdboxCloudProvider) Clusters() (cloudprovider.Clusters, bool) {
	return nil, false
}

// Routes returns a routes interface along with whether the interface is supported.
func (t *ThebsdboxCloudProvider) Routes() (cloudprovider.Routes, bool) {
	return nil, false
}

// HasClusterID provides an opportunity for cloud-provider-specific code to process DNS settings for pods.
func (t *ThebsdboxCloudProvider) HasClusterID() bool {
	return false
}

`loadbalancer.go`

Our LoadBalancer source code again has to match the interface as expressed here, we can see those functions defined below and exposed as methods as part of our thebsdboxLBManager struct.

package thebsdbox

import (
	“context”
	“fmt”

	v1 “k8s.io/api/core/v1”
	“k8s.io/client-go/kubernetes”
	cloudprovider “k8s.io/cloud-provider”
	“k8s.io/klog”
)

//thebsdboxLBManager - 
type thebsdboxLBManager struct {
	kubeClient     *kubernetes.Clientset
	nameSpace      string
}

func newLoadBalancer() cloudprovider.LoadBalancer {
	// Needs code to get a kubeclient => client
	// Needs code to get a namespace to operate in => namespace

	return &thebsdboxLBManager{
		kubeClient: client,
		namespace: ns,}
}

func (tlb *thebsdboxLBManager) EnsureLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) (lbs *v1.LoadBalancerStatus, err error) {
	return tlb.syncLoadBalancer(service)
}
func (tlb *thebsdboxLBManager) UpdateLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) (err error) {
	_, err = tlb.syncLoadBalancer(service)
	return err
}

func (tlb *thebsdboxLBManager) EnsureLoadBalancerDeleted(ctx context.Context, clusterName string, service *v1.Service) error {
	return tlb.deleteLoadBalancer(service)
}

func (tlb *thebsdboxLBManager) GetLoadBalancer(ctx context.Context, clusterName string, service *v1.Service) (status *v1.LoadBalancerStatus, exists bool, err error) {

	// RETRIEVE EXISTING LOAD BALANCER STATUS	

	return &v1.LoadBalancerStatus{
		Ingress: []v1.LoadBalancerIngress{
			{
				IP: vip,
			},
		},
	}, nil
}

// GetLoadBalancerName returns the name of the load balancer. Implementations must treat the
// *v1.Service parameter as read-only and not modify it.
func (tlb *thebsdboxLBManager) GetLoadBalancerName(_ context.Context, clusterName string, service *v1.Service) string {
	return getDefaultLoadBalancerName(service)
}

func getDefaultLoadBalancerName(service *v1.Service) string {
	return cloudprovider.DefaultLoadBalancerName(service)
}
func (tlb *thebsdboxLBManager) deleteLoadBalancer(service *v1.Service) error {
	klog.Infof(“deleting service ‘%s’ (%s)”, service.Name, service.UID)
	
	// DELETE LOAD BALANCER LOGIC

	return err
}

func (tlb *thebsdboxLBManager) syncLoadBalancer(service *v1.Service) (*v1.LoadBalancerStatus, error) {

	// CREATE / UPDATE LOAD BALANCER LOGIC (and return updated load balancer IP)

	return &v1.LoadBalancerStatus{
		Ingress: []v1.LoadBalancerIngress{
			{
				IP: vip,
			},
		},
	}, nil
}

Instantiating the LoadBalancer `newLoadBalancer()`

This function is called when the cloud-provider itself is initialised and can be seen in cloud.go as part of the newThebsdboxCloudProvider() method. The newly created load balancer object once created is then added to the cloud-providers main object for use when needed.

Interface methods

`EnsureLoadBalancer`

Creates a LoadBalancer if one didn’t exist to begin with and then return it’s status (with the load balancer address)

`UpdateLoadBalancer`

Updates an existing LoadBalancer, or will create one if it didn’t exist and then return it’s status (with the load balancer address)

`EnsureLoadBalancerDeleted`

Calls GetLoadBalancer first to ensure that the load balancer exists, and if so it will delete the vendor specific load balancer. If this completes successfully then the service of type: LoadBalancer is removed as an object within Kubernetes.

`GetLoadBalancer`

This will speak natively to the vendor specific load balancer to make sure that it has been provisioned correctly.

`GetLoadBalancerName`

Returns the name of the load balancer instance.

`main.go`

This is a the standard main.go as given by the actual cloud-controller-manager example. The one change is the addition of our // OUR CLOUD PROVIDER which adds all of our vendor specific cloud-provider methods.

init() in cloud.go is called and registers our cloud-provider and the call back to our newCloudProvider() method.
The command.Execute() in main.go starts the cloud-controller-manager
The cloud-controller-manager method will look at all of the registered cloud-providers and find our registered provider.
Our provider will have its newCloudProvider() method called which sets up everything that is needed for it to be able to complete it’s tasks.
Our cloud provider is now running, when a user tries to create a resource that we’ve registered for (Load Balancers) our vendor code will be called to provide this functionality.

package main

import (
	“math/rand”
	“os”
	“time”

	“k8s.io/component-base/logs”
	“k8s.io/kubernetes/cmd/cloud-controller-manager/app”

	_ “k8s.io/component-base/metrics/prometheus/version” // for version metric registration
	// NOTE: Importing all in-tree cloud-providers is not required when
	// implementing an out-of-tree cloud-provider.
	_ “k8s.io/component-base/metrics/prometheus/clientgo” // load all the prometheus client-go plugins
	_ “k8s.io/kubernetes/pkg/cloudprovider/providers”

	// OUR CLOUD PROVIDER
	_ “github.com/user/pkg/thebsdbox/”

)

func main() {
	rand.Seed(time.Now().UnixNano())
	command := app.NewCloudControllerManagerCommand()

	logs.InitLogs()
	defer logs.FlushLogs()

	if err := command.Execute(); err != nil {
		os.Exit(1)
	}
}

Summary

Hopefully this is of some use as to how a Kubernetes cloud-provider is architected, to understand a few more examples I’ve included some other providers:

Good luck and any feedback feel free to message me

A Decade in review 2009-2019

Sat, 18 Jan 2020 16:42:40 +0000

This post is relatively delayed due to ~~laziness~~ “business reasons”, also the last post about bare-metal Kubernetes deployments took too long.

A lot of people on the twitter sphere published a decade and/or a year in review.. so I’ve raided my iCloud photo library and will attempt to put something together.

Pre-2009

Added this in because it’s relevant to where I found myself ending up in 2009..

So prior to 2009 I had been through a couple of support roles, being a DB admin and running a support team of Linux/UNIX, Windows, DBs and backup engineers inside a (B)ritish (T)elecommunications company. Following this I became a consolidation architect with Sun Microsystems (/me pours one out for SUN), with the project goal of shrinking five creaking old data centres into a shiny brand new one for the same (B)ritish (T)elecommunications company. The main goal was taking their big old servers (E10Ks, V440s etc..) and migrating their applications into Solaris Zones.

E10K with human for comparison

You might be thinking why does this matter, well somehow I’ve ended up doing the same thing over ten years later ¯\_(ツ)_/¯

Back to the timeline, Towards the end of 2008 things took a pretty grim turn and the financial crash caused a 50 person team (In September) to be down to 4 people by November at which point it became a 3 person team (me being the 4th person 😔).

2009

In mid-January I had been out of work for three months, the financial crash combined with the Christmas period meant that the job market was pretty much none-existent. I think we had enough money for a month or two of rent for where we were living and then that would be it 😐…

Then finally a random recruitment email appeared (which I was surprised to find that I still have it in my gmail😀)

Hello Daniel,

Would you be interested in a permanent role with the 
European Space Agency based in Frankfurt?

Regards

A number of interviews later I accepted the role, packed bags, booked flight to Frankfurt … and then got the bus to Darmstadt

So not actually Frankfurt after all

The European Space Agency (ESOC - Space Operations Centre) 2009-2012

I spent four crazy years living in Germany where I made a ton of friends from all over the world (Australia, Indonesia, Bulgaria, Italy, America, Spain, Greece and of course Germany). From a social perspective Germany has a fantastic outdoor lifestyle:

Endless wine and beer festivals
Dedicated cycling lanes and walking routes throughout the countryside
Excellent weather for lazy evenings
Cafes and restaurants have a focus chilling outdoors
Towards the end of the year it’s the autumn festivals and then the Christmas markets
Did I mention the beer festivals?

The work itself was pretty varied, sometimes a focus on writing reams of paperwork (100+ page guide to installing SuSE 🙄) to architecting and building platforms and systems for actual space missions. Also there was the opportunity to support some of the actual satellite missions which was always both exciting and pretty terrifying be part of.

Everyone terrifying witness the ATV (automated telemetry vehicle) dock un-assisted with ISS

Before we modernised the command centre

All brand new, (still powered by some of my hacky kernel modules)

Back to UK, SKY 2012-2014

With Kim unable really to find the role that made sense for her, we made the decision to head back to the UK as the market had started to improve. I took a role with Sky that was involved some of the heaviest infrastructure work I’ve ever been part of, but a fantastic experience none-the-less. Deploying a Disaster recovery site almost single handedly was a highlight! (Ah site B) Everything from Servers(HPE/Cisco with vSphere), to networking (Cisco Nexus switches) and storage (EMC/IBM XiV/Brocade) I got to play with and learn about. I also had the opportunity to decommission the original site, which was also fun and also a little bit on the gross side… When your cabling is “organically” grown it ultimately becomes an unfathomable under the floor mess for someone else (me) to sort out.

This took days to sort out

Suits, travel and presenting, HPE 2014 - 2017

By now Kims career had taken off, I’d spent a fantastic few years getting pretty low-level with various infrastructure vendors when one reached out to me. Approximately 48 interviews later on I’d bought a suit (mandatory) and was part of a team in HPE focussed on the merits of “converged infrastructure”!

I certainly did not choose the title on this 🙄

Over the three to four years at HPE I created a lot of powerpoint, learnt a great deal about PCI-E switching ¯\_(ツ)_/¯, memristors and a variety of other technologies. I was more than a little insecure about being the only person on the team without several CCIE certifications amongst other various industry accolades, however the entire team was 110% focussed on knowledge sharing and ensuring we all learnt from one another as much as possible (something I’m passionate about). It was here that I was asked to present in-front of an audience for the first time, which I still have nightmares about too this day… standing on stage shaking and stumbling over my words whilst trying to remember all my content about “sustainable data centre energy use” 😬. Although somehow after getting through that presentation I ended up being drafted to present more and more, to the point where I was regularly travelling around doing roadshows about various new HPE products. At some point I was asked to work on a new HPE partnership with a start up called Docker 🐳…

As part of some of the work I’d been doing at HPE around automating bare-metal, I’d stumbled across a project called Docker InfraKit and had written a plugin so that it could be extended to automate HPEs converged infrastructure. This lead to a chance email from an employee at Docker that asked if I’d be interested in participating in another project they were developing. I immediately said “yes” as it sounded super exciting to be part of! I received an email invite to a private GitHub repository to discover a new project for building Linux distributions .. to my horror the entire project was written in GO (I’d never written a single line of GO code at this point) 😱

Too embarrassed to say anything I decided to try and quickly get up to speed, I quietly worked away on simple fixes to command line flags and updating documentation. Finally I managed to write some code that “worked” and allowed the tool to build Linux images as VMDK files and run them in VMware Fusion and vSphere clusters.

A random start to a startup, Docker 2017 - 2018

Whilst still at HPE I went to DockerCon US 2017 in Austin, which turned out to be a very bonkers experience…

I had the privilege of joining the Docker Captains program, which is a group of people that are ambassadors around all things Docker.

I got to see the project I’d become part of finally be renamed (Moby -> LinuxKit) and be released to the wider community!

I made some amazing friends at Docker including Betty and Scott, amongst others :-)

Also getting to hang around with Solomon was great fun :-)

Whilst also in Austin I was asked to join the Docker team 🐳😀

It was a fantastic 18 months which involved:

Continuing working on InfraKit/LinuxKit
Working on Docker Swarm (swarmkit)
Presenting at DockerCon EU 17 and DockerCon US 18
Helping customers around EMEA successfully deploy Containers and clusters, also how to migrate their applications correctly
Building a modern app migration strategy for to Docker Enterprise Edition

36 hours in Seattle, vHeptio 2018 - now

Toward the end of the 18 months the changes in strategy had started to push me to consider other options, and after a “chance” meeting with Scott Lowe in a coffee shop in San Francisco I decided to go have a real conversation with Heptio (I’d also had a conversation with $current_employer but that’s another story). I flew from the UK to spend a day and a half in Seattle to speak with Heptio, I pretty much spent the same amount of time in airports and planes as I did in Seattle itself but it was worth it.

The Seattle Space Needle

Seattle has some amazing buildings

After accepting I became the sole “field engineer” in EMEA and ended up fiddling with, breaking and fixing Kubernetes clusters for a living :-)

Heptio was acquired by VMware in early 2019, and we’ve largely been continuing the same work.. The only change is that I’m now no longer the only person in EMEA 😀

Designing and Building HA Kubernetes on Bare-Metal

Thu, 02 Jan 2020 17:19:58 +0000

Undeterred by the memes and “thought leadership” I decided I’d finally complete this architectural overview of deploying Kubernetes on bare-metal.

To begin with we need to fix the naming.. no-one has ever deployed Kubernetes on bare-metal. You deploy Kubernetes on an Operating System, that OS can be installed on bare-metal hardware or virtual hardware etc.. (moving on).

This post ended up sprawling on much further than I ever really intended it to, so It’s broken into four main sections:

Why would I bother with bare-metal in the first place?
What does the operating environment look like, i.e. what do I get to build my cluster with?
Architectural decisions and designs when deploying a cluster on bare-metal
Actual steps to deploy the cluster… (probably what most people would care about)

One other assumption that this post takes when considering bare-metal is that the we’re typically in an on-premise environment or in an environment that doesn’t present any existing services so everything will need to be created to build the HA cluster. For further clarification about what bare-metal means you can start here

So lets begin!

Why Bare-Metal?

After having a colleague read through a draft of this post, his immediate thoughts were:

“I’m left feeling a bit ‘why would I do this’ 🙂”

It turned out that I’d not really given any actual positive reasons for running Kubernetes on Bare-Metal… So why would anyone go to the trouble?

Control

If we’re not building our Kubernetes clusters on bare-metal, then typically we’re building them in “the cloud” !

If that is the case then we typically will be presented with two options to deploy our clusters:

Managed as a service (KaaS/CaaS or whatever marketing call it these days), but effectively the cloud provider architects and owns the cluster architecture and you get access to deployed solution.
DIY, virtual hardware + virtual networking followed by a bit of rolling up the sleeves and “hey-presto”!

Both of these options are fantastic and provide a solution if you need to move quick (KaaS) or need a degree of flexibility (DIY). Given that these two examples appear to cover most use-cases why should we care?

Well mainly control and customisation.. I’m not sure if it’s just me but anything that is a prebuilt “completed” solution warrants a desire to immediately deconstruct to understand, and customise to suit needs that I didn’t know that I had.

A lot of this can crop up in cloud environments, where architectural decisions around the entire cluster (KaaS) have already been decided and can’t be changed. This can also exist even in a DIY cluster in cloud environments, where virtual machines sizes or configurations are fixed limiting cluster flexibility. It can even extend to design decisions on networking functionality or technologies to limitations on scale.

Design / Architecture and Edge

This is tied to the above point, but it is still relevant to be called out as it’s own reason. One of the most common reasons for a bare-metal requirement is that the design and architecture requirements. Often the applications that are being migrated or being re-designed to “cloud-native” principles may need access to existing systems that can’t be changed or updated. Alternatively there may be a requirement on things like FPGAs or types of GPUs that simply aren’t available within alternative environments.

One other requirement that is starting to become more and more common are edge clusters, which are typically small on-site clusters that will handle local processes in things like stores, offices, warehouses etc.. and will send the results back to central processing once complete. In most use-cases the infrastructure can be 2-3 1U servers to a stack of Raspberry PIs, all driven by things like application needs and physical space and power.

Workloads

In a lot of scenarios running a “simple” workload on bare-metal versus within a cloud environment won’t look any different. However there will be a number of application workloads that will require bare-metal:

100% predictable performance that dictate no noisy neighbours or overhead
Custom or specific hardware: Crypto, GPUs or FPGAs
Data locality and/or local laws
Application network requirements or hard coded addresses
Operational team skillset (having to learn a cloud provider technology stack)

Security and locality

Some applications that process customer, financial or personal data may (by law) have requirements that dictate that all communication paths or compute locations have to be verifiable. Unfortunately the nebulous behaviour of the cloud means that in some cases it’s not possible at the time of an audit to reliably say where customer data is located. In these situations the most common strategy is to have both the data and processing work done in an owned or co-located datacenter that meets security and legal requirements.

Also with bare-metal the onus to provide security is on the bare-metal or infrastructure operators, allowing them to be as bleeding edge when it comes to patches and updates. This is true not just for the applications, the Kubernetes cluster and the Operating System but also the hardware devices such as server firmware and networking devices. With a cloud environment the underlying infrastructure vendors or it’s vulnerabilities isn’t always commonly disclosed, meaning that you’re at the mercy of the cloud provider to handle this level of patching and fixes.

Lock-in?

Finally, lock-in … I don’t work in sales so I don’t necessarily buy this as a reason. A “well” architected solution can negate a lot of the lock-in, however when things are rushed or you opt to deeply couple with too many cloud services then you can end up in a situation where you’re dependant on this functionality and at which point you may be technically locked in, with a lot of work required to un-pick and re-build this functionality elsewhere.

…

So, with all this in mind what do we have to work with in a typical bare-metal environment ?

Operating Environment

This section will have a brief overview of what is typically available to us in a bare-metal environment, again I typically will consider bare-metal a non-hypervisor on-premises environment.

Compute hardware

With a bare-metal deployment we usually mean installing software onto hardware with no hypervisor present. In some circumstances this may or may not provide better performance or save both money or reduce complexity.

However, if we take into consideration the size of a standard enterprise class server (TBs of ram, 10s of cores/CPUs) we can start to see that we have a huge amount of computing power restrained to a single instance of both Operating System and Kubernetes use-case (worker/control plane). If we just consider the Kubernetes use-case for control plane nodes (low memory and CPU) then bare-metal servers can immediately lead to hugely under-utilised hardware.

We also need to consider the actual deployment of the Operating system, which still relies on technologies that are ~20 years old. With virtualisation we can utilise templates and OS images to rapidly deploy a node on virtual hardware. With bare-metal we need to care about the physical hardware and either take a CD to the machine or aim to deploy using a remote method.

Load-balancing

A highly available Kubernetes cluster requires a load-balancer to provide availability to the control plane in the event a node fails and to balance the load into the control plane. In a cloud environment an end-user clicks the [load-balancer] button and * magic * occurs, followed by a virtual IP that an end user will access to provide both HA and load-balancing to nodes underneath it.

When thinking about a bare-metal ~~on-premises~~ environment we have only a handful of architectural options.

Hardware (appliance) Load Balancer

If budget allows then as part of the architecture we can use a pair of load balancer appliances. We will require a pair of them to provide redundancy incase the appliance fails or requires maintenance. These appliances will typically provide the capability to create an external virtual IP address that can load balance over a number of physical IP addresses.

In some circumstances the load-balancers may provide an API or capability to integrate their functionality into a Kubernetes cluster making it much easier for applications deployed within the cluster to utilise these hardware load balancers for application redundancy.

Software Load balancers

The alternative is to use a software based load balancer which are usually simple to deploy. However in order to provide both load-balancing and high-availability then we will have to implement additional software to sit along side the software load-balancers. These two pieces of functionality are:

Virtual IP address

This functionality provides the capability of having an externally accessible IP address that can move between functioning nodes. This means that users attempting to access a service will use this VIP (virtual IP), which will always be exposed on a functioning node.

Network Service (application) Load-balancing

Load-balancing provides two pieces of functionality, it provides high-availability by ensuring that traffic is directed to a working node. It also ensures that traffic can be shared between a pool of working nodes ensuring that load is balanced. This provides the capability of having a larger amount of available service capacity than a single host, that can be scaled up by increasing nodes in the pool.

With both of these pieces of functionality in place we have a single virtual IP that will always direct us to a working load-balancer instance, which in turn will load-balance our access to the network service we want to access.

Automation

The combination of {x}aaS and virtual machines both massively simplify or obscure a lot of the implementation work for deploying servers/services.

Going back to bare-metal we suddenly are back to a large amount of very old and very hard to automate pieces of work, which I’ve previously discussed here https://thebsdbox.co.uk/2019/12/08/The-alchemy-of-turning-bare-metal-into-a-cloud/. We will typically have to follow older installation methods to do the actual provisioning such as kickstart or preseed and only after this can we look at automation such as ansible to automate the deployment of Kubernetes.

Architectural choices

This section provides design decisions that need to be considered when deploying a highly-available Kubernetes cluster on bare-metal hardware and without cloud services. Using some of these design decisions can allow you to be both more efficient in the use of modern hardware and provide a lot of the same sorts of service that people come to expect from a cloud environment!

Nodes

Node sizing typically falls into two categories:

Control-plane node(s) scaled based upon how many workers will be managed, and how much additional API requests will be hitting the manager components
Worker node(s) scaled based upon the application requirements, often clusters will have more than one application running on them. So your capacity planning will need to ensure that all requirements are captured, along with sufficient multi-tenancy and node labelling where needed

I’ve been back and forth through https://kubernetes.io, looking for anything documents that explicitly say that a control-plane node should look like {x} but I’ve drawn a blank. In the end the only sizing I can actually find is within the documentation for kubeadm, which states:

Supported OS (ubuntu/centos etc..)
2 GB or more ram
2 CPUs
Connectivity

Try finding a server with those specs without a time machine to the year 2000. In a number of engagements that I’ve been part of have consisted of server farms all built from a pre-chosen server specification. Recently in one engagement I came across 2u servers with 2x24 cores and 512GB of ram as their standard build, this meant that this relatively small Kubernetes HA control plane was underutilising:

138 cores
1530 GB of ram
Power, cooling and support/maintenance …

What are the options for ring-fencing these control-plane nodes, to allow us to use the remaining capacity …

Virtualise control-plane only

This seems a bit like a cheat, but given the reasonably small requirements for the control-plane components it does make good sense to run the control-plane nodes as virtualised machines (with the adequate resources guaranteed). Regardless of hypervisor or vmm (virtual machine manager) typically a small amount of overhead is required for the emulation of physical hardware along with minuscule performance overheads on I/O. However the benefit of freeing the remaining capacity to be used for other use-cases hugely outweighs any tiny performance or virtualisation inefficiencies.

Kubernetes resource control

In a production environment the control-plane nodes should only be running the control-plane components. This means that anything application specific is only ran on the worker nodes. The main reason that this is a recommendation or usually a best-practice is mainly down to a few key reasons:

Security: Applications strive to be secure, however we can help security by keeping the running workloads separate to the control-plane
Management and Operations: Keeping application workloads separate allows easier and clearer monitoring of both the application and the infrastructure (node behaviour)
Stability: Applications can be end-user facing and sometimes subject to Attacks or denials of service, keeping them separate will ensure that the control-plane is unaffected by these events.

There are a number of options available to us that would allow some workloads to be safely ran next to the control-plane components. All of these would involve modifying the kubelet configuration on the nodes that will be running the kubernetes management components, along with the manifests for the management pieces.

            **Would NOT recommend doing this in production**

Kubelet Reserved CPU configuration

(Optional, 1.17 required)

Modify /var/lib/kubelet/config.yaml to ensure that a set of CPUs are reserved for things like the kubelet and system daemons.

In a 24 CPU system the below addition will pin the first four CPUs for system and kubernetes daemons.
reservedSystemCPUs: 0,1,2,3

CPU Manager (1.10+ required))

This is discussed in more detail here however, this setting when enabled allows the kubelet to have more “robust” control over how CPU utilisation is allocated to pods. To enable this modify /var/lib/kubelet/config.yaml to ensure that the cpuManagerPolicy: static exists or is modified from none.

Control Plane components

In order for these components to be secured by the CPU Manager we will need to modify their Spec so that they are given the Guaranteed QoS class. We can find the manifests for the control-plane components under /etc/kubernetes/manifests and with the above configuration enabled we can modify these manifests with configuration that will tie them to resources and ensure their stability.

In order to give this QoS class we need to ensure that:

Both the requests and limits match (or that request is ommitted)
The limits for cpu are a whole number (not a fraction)

Modifying the /etc/kubernetes/manifests/kube-apiserver.yaml to have the following resource section, will bind it to a Guaranteed QoS class

resources:
  limits:
    memory: “500Mi”
    cpu: “1”

We can see this by describing the pod:

kubectl describe pods -n kube-system kube-apiserver-cp01
...
    Limits:
      cpu:     1
      memory:  500Mi
    Requests:
      cpu:        1
      memory:     500Mi
...
QoS Class:         Guaranteed

The remaining control-plane components will need modifying in the same manner in order to ensure that they all have Guaranteed resource leaving the remaining capacity for other use-cases.

Utilise this “freed” capacity

With some level of protection around the control-plane components we can look into what could make sense to run on this same infrastructure. Both of the above examples should ring-fence resources around processing capacity CPU and application memory. However the control-plane can still be impacted by things like slow I/O, in the case that something else was thrashing the same underlying storage, we could end up in a position where the control-plane components fail or etcd nodes fail due to high latency. A simple solution for this would be to ensure that these two use-cases use different underlying storage, so that neither can impact each other. One other area is system bandwidth, if this additional capacity is used by applications with high bandwidth requirements then it could potentially effect the control-plane components. Again in this scenario consider additional network interfaces that ensure that traffic is completely segregated from the control-plane traffic.

With the above in mind what other utilisation considerations could be taken into consideration?

Use-case: Ideally keeping the use-cases aligned makes it easier from a security perspective as only the same teams would need to interact with the same node.
Application load: No/never

Example workloads that *could be considered:

Logging
Dashboard
Ingress
Load-Balancing
Container registry (more for HA, if supported)
Image scanning

All of the above is a fine balance between getting more utilisation from your bare-metal servers, and ensuring that additional workload is non-impacting to the control-plane whilst introducing minimal operating overhead or security issues.

Networking (load-balancing)

This section is limited to both the networking function of load-balancing and the control-plane for kubernetes. The load-balancing for applications and services that are running within a Kubernetes cluster can be hosted elsewhere and usually is more application focused.

In the event that hardware appliances such as F5s (docs are here) are present then follow the vendor documentation for deploying that particular solution. However in the event we need to roll our own, then we will discuss the architecture decisions and options in this section.

As mentioned above, we require two components for this to be a completely HA and resilient solution:

Virtual IP, ensuring that an active load balance can always be addressed and accessed.
Load-Balancer instance, ensuring traffic is load balanced between Control-plane nodes.

The two software solutions for this that we will be using are keepalived for the Virtual IP address and HAProxy for the load-balancing.

Then finally there are two architectures that we will discuss that can be considered in order to provide a HA design:

External (to the control-plane) Load-Balancer

As with the discussion of stacked vs unstacked control plane nodes (etcd on the same nodes), we also have the architectural decision of co-locating the load-balancing components on the same nodes. This first architecture will utilise two systems external to the Kubernetes nodes to create an external load-balancer pair, that under most circumstances would in a similar manner to a load-balancing appliance.

Pros

Load-Balancer is physically separate from Kubernetes Control pane nodes
Only requires two instances/deployments of the required components
Easy to scale as not tightly coupled to the control plane nodes
Opportunity for Kubernetes services running in the cluster or other hosts to be load-balanced by the same external load-balancer

Cons

Requires it’s own hardware (or virtual hardware)
Additional management and operational overhead

Stacked (on the control-plane node(s)) Load-Balancer

The opposite the above architecture would be locate the load-balancer components locally to the Kubernetes control plane components. Simplifying the architecture, but creating it’s own architectural challenges.

Pros

Simplifies deployment, control plane nodes are always deployed with all of the required components
Scaling becomes simplified, going from 3 -> 5 control plane nodes involves the deployments of 2 more control plane nodes and adding into load balancer/VIP configuration
Reduces the infrastructure requirement as load-balancing shares the same infrastructure as the control plane components
Possibility of one-day having kubeadm manage the load-balancing lifecycle in the same way that it now manages etcd

Cons

Tight coupling, can result in hard to debug networking, performance or configuration issues.
An issue with either of the load-balancing components could have a knock-on effect to the control pane nodes without proper precautions in place.

Stacked Port conflicts

Another thing that can appear confusing with co-locating the load-balancers on the same node(s) as the Kubernetes API control plane components are the port configurations. We can’t have two applications listening on the same port, which means if we try to configuration the load-balancer to expose its services on the standard Kubernetes API-server port (6443) then we can’t also have the api-server trying to use the same port.

The architecture diagram below depicts the load-balancer binding to port 6443 and sending requests to one of the API-Servers listening on port 6444:

In order to allow the API-server to behave as expected we expose the load-balanced API-servers through the standard port (6443) and configure all of the API-servers to bind to the port (6444). This means that there are no port conflicts, but can cause confusion when trying to debug issues e.g. if the load-balancer isn’t behaving as expected or is down then to connect to a specific Kubernetes API-server we need to remember to connect to controlplane0{X}:6444. Finally when using kubeadm to perform the installation we need to do two additional things:

Use the VIP and load-balancer port 6443 --control-plane-endpoint “$VIP_ADDRESS:6443”
Ensure that the actual Kubernetes API-server binds to port 6444 --apiserver-bind-port 6444

Note: The --apiserver-bind-port is required for initialising the first control plane and joining of additional control plane nodes.

The outlier (worker) architecture

It could be possible to have the load-balancing components deployed first on identified worker nodes, which would free up any risk of the either workload causing performance issues or outages. However in this scenario we break the concept of workers being something we care about ( ~~cattle vs pets~~ , whatever the analogy is for not having to worry too much about the infra). If workers are suddenly tied to particular workloads that we need to deeply worry about, then we’re in a position where we’ve broken the model to easily replace, scale and destroy our worker pools.

The other issue is that we would need the workers in place before we can actually deploy our control plane, which introduces more chicken and egg architectural design choices.

With all this in mind, lets look at how we actually would get this deployed …

Getting Kubernetes deployed on Bare-Metal

This section will detail the majority of steps that are required to deploy the load-balancing and the Control-plane components of a cluster in a typical bare-metal environment. Worker nodes aren’t covered, but as this deployment will be making use of kubeadm then the workers will be managed through the kubeadm join command.

For workloads that target specific nodes (e.g. nodes with lower core higher frequency for single thread or GPU(s)) then we should label the nodes and use a nodeSelector in the spec as detailed here.

Finally, these steps are for Ubuntu.. If you’re deploying on some-other distro the steps or package names shouldn’t be too different in order to install and configure in the same manner.

The Infrastructure

In this deployment, we will be deploying a stacked architecture which means that the control plane nodes will have everything installed on each node:

Kubernetes Control plane components
etcd
Load-Balancer components

The nodes will consist of three Ubuntu 16.04 or 18.04 standard installations with a user that has Sudo installed.. How that OS gets there, for remote installations I can highly recommend plunder (“Shameless plug”). With the bare-metal nodes up, Operating Systems installed and permanent network addresses given we need to consider one final step (due to the VIP), the network addressing for our cluster.

In our deployment we’ll manage this address scheme quite simply, with the first address for the VIP and all subsequent addresses following in parallel.

Example IP address table

Node	Address
VIP	10.0.0.100
controlPlane01	10.0.0.101
controlPlane02	10.0.0.102
controlPlane03	10.0.0.103

The Kubernetes install one-liner

With our infrastructure in place, we can now begin the installation of Kubernetes! The installation will be following the steps from https://kubernetes.io using kubeadm. However a much more condensed version is below, I like to refer to it as a one-liner as it is technically a one-liner(ish)…

Once the version is set then the next step will update Ubuntu, update repositories and install all of the components that we need to install a HA Kubernetes cluster.

Set the version

This environment variable is used to determine the version of Kubernetes that will be installed by the below one-liner.

export kVersion="1.17.0-00"

Install all the packages

This will install “everything” needed (on ubuntu) for a control-plane node with load-balancing and high-availability.

sudo -E sh -c '
apt-get update; \
apt-get install curl apt-transport-https gnupg-agent ca-certificates software-properties-common -y; \
echo "deb https://download.docker.com/linux/ubuntu xenial stable" >> /etc/apt/sources.list.d/docker.list; \
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" >> /etc/apt/sources.list.d/kubernetes.list; \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - ; \
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - ; \
apt-get update; \
apt-get install -y ethtool socat ebtables conntrack libnetfilter-conntrack3 docker-ce=18.06.1~ce~3-0~ubuntu kubelet=$kVersion kubeadm=$kVersion kubectl=$kVersion kubernetes-cni cri-tools haproxy keepalived; \
systemctl enable kubelet.service
'

Once the above is complete the system is ready to be configured, however ensure that this is repeated on all of the control plane nodes.

Load-balancing the load-balancing

Virtual IP / KeepaliveD configuration

Below are the important sections for the /etc/keepalived/keepalived.conf that will be required on each of the nodes.

Global Definition of the keepalived configuration

global_defs {
   # Name of VIP Instance
   router_id KubernetesVIP
   
   # Enable SNMP Monitoring (Optional)
   # enable_traps
}

Virtual Service Definition (this is where we define the configuration of our VIP)

As mentioned in the comments on the example configuration below, typically the first node 01 we need to set the state to MASTER. The means that on startup that this node will be the node be allocated the VIP first. The priority number is used during the keepalived cluster elections to determine who will become the next MASTER and the highest priority wins.

For further details on these configurations the keepalived documentation can be found here.

vrrp_instance APIServerVIP {
    # Interface to bind to
    interface ens192

    # This should be set to MASTER on the first node and BACKUP on the other two
    state MASTER
    # This should be 50+ lower on the other two nodes to enable the lead election
    priority 100

    # Address of this particular node
    mcast_src_ip $node_IP

    # A unique ID if more than one service is being defined
    virtual_router_id 61
    advert_int 1
    nopreempt

    # Authentication for keepalived to speak with one another
    authentication {
          auth_type PASS
          auth_pass $bloody_secure_password
    }

    # Other Nodes in Cluster
    unicast_peer {
        $other_node_IP
        $other_node_IP
    }

    # Kubernetes Virtual IP
    virtual_ipaddress {
        10.0.0.100/24
    }

    # Health check function (optional)
    #track_script {
    #    APIServerProbe
    #}
}

Health check (optional)

This can be used to determine if the Kubernetes API server is up and running, if not fail the VIP to another node, but with the load-balancer performing the same task it’s not a requirement.

vrrp_script APIServerProbe {
    # Health check the Kubernetes API Server
    script "curl -k https://$node_IP:6443"
    interval 3
    timeout 9
    fall 2
    rise 2
}

Control-plane load-balancing / HAProxy configuration

Below are the additions to the /etc/haproxy/haproxy.conf that will be there by default, ensure you back up the original before modifying and then append the configuration below. As mentioned here we need to remember that the frontend will expose itself on port 6443, and it will load-balance to the kubernetes API-servers listening on port 6444.

frontend http_stats
   bind *:8080
   mode http
   stats uri /haproxy?stats

frontend kube_api_server
    bind 0.0.0.0:6443
    mode tcp
    option tcplog
    timeout client  10800s
    default_backend controlPlanes

backend controlPlanes
    mode tcp
    option tcplog
    balance leastconn
    timeout server  10800s
    server controlPlane01 10.0.0.101:6444 check
    server controlPlane02 10.0.0.102:6444 check
    server controlPlane03 10.0.0.103:6444 check

Install Kubernetes Control-Plane

With the VIP and load-balancer up and running, we can now install the kubernetes control plane components in a HA configuration using kubeadm.

First node

On the first node we will initialise our Kubernetes cluster using the kubeadm init command with these additional flags.

sudo kubeadm init \
--control-plane-endpoint "10.0.0.100:6443" \
--apiserver-bind-port 6444 \
--upload-certs

If we wanted to use the flags mentioned above in the section Kubernetes resource control then we can use kubeadm to print out all of the configuration yaml and we can edit the sections that are identified using the kind: key.

Step one: Print configuration and save configuration

kubeadm config print init-defaults --component-configs KubeletConfiguration > config.yaml

Step two: Edit InitConfiguration

We need to remove the advertiseAddress as it defaults to a ridiculous default (not sure why), and edit the bindPort to 6444 as this is what the API-Server needs to listen on in order to not conflict with the load-balancer.

Step three: Edit ClusterConfiguration

We need to add the line controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" to this section so that it uses our VIP address and uses the port of the load balancer.

Step four: Edit KubeletConfiguration

If we want to use the ReservedCPU functionality then add the line: reservedSystemCPUs: 0,1 (edit for the number of CPUs to save).

If we want to have Guaranteed QOS for our Pods then we can add the line: cpuManagerPolicy: static.

We can then apply our configuration with the command:

1	sudo kubeadm init --config ./cluster.yaml

This will initialise our control plane on the first node and print out the subsequent join command for other control plane nodes:

  kubeadm join 100.0.0.100:6443 \
--token fg55b4.o5n57eyna50627dc \
--discovery-token-ca-cert-hash sha256:<massive hash>  \
--control-plane --certificate-key <massive key>

Remaining Nodes

When joining the remaining nodes to the cluster, ensure that you add the --apiserver-bind-port 6444 to ensure that the Api-server binds on a different port to the load balancer sitting above it.

With the VIP and load-balancers up and running, along with our cluster initialised we can now add in our additional nodes and build out a HA/Load-balancer Kubernetes cluster.

$ kubectl get nodes
NAME             STATUS   ROLES    AGE     VERSION
controlplane01   Ready    master   2m33s   v1.17.0
controlplane02   Ready    master   108s    v1.17.0
controlplane03   Ready    master   56s     v1.17.0

To test the cluster, we can stop and start the VIP with sudo systemctl stop keepalived and ensure that kubectl get nodes continues to act as expected. Rebooting of nodes will also create the same experiences as having node failures. We should be able to see logs showing that keepalived is moving our VIP to working nodes and ensuring that access always remains into the running cluster.

Considerations for an unstacked load-balancer

The above guide details all of the steps required to build a HA Kubernetes cluster that has the load-balancing components co-located on the same nodes as the Kubernetes components. If we wanted to build an external or unstacked load-balancing pair of nodes then the process is very similar and covered in brief below.

On both of the load-balancing nodes (loadBalancer01/02) we will need to install the components for the load-balancing:

1	apt-get install -y haproxy keepalived

With these components installed on both nodes we can now configure them in the exact same way as before except. The two load-balancing nodes will need the VIP and HAProxy configured in an almost identical manner, however the only consideration is the ports of the Kubernetes API server. As the load-balancer and the Kubernetes API-Server will be on different nodes they won’t have to deal with a port conflict, this means that we can leave the API-Server listening on power 6443 on the controlPlane01/02/03 nodes.

Using Nginx instead of HAProxy

The above examples all use HAProxy as it’s been my default, however Nginx can also be used as a load-balancer that sits between the VIP and the Kubernetes API-server. Below is an identical configuration to the HAProxy configuration above that can be appended to /etc/nginx/nginx.conf:

events { }

stream {
    upstream apiServer_backend {
        least_conn;
        # controlPlane01
        server 100.0.0.101:6444;
        # controlPlane02
        server 100.0.0.102:6444;
        # controlPlane03
        server 100.0.0.103:6444;
    }
       
    server {
        listen        6443;
        proxy_pass    apiServer_backend;
        proxy_timeout 3s;
        proxy_connect_timeout 1s;
    }
       
}

Note: The above is a stacked example that has the load-balancer binding to port 6443 and using the Kubernetes API-servers that are bound to port 6444.

Feedback

Either write something in the comments below or hit me up at @thebsdbox on twitter :-)

thebsbdox

Adding a gateway to kube-vip

kube-gateway (v0.0001)

The gateway watcher

Annotating your pod for mTLS 🔐

Annotating your pod for AI 🤖

Annotate your LLM

Annotate your AI workloads

eBPF, mTLS and AI oh my..

What’s next?

mTLS

AI workloads

Egress v2 in kube-vip

iptables & nftables

The state of iptables & nftables

What, you want some details.. fine

When is iptables not iptables

So why is no-one writing nftables rules?

The (finally) growing ecosystem for nftables

A pure Go package for manipulating nftables 🎉

Egress v2 finally 🥹!

Egress v2 tables and chains

The TL;DR

Coming soon!

Continuing building your service mesh

The proxy

What is in a proxy

Proxy startup

Proxy running lifecycle

Creating Certificates 📝

The final piece is the injector 💉

In Summary

What next..

Building your own service mesh

The service mesh shopping list

Traffic redirector 🚦

The Proxy

The Injector 💉

Certificates 📝

Lets get started 🐝

The eBPF 🐝 magic 🪄

Why do we pass the pid of the proxy into the eBPF program? (I hear you ask)

Abridged logs

Summary

Egress

Kubernetes Networking

Kube-Proxy

Egress Magic 🪄

Egress with kube-vip 🐙

The same, but backwards

Where is the eBPF 🐝

“But isn’t there an egress hook for TC?”

Thanks

Highly Available workloads across Mulitple Kubernetes Clusters

High Availability

Implementing High Availability (back in the day)

Shared storage

Re-architecting the application

Networking

Quorum

Did HA work?

So what went wrong?

High Availability in Kubernetes

High Availability across Kubernetes

Perfecting Protocol Parsing (Probably) with eBPF

Protocols

The joys of padding

Getting data from the skb

Understanding application data

Modify the BGP data

Outro

Application traffic with eBPF

The maps!

The eBPF programs!

Application Data !!

HTTP Data :-)

Find the HTTP method

Acting on the HTTP application request

Conclusion

eBPF adventures in networking

Why do we pass the `pid` of the proxy into the eBPF program? (I hear you ask)

Getting data from the `skb`

`XDP` vs `TC`, or even `sysprobes`

Create the project with `kubebuilder`

Edit the Directions Specification `DirectionsSpec`

Edit the Directions Status `DirectionsStatus`

`Sheffield.yaml`