Kube-Vip and BGP loadbalancers with Unifi
Since starting at
Packet Equinix Metal i’ve had to spend a bit of time getting my head around a technology called BGP. This is widely used at Equinix Metal, the main use case is to allow an Elastic IP address to route traffic externally (i.e. from the internet) into one or more physical machines in Equinix’s facilities. An external address will usually be routed into one of the Equinix datacenters, we can configure physical infrastructure to use this external IP and communicate to the networking equipment in this facility (using BGP) that traffic should be routed to these physical servers.
I’ve since been working to add BGP functionality to a Kubernetes load-balancing solution that i’ve been working on in my spare time kube-vip.io. This typically works in exactly the same method as described as above, where an EIP is advertised to the outside world and the BGP functionality will route external traffic to worker nodes and, where the service traffic is then handled by the service “mesh” (usually
iptables rules capturing service traffic) within the cluster.
I’m hoping to take this functionality further and decided it would be nice to try and emulate the Equinix Metal environment as much as possible, i.e. having BGP at home. There are two methods that I could go down in order to have this sort of functionality within my house:
- Create a local linux router, with it’s own network and use
bgpdso I can advertise routes
- Utilise my Unifi router, which after some light googling it turns out supports
a. It’s Sunday
b. I’m Lazy
c. The instructions for setting up a local linux router looked like a
pita I opted to just use my USG.
d. The USG will enable me to advertise addresses on my existing network and the USG will route traffic the advertising hosts without me doing anything (exacerbating the laziness here)
These instructions are for the Unifi Security Gateway, however I suspect that the security gateway that is part of the “dream machine” should support the same level of functionality.
Before we begin we’ll need to ensure that we understand the network topology of the network so that we can configure for
bgp to function as expected.
In the above table we can see a subset of hosts on my network, the first host that is important is the gateway address on my network (which is the address of the USG). All hosts on my network have this set as the default gateway, which means that when a machine needs to access an IP address that isn’t
192.168.0.1-254 then it will send the traffic to the gateway for it to be routed to that specific address.
Your gateway can be found by doing something like the following:
$ ip route | grep default
The other addresses that are worth noting are the three worker hosts in my Kubernetes cluster that will advertise their
bgp routes to the gateway.
All of the next steps require
ssh access into the appliance in order to enable and configure the USG so that
bgp is enabled on the network.
If you don’t know your ssh password for your gateway then help is at hand.. because for some bizarre reason you can get the ssh username/password straight from the web UI.. Navigate to
Network Settings ->
Device Authentication and here you’ll find the ssh username and password.
Whilst it is possible to enable
bgp functionality through the cli, the actual web client isn’t aware of the
bgp configuration. This will result in the
bgp configuration being wiped when you do things like modify firewall rules or port forwarding in the web UI.
To begin with we’ll need to
ssh to the Unifi USG with the credentials that can be found from the web portal.
Below is the configuration that we’ll add, i’ll break it down below:
configure - will enable the configuration mode for the USG.
set protocols bgp 64501 parameters router-id 192.168.0.1 - will enable
bgp on the USG with the router-id as it’s IP address
192.168.0.1 the AS number
64501 is used to determine our particular
set protocols bgp 64501 neighbor x.x.x.x remote-as 64500 - will allow our
bgp instance to take advertised routes from
x.x.x.x with the AS identifier
commit | save - will both enable and then save the configuration
This is it.. the USG is now configured to take routes from our Kubernetes workers!!!
I’ve recently started on a release of
kube-vip that will provide additional functionality in order to automate some of the
bgp configuration, namely the capability to determine the host IP address for the
server-id for advertising from a pod.
We can follow the guide for setting up kube-vip with bgp with some additional modifications show below!
Ensure that the version of the
kube-vip image is
0.2.2 or higher
bgp_routerinterface will autodetect the host IP address of the interface specified and this will become our
server-id for each pods
bgp peering configuration. The
bgp_peer<as|address> is the remote configuration for our USG as specified above.
Create a easy deployment, we can use the
nginx demo for this:
kubectl apply -f https://k8s.io/examples/application/deployment.yaml
Given that the USG is our default gateway we can actually advertise any address we like as a
type:LoadBalancer and when we try and access if from inside the network the USG will route us back to the the Kubernetes servers!
For example we can create a load-balancer with the address
kubectl expose deployment nginx-deployment --port=80 --type=LoadBalancer --name=nginx-bgp --load-balancer-ip=10.0.0.1
Given the home network is in the 192.168.0.0/24 range anything that is outside of it will need to go through our USG router, where we’ve advertised (through BGP) that traffic needs to come to any of the Kubernetes workers.
We can see this below on the USG with
show ip bgp
admin@Gateway:~$ show ip bgp