Kube-Vip and BGP loadbalancers with Unifi

Posted on Edited on Disqus:

Since starting at Packet Equinix Metal i’ve had to spend a bit of time getting my head around a technology called BGP. This is widely used at Equinix Metal, the main use case is to allow an Elastic IP address to route traffic externally (i.e. from the internet) into one or more physical machines in Equinix’s facilities. An external address will usually be routed into one of the Equinix datacenters, we can configure physical infrastructure to use this external IP and communicate to the networking equipment in this facility (using BGP) that traffic should be routed to these physical servers.

I’ve since been working to add BGP functionality to a Kubernetes load-balancing solution that i’ve been working on in my spare time kube-vip.io. This typically works in exactly the same method as described as above, where an EIP is advertised to the outside world and the BGP functionality will route external traffic to worker nodes and, where the service traffic is then handled by the service “mesh” (usually iptables rules capturing service traffic) within the cluster.

I’m hoping to take this functionality further and decided it would be nice to try and emulate the Equinix Metal environment as much as possible, i.e. having BGP at home. There are two methods that I could go down in order to have this sort of functionality within my house:

  • Create a local linux router, with it’s own network and use bgpd so I can advertise routes
  • Utilise my Unifi router, which after some light googling it turns out supports bgp

Given that:
a. It’s Sunday
b. I’m Lazy
c. The instructions for setting up a local linux router looked like a pita I opted to just use my USG.
d. The USG will enable me to advertise addresses on my existing network and the USG will route traffic the advertising hosts without me doing anything (exacerbating the laziness here)

Configuring the Unifi Security Gateway

These instructions are for the Unifi Security Gateway, however I suspect that the security gateway that is part of the “dream machine” should support the same level of functionality.

Before we begin we’ll need to ensure that we understand the network topology of the network so that we can configure for bgp to function as expected.

Device Address
gateway 192.168.0.1
…
k8sworker01 192.168.0.70
k8sworker02 192.168.0.71
k8sworker03 192.168.0.72

In the above table we can see a subset of hosts on my network, the first host that is important is the gateway address on my network (which is the address of the USG). All hosts on my network have this set as the default gateway, which means that when a machine needs to access an IP address that isn’t 192.168.0.1-254 then it will send the traffic to the gateway for it to be routed to that specific address.

Your gateway can be found by doing something like the following:

1
2
$  ip route | grep default
default via 192.168.0.1 dev ens160 proto static

The other addresses that are worth noting are the three worker hosts in my Kubernetes cluster that will advertise their bgp routes to the gateway.

NOTE:
All of the next steps require ssh access into the appliance in order to enable and configure the USG so that bgp is enabled on the network.

If you don’t know your ssh password for your gateway then help is at hand.. because for some bizarre reason you can get the ssh username/password straight from the web UI.. Navigate to Network Settings -> Device Authentication and here you’ll find the ssh username and password.

NOTE 2:
Whilst it is possible to enable bgp functionality through the cli, the actual web client isn’t aware of the bgp configuration. This will result in the bgp configuration being wiped when you do things like modify firewall rules or port forwarding in the web UI.

Enabling BGP

To begin with we’ll need to ssh to the Unifi USG with the credentials that can be found from the web portal.

Below is the configuration that we’ll add, i’ll break it down below:

1
2
3
4
5
6
7
8
configure
set protocols bgp 64501 parameters router-id 192.168.0.1
set protocols bgp 64501 neighbor 192.168.0.70 remote-as 64500
set protocols bgp 64501 neighbor 192.168.0.71 remote-as 64500
set protocols bgp 64501 neighbor 192.168.0.72 remote-as 64500
commit
save
exit

configure - will enable the configuration mode for the USG.

set protocols bgp 64501 parameters router-id 192.168.0.1 - will enable bgp on the USG with the router-id as it’s IP address 192.168.0.1 the AS number 64501 is used to determine our particular bgp network

set protocols bgp 64501 neighbor x.x.x.x remote-as 64500 - will allow our bgp instance to take advertised routes from x.x.x.x with the AS identifier 64500

commit | save - will both enable and then save the configuration

This is it.. the USG is now configured to take routes from our Kubernetes workers!!!

Configuring Kube-Vip

I’ve recently started on a release of kube-vip that will provide additional functionality in order to automate some of the bgp configuration, namely the capability to determine the host IP address for the bgp server-id for advertising from a pod.

We can follow the guide for setting up kube-vip with bgp with some additional modifications show below!

Ensure that the version of the kube-vip image is 0.2.2 or higher

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
env:
  - name: vip_interface
	value: "lo"
  - name: vip_configmap
	value: "plndr" 
  - name: bgp_enable
	value: "true"
  - name: vip_loglevel
	value: "5"
  - name: bgp_routerinterface
	value: "ens160"
  - name: bgp_as
	value: "64500"
  - name: bgp_peeraddress
	value: "192.168.0.1"
  - name: bgp_peeras
	value: "64501"

The bgp_routerinterface will autodetect the host IP address of the interface specified and this will become our server-id for each pods bgp peering configuration. The bgp_peer<as|address> is the remote configuration for our USG as specified above.

Testing it out !

Create a easy deployment, we can use the nginx demo for this:

1
kubectl apply -f https://k8s.io/examples/application/deployment.yaml

Given that the USG is our default gateway we can actually advertise any address we like as a type:LoadBalancer and when we try and access if from inside the network the USG will route us back to the the Kubernetes servers!

For example we can create a load-balancer with the address 10.0.0.1

1
kubectl expose deployment nginx-deployment --port=80 --type=LoadBalancer --name=nginx-bgp --load-balancer-ip=10.0.0.1

Given the home network is in the 192.168.0.0/24 range anything that is outside of it will need to go through our USG router, where we’ve advertised (through BGP) that traffic needs to come to any of the Kubernetes workers.

We can see this below on the USG with show ip bgp

1
2
3
4
5
6
7
8
9
10
admin@Gateway:~$ show ip bgp
BGP table version is 0, local router ID is 192.168.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
			  r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
* i10.0.0.1/32      192.168.0.45                  100      0 i
*>i                 192.168.0.44                  100      0 i
* i                 192.168.0.46                  100      0 i