EVO:RAIL – LoudMouth aka Zeroconf

What is Zeroconf?

Zeroconf was first proposed in November 1999 and finalised in 2003 and has found the largest adoption in Mac OS products, nearly all networked printers and other network device vendors. The most obvious and recognisable implementation of zeroconf is bonjour, which has been part of Mac OS since version 9 and is used to provide a number of shared network services. The basics of Zeroconf are explained quite simply on zeroconf.org with the following (abbreviated statement) “making it possible to take two laptop computers, and connect them … without needing a man in a white lab coat to set it all up for you”.

Basically zeroconf allows a server/appliance or client device to discover one another without any networking configuration. It is comparable to DHCP in some regards in that a computer with no network configuration can send out a DHCP request (essentially asking to be configured by the DHCP server), the response will be an assigned address and further configuration allowing communication on the network. Where it differs is that zeroconf also allows for advertisement of services (time capsule, printer services, iTunes shared libraries etc.), it also can advertise small amounts of data to identify itself as a type of device.

A Time machine advertisement over zeroconf: (MAC address removed)

[dan@mgmt ~]$ avahi-browse -r -a -p -t | grep TimeMachine
+;eth0;IPv4;WDMyCloud;Apple TimeMachine;local
=;eth0;IPv4;WDMyCloud;Apple TimeMachine;local;WDMyCloud.local;;9;"dk0=adVN=TimeMachineBackup,adVF=0x83" "sys=waMA=00:xx:xx:xx:xx:xx,adVF=0x100"

How the EVO:RAIL team are using Zeroconf

From recollection of the deep-dive sessions, I may have mistaken the point (corrections welcome).

Zeroconf has found the largest adoption in networked printers and apple bonjour services, however in the server deployment area a combination of DHCP and MAC address matching is more commonly used (Auto deploy or kickstart from PXE boot).

The EVO:RAIL team have implemented a Zeroconf daemon that lives inside every vSphere instance and inside the VCSA instance. The daemon inside the VCSA wasn’t really explained however the vSphere daemon instances allow the EVO:RAIL engine to discover them and take the necessary steps to automate their configuration.

Implementing Zeroconf inside vSphere(esxi)

The EVO:RAIL team had to develop their own zeroconf daemon named loudmouth that is coded entirely in python. The reason behind this was explained in one of the technical deep dives, the problem being that the majority of pre-existing zeroconf implementations have dependancies on various linux shared libraries.

/lib # ls *so | wc -l
/lib # uname -a
VMkernel esxi02.fnnrn.me 5.5.0 #1 SMP Release build-1331820 Sep 18 2013 23:08:31 x86_64 GNU/Linux
[dan@mgmt lib]$ ls *so | wc -l
[dan@mgmt lib]$ uname -a
Linux mgmt.fnnrn.me 3.8.7-1-ARCH #1 SMP PREEMPT Sat Apr 13 09:01:47 CEST 2013 x86_64 GNU/Linux

As the quick example above shows (32bit libs) a vSphere instance contains only a few elf based libraries providing a limited subset of shared functionality. This means that whilst elf based binaries can be moved from a linux distribution over to a vSphere instance, the chance is that a requirement on a shared library won’t be met. Further more building a static binary possibly won’t help as the VMKernel (VMwares kernel implementation)doesn’t implement the full set of linux syscalls, which makes sense as it’s not an OS implementation the userland area of the vSphere is purely for management of the hypervisor. The biggest issue that an implementation of zeroconf which relies on UDP and datagrams is the lack of implementaion of IP_PKTINFO.

This rules out avahi, Zero Conf IP (zcif), and linux implementations of mDnsResponder.

What about loudmouth?

Unfortunately it is yet to be said if any components of EVO:RAIL will be open sourced or back ported to vSphere, so whilst VMware have a zeroconf implementation for vSphere it is likely it will remain proprietary.

What next…

I’ve improved on where I’ve been with my daemon, however i’m hoping to upload it to github sooner rather than later. Unfortunately work has occupied most of the weekend and most evenings so far .. that tied with catching up on episodes of elementary and dealing with endless segfaults as I add any simple functionality have slowed the progress more than I was expecting.

Also I decided to finish writing up this post, which took most of this evening 😐

Debugging on vSphere

A summary of what to expect inside vSphere can be read here and there is no point duplicating existing information (http://www.v-front.de/2013/08/a-myth-busted-and-faq-esxi-is-not-based.html). More importantly when dealing with the vSphere userland libraries or more accurately lack of, then the use of strace is hugely valuable. More details on strace can be found here (http://dansco.de/doku.php?id=technical_documentation:system_debugging).

EVO:RAIL – “Imitation is the sincerest form of flattery”

EVO:RAIL Overview

At VMworld ’14 I managed to catch a few excellent sessions on EVO:RAIL including the deep dive session + Q&A that really explained the architecture that makes up the EVO:RAIL appliance. The EVO:RAIL appliance is only built by authorised vendors or as VMware call them QEPs (Qualified EVO:RAIL Partners) from a hardware specification that is determined by VMware.

VMW-LOGO-EVO-Rail-108Also the EVO:RAIL software/engine is provided to the QEPs for them to install and requires a built in hardware ID (also provided by the QEPs) in order for the engine to work correctly. This means currently that the EVO:RAIL appliance is a sealed environment that has only been designed to exist on pre-determined hardware and that anyone wanting to use any of this functionality on their pre-existing infrastructure will be unable to do so.

So what are the components of EVO:RAIL:
  • QEP hardware (a 2U appliance that has four separate compute nodes)
  • Loudmouth, a zeroconf daemon written in python that also detects the hardware ID
  • EVO:RAIL engine that consists of python/bash scripts, a comprehensive UI and automation ability for deployment
  • VCSA appliance (contains loudmouth, EVO:RAIL) this is pre-installed on one node in every appliance.
How is it built and how is it configured:EVO Rail UI

The idea is that a customer will speak to their account manager at a QEP place an order on a single SKU and provide some simple configuration details. The vendor then will pre-provision the four nodes with the EVO:RAIL version of vSphere, one of these nodes will be also provisioned with the VCSA appliance (also the EVO:RAIL version). The VCSA node will be configured with some IP addresses provided by the customer so that they can complete the configuration once the appliance has been racked.  The EVO:RAIL engine combined with the loudmouth daemon will detect the remaining nodes in the appliance and allow them to be configured, the same goes for addition appliances (maximum 4) that are added.

This simplified UI that was crafted by Jehad Affoneh and provides a HTML5 + web sockets interface that provides real-time information for the end-user as they complete the EVO:RAIL configuration. Once the initial configuration (networking/passwords etc.) is complete the EVO engine will then handle the automation of the following tasks:

  1. vSphere instance configuration (hostnames, passwords, network configuration)
  2. Deploy and configure the VCSA (once complete add in the vSphere instances, and configure VSAN)
  3. Tidy up
  4. Re-direct user to the EVO:RAIL simplified interface for VM deployment.



The final screen that the user is presented with is the EVO:RAIL simplified interface, this is a “reduced functionality” user interface that allows people to complete simple tasks such as deploy a simple VM (from simplified pre-determined parameters such as sizing) or investigate the health of vSphere host. The “real” management interface i.e. vCenter is still there in the back ground and the EVO:RAIL interface still has to interact with it through the awful vCenter SOAP SDK (which hopefully will change in the next releases, thus requiring a re-write for the EVO engine).This vCenter can still be accessed through the URL on port 9443, direct with the infrastructure client or alternatively there is a small link in the EVO interface in the top right hand corner.

What next?

EVO:RAIL has been described by JP Morgan with – “EVO products could be an iPhone moment for enterprise use of cloud computing”. I see this in two ways:

Incredible simplification and ease of use, deployment of four nodes is meant to take no more that fifteen minutes. The simplified interface for VM deployments takes 3-4 clicks and your virtual machine is ready to use. The use of the LoudMouth service truly makes deployment plug and play as more capacity is added.

The walled garden, the comparison to the iPhone covers this point perfectly as this product is a closed off product and only available to authorised partners. There are some really clever design ideas here that could really be expanded on and back ported to the existing vSphere to provide some really great functionality.

  • Large scale deployment with the use of the LoudMouth daemon for discovery
  • Disaster recovery would be simplified again via the LoudMouth daemon advertising where virtual machines are located (in the event that vCenter doesn’t auto re-start after failure).


After speaking with the designer of the UI and sitting through the deep dive sessions there were a few design decisions or “short cuts” that had to be taken in order to get functionality to work, I decided to see what I could improve or at least imitate in my vSphere lab. To begin with I started with the zeroconf agent and how it could be implemented or improved upon, in EVO:RAIL this had to be written from scratch in python due to the development team not managing to get any pre-existing solution (which is understandable AVAHI is hideous and has dependencies on everything).

So I introduce “boaster” which is a tiny daemon written in C that handles simple mDNS service announcement, it’s only a PoC however I intend to add more functionality in the next few days. At the moment a vSphere hypervisor will advertise itself and it’s DHCP address to a bonjour browser or avahi-browse..



.. More to follow.

Layer 2 over Layer 3 with vSwitch and Mikrotik virtual routers

I’ve trialled a number of different ideas to have a number of different vSwitches that have virtual machines attached, when dealing with a vSphere host that had a single interface. The problem lies in that only one of your vSwitches has a physical interface (uplink) present, which obviously means that traffic can go between the virtual machines on that vSwitch but can’t go northbound to other devices on the network. I decided to give the Mikrotik virtual router a go as it’s requirements are so tiny it doesn’t have a noticeable footprint on my small infrastructure (the virtual routers require 64MB of ram).

Using the two software routers it is possible to bridge interfaces on numerous vSwitches and then use EoIP to create another layer 2 bridge northbound over layer 3. In a simple example we will use two simple vSphere hosts (01 / 02), in real life both are a pair of Gigabyte Brix hosts that whilst good for small lab environments only have a single Gigabit interface. This is limiting with regards to what network based lab environments you can put together as any vSwitch that doesn’t have a physical interface can’t broadcast traffic anywhere other than inside that vSwitch and having different configurations on each host means that vmotion will break the hosts network connectivity.

Below is the configuration I currently have:

layer2 over layer3


Although not explicitly mentioned in the diagram the interface on vSwitch0 is ether1, this interface will be on the same vSwitch that has a physical interface and thus will allow outbound traffic from the esxi host. This interface will need configuring to enable connectivity to the switch and also to route out to the internet (if required).

 Configuring router01

Configure ether1

Enable the interface and assign a reachable address (

/interface enable ether1
/ip address add address= interface=ether1 comment="External Interface"

Also add another interface that will be used as an EoIP end point.

/ip address add address= interface=ether1 comment="EoIP endPoint"

Adding a default gateway ( which is most peoples router.

/ip route add dst-address= gateway=

Create an Ethernet over IP interface

This EoIP interface is required to encapsulate layer2 frames into layer3 packets that can be routed etc..

/interface eoip add comment="eoip interface" name="eoip01" remote-address= tunnel-id=1

Create a bridge and add interfaces

The bridge is required for allowing layer2 traffic between interfaces that will sit on the different vSwitches.

/interface bridge add comment="Bridge between vmnics" name=esx-bridge protocol-mode=rstp
/interface bridge port add bridge=esx-bridge interface=eoip01
/interface bridge port add bridge=esx-bridge interface=ether2


 Configuring router02

Configure ether1

Enable the interface and assign a reachable address (

/interface enable ether1
/ip address add address= interface=ether1 comment="External Interface"

Also add another interface that will be used as an EoIP end point.

/ip address add address= interface=ether1 comment="EoIP endPoint"

Adding a default gateway ( which is most peoples router.

/ip route add dst-address= gateway=

Create an Ethernet over IP interface

This EoIP interface is required to encapsulate layer2 frames into layer3 packets that can be routed etc..

/interface eoip add comment="eoip interface" name="eoip01" remote-address= tunnel-id=1

Create a bridge and add interfaces

The bridge is required for allowing layer2 traffic between interfaces that will sit on the different vSwitches.

/interface bridge add comment="Bridge between vmnics" name=esx-bridge protocol-mode=rstp
/interface bridge port add bridge=esx-bridge interface=eoip01
/interface bridge port add bridge=esx-bridge interface=ether2

 Testing and DHCP on vSwitch1

Connectivity between the two switches can be tested by pinging the alternative EoIP end points from either host.

e.g. router01 pinging and vice-versa

The final testing is placing DHCP on your vSwitch1 interface and ensuring that clients on both sides of the network receive DHCP leases.

Creating the DHCP pool

/ip pool add name=vswitch1_pool  ranges

Creating the DHCP server

/ip dhcp-server add address-pool=vswitch1_pool disabled=no interface=ether2 name=vswitch1_dhcp



Enable snmp in VMWare VCSA 5.5

After deploying Oberservium around my home infrastructure, clearly a dull evening. I wondered if it was possible to monitor my vCenter instance, at the end of the day it’s still a linux instance. Logging into the VCSA with ssh and looking around revealed that net-snmp wasn’t installed and obviously this is a requirement for snmp monitoring of a host.

1. Pre-requirements:

  • libsensors4-3.2.0-9.12.1.x86_64.rpm
  • perl-TermReadKey-2.30-277.1.x86_64.rpm

These two rpms need to be downloaded and installed first as a requirement for net-snmp.

2. SLES 11 SP2:

To install net-snmp the correct repository needs adding to YaST2.

  1. ssh into the vcenter instance
  2. start yast2 and select Software -> Software Repositories
  3. Add the following URL: http://download.opensuse.org/repositories/net-snmp:/factory/SLE_11_SP2/ http://ftp.twaren.net/Linux/OpenSuSE/repositories/net-snmp:/factory/SLE_11_SP2/  (thanks Joan)
  4. Go back to the main menu and select Software -> Software Management
  5. Search for net-snmp, select it with the space bar and then choose Accept.

3. Post-install configuration:


This file will be installed by default, and ideally should be moved/backed up and replaced with a simple version 2c config containing the following:

rocommunity <community name> 
syslocation <location> 
syscontact <contact details>

System startup details:

chkconfing on snmpd
/etc/init.d/snmpd start

4. Firewall / Host access

At this point the snmpd daemon will be running and this can be tested by looking at what ports are listening on VCSA host e.g:

netstat -an | grep 161

However nothing external will be able to connect to this host until the tcpwrappers configuration is modified (this took me about 40 minutes of fiddling to notice (who uses tcpwrappers anymore 😐 )). The following line (WARNING: this allows all hosts access) will need adding to /etc/hosts.allow

snmpd: ALL : ALLOW

5. External testing

From another host preferably on the same network/subnet (avoiding any other firewalls in this instance) test access with snmpwalk e.g:

snmpwalk -v 2c -c <community name> <vcsa host address/IP>

Update to the sshwrapper

Had quite a few emails recently about using the ssh wrapping class I wrote aaaages ago. I’ve traded a couple of emails back and forth.. and decided that it would be easier for everyone if I just updated these old classes.

So the changes:

  • Added DFSSHConnectionType, this class is used to define how ssh will attempt to connect (password/key/keyboard)
  • Moved everything to a namespace (DF)
  • ARC
  • Tidied up the code, and sorted an issue with CStrings making a mess when converting to an NSString
  • Other things I did ages ago.. (no idea)

It’s uploaded to github.. let me know if there is any problems..



[UPDATE]: Added the ability to place a timeout on a command sent over ssh…

Objective-C graphing and plotting with little-plot

As development has continued on a personal project it became obvious that I would need to implement UI elements that simply weren’t part of the Cocoa UI-kit. Essentially the main goal is presenting the user with a graph interface allowing them to quickly see a data set without having to read through line after line of figures. I looked at Core Plot (http://code.google.com/p/core-plot/), which whilst providing some great functionality looks like a HUGE amount of overkill when wanting a simple UI element.

So after a few days of tinkering I’ve created a couple of NSView subclasses allowing either manually created Views that can be presented arrays and will display the data accordingly.
I present Little-plot :

The above screenshot consists of three NSViews (LineView, PieView and LabelView), which each display a line graph, a pie chart and graph labels (or legends).

The project is available on GitHub here.

Updates will appear soon, along with some real documentation.

Face Detection in OSX

After reading this (http://maniacdev.com/2011/11/tutorial-easy-face-detection-with-core-image-in-ios-5/) tutorial for iOS 5 face detection I decided to try it for plain olde osX, I do find it somewhat annoying that the objective-C community is only focused on development using the iOS SDKs. The main changes involved turning the UI classes into NS classes and find ways around missing methods. I’ve uploaded my Source Code, which you can download and play with.


LVM2 device names and device paths

If you use LVM (Linux volume manager) you’ll be no doubt aware that the ability to place your volumes in groups and name the volumes accordingly makes administration much easier. Having a volume group called oracle and then a binaries and data volume means a tiny glance at a system identifies what is doing what (or so I was under the impression). When it comes to doing any sort of low level administration or system metrics, you may notice that device paths change. Using fdisk or looking at /proc metrics and you’ll suddenly come across dm-0 etc..  devices, which clearly are the lvm devices under another name. Scanning quickly through a ton of web pages and the lvm2 documentation turned up nothing.

Then I realised that the first two columns in /proc/diskstats where the major/minor device numbers, which when compared with the entries in /dev/mapper allowed me to match up the devices. This can also be achieved by looking at the major/minor numbers from lvdisplay and dmsetup ls. However the latter two tools require root privileges so clearly were not going to work with my current project.

Turned out that the simplest method is to look through the sysfs file hierarchy as the block devices can identify other names that they have. The following command will iterate all of the block devices with alternative names and print them in a format that can be piped into anything else, or in my case be read easily into an NSArray.

grep -H "." /sys/devices/virtual/block/*/dm/name \
| sed 's/\/sys\/devices\/virtual\/block\///g' \
| sed 's/\/dm\/name:/ /g'


Window handling from the dock icon (Objective-C / Xcode)

I should probably post this stuff to stackoverflow, however I find that most of the people on there are rude and spend far too much time just berating anyone who asks for the slightest help. I apologise for anyone who had to waste time looking a little bit longer for this tip.

The default behaviour for a cocoa window when it’s closed with cmd+w or pressing the red X button is for the window to be closed ((but not destroyed) this applies to the main window, others may be created to release etc..). This means that the window simply needs being passed the makeKeyAndOrderFront: method to be made visible when its dock icon is clicked on:


-(BOOL)applicationShouldHandleReopen:(NSApplication *)theApplication hasVisibleWindows:(BOOL)flag
if (flag)
 NSLog(@"Window already open");
  } else {
   [_window makeKeyAndOrderFront:self];
 return YES;