Removing a Persistent Bridge in xen after reboots

The most common method for networking using xen is to use a network bridge with your physical ethernet and then the virtual nics associated with xen domains. The default install and configuration will result in a default xen bridge (xenbr0) which will have your ethernet and virtual nics in. This information has to be explicitly declared in the various xen configuration files, however xen will take care of the actual plumbing and configuring of the bridge and the interfaces. 

However, xen has an oddity (in my opinion) it appears that xend will monitor the bridges on your dom0 and add any other bridges to it’s state. Which means should you manually create a bridge e.g. 

spike ~ # brctl addbr testbr0

xend will see that this bridge has been created and add it to it’s state configuration. On a reboot or restart of xend, what will happen is that xend will configure the networking back to the state that it recorded. This means that not only will xenbr0 be created so will testbr0, which wouldn’t be a problem if your virtual nics were added to the correct bridge (xenbr0). However they more than likely be added to testbr0 meaning that your domU’s will have no networking, unless you manually move the virtual nic to the correct bridge. 

To permanently remove bridges you need to stop xend (/etc/init.d/xend stop *BEWARE* this will stop all domUs) then go to /var/lib/xend/state/ and edit the network.xml. The entire network uuid section containing the bridge you want removing will need deleting, ensure you back everything up before hand. 

Starting xend now will result in only the correct bridges being created and the domUs nics will be added to the correct bridge.

Netbsd 4.0 domU How-to

Not a 100% straight forward to install, and there were gaps in most FAQ’s and guides I found on the internet. However collating them gave me the information i need to have a fully working Netbsd4 install in my xen enviroment.

Step 1.  (find the kernel)

If you go to their website and read the xen install guide it pretty much will tell you that you need a xen kernel (of course) which are are in the binary kernel downloads. Looking on their download pages and through mirrors would only find me a i386 kernel which is no good. Under the amd64/ folder was only their general kernels and not a dom0 or dumU kernel. A bit of googling found be a link that didn’t work, however after traversing the link i worked out that the xen kernels are under the daily builds folder (my own fault for not reading things thoroughly enough). So click here and get yourself a xen kernel — > CLICK HERE

Step 2. (create domU)

I’m assuming (never a good thing) that if you’re googling or searching for such things as a netbsd domU install then you know the basics for making a domU. Creating you domU config file is straight forward, allocate memory, a name, a disk (file or physical path) and then give it a path to the kernel. You may have noticed that there were a couple of xen kernels on the ftp site, two are required:


To begin with use the install one (pointing out the obvious here) this contains a ramdisk which will configure your hardware (such as network cards) and allow you to do a step by step install. Be mindful of the path used in the ftp/http install as the default one is incorrect. Once the installation has completed *DONT REBOOT*

Step 3. (Fix domU)

Once installation takes you back to the main install screen, open the utilities and drop to the shell /bin/sh. The xen device files need moving over to the actual file system otherwise on reboot it wont find any disks, which can be irritating as it took me some messing around until i found this solution.

mount /dev/xbd0a /mnt 
cp -pR /dev/rxbd* /mnt/dev 
cp -pR /dev/xbd* /mnt/dev 
halt -p

This will mean that when you reboot the system will see the xen devices and actually have a disk to boot from. Once this is done and the system is halted you probably will need to control-5 out and do an xm destroy on the netbsd domU (doesn’t matter as the system is halted). The final step is to change the kernel in your domU config file from the install one over to the actual domU kernel.

Step 4. (boot your working netbsd domU)

xm create -c <your_config>    

Enjoy… oh and cheers for asistance

chasing your tail

In my earlier builds (v66) of opensolaris xen domUs i’d already been through the playing around with the networking bug. So realistically you would document all of the steps that you took with the physical nic and virtual nic’s etc.. whoops

I decided that i’d do a fresh install with (v77) of opensolaris, which after discovering in the bug report has a kernel problem with mounting HSFS. This means that it can’t use an install cd properly (genius), the bug report suggests doing a NFS install which is fantastic(regarding the networking bug requires a reboot and the boot_archive updating).


bash-3.2# uname -a
SunOS host-a 5.11 snv_77 i86pc i386 i86xpv

Notes to follow

Bleeding edge sometimes makes too much mess

As my continual ‘tinkering’ I prefer to think of it as fannying around I decided that my half-baked/uncompleted solaris domU’s may as well be built from the latest version of open solaris, generally released quite regularly. Latest is 77, which as i managed to find after an hour of fiddling has a bug — YOU CANT INSTALL IT.

So luckily i’ve found version 75 which im downloading ever so slowly from a russian site, hopefully by tomorrow I can start to re-do my domU 🙂

Networking and Bridges and such

Searching the internet for solutions for the strangeness created by xen’s networking solution really comes up with snippets from email chains or highly over complicated network diagrams, why? i’m not entirely sure. The default method for networking with Xen consists of a collection of ‘pokey’ scripts that seem to get (at least on my system) 90% of the way there.  I assume this again may be a ‘Gentoo’ issue however here are the steps (From a simplistic view) that are taken to create Xen networking:

  1. Original system consisted of eth0 and lo, eth0 has an ip of etc.
  2. Once the system comes up and xen starts its scripts using brctl create a network bridge, this is then used to bridge the physical interface (currently still eth0 and virtual interfaces, called vifs)
  3. xend, the xen daemon uses brctl to create a bridge called xenbr0 then things get a bit random.
  4. eth0 is renamed peth0 (peth = phystical ethernet)
  5. The ip information is taken from peth0 and peth0 is then added to xenbr0
  6. Once the peth0 is added to xenbr0 the ip information is taken from peth0 and applied to xenbr0
  7. Any xen domU that is created afterwards creates a vif which it uses, this vif is then added to the xenbr0 allowing it to communicate on the network

This is a very sparse/dumbed down version of events, however it gives an idea of whats happening. The problem that occured with Gentoo is that step 5. never happens.

What this results in is that Gentoo comes up, brings eth0 up and we have network activity for a few seconds until xen starts to get it’s claws into the network configuration. However the most simple method for repairing this involves a small configuration change in /etc/conf.d/local.start

# 00/00/00 -- IP Allocation to Xenbr0
ifconfig xenbr0 netmask
route add default gw

This is an example taken from mine, you’ll need to alter the gateway and ip address information, but put simply this will execute after every other service has been started, resulting on your domO being visible and network aware etc…

Virtualized.. Virtualised .. ?

Recently I managed to finally build and configure the Xen Hypervisor and put together a bunch of virtual machines all together. I forked out for a new box (quad core, 4Gb Ram) two months ago and getting to point that we’re at now has been a lot of tweaking, googling, patching and the odd bit of botching.

My original plan was a straight forward Vmware esx install, however it turns out that Vmware are obviously working tightly with harware vendors so it wouldnt use any old disk controller i.e. the one i have on my motherboard.

Then after a bit of deliberation it was decided that Xen would be best method to use to host virtual machines. Gentoo was selected as i decided that the ability to hand-tailor my kernel compile etc. would allow me that extra speed advantage, a bit over the top in hindsight. The build process is all well documented on the gentoo wiki -> here

Once completed the initial Gentoo install using the emerge tool took care of most of the xen install etc.. following the HOW-TO gentoo-xen guide in the wiki pretty much covers all -> here

Once the reboot was complete the box came up and then the shortfalls in the how-to became apparent:

  • the network-bridge script never worked correctly
  • finding alternate documentation is pretty hard work
  • networking once up needs tweaking for some domU’s (TCP checksums)
  • there is barely any documentation on enabling console anywhere

More to follow …