Centos 5.1 Xen build (domU)

Here is a quick how to for installing centOS 5.1 in a domU, this is a netinstall so all thats required is a kernel and a ramdisk. The latest release is 5.1 and i’m in the uk so I’m using the UK mirror.

The Kernel and ramdisk are found here, download these using wget or any of your favourite methods and place somewhere in your dom0. In my templates folder/xen you will find a simple template for a centos config, change the paths to match your kernel and ramdisk… you’ll also need to change the disk to whatever you’re using, disk image or another LV.

then xm create -c <centos_config> and you’re away.

 The templates folder can be found here.


I’ve used GNU screen for absolutely ages,  a recent chat made me have a look around for an alternative, the alternatives are at best weak. The best i found was a program called twin, which cleverly enough stands for Text WINdow manager….  🙁

 It works *just*  there is a screenshot, but I think im going to stick to screen for now

chasing your tail

In my earlier builds (v66) of opensolaris xen domUs i’d already been through the playing around with the networking bug. So realistically you would document all of the steps that you took with the physical nic and virtual nic’s etc.. whoops

I decided that i’d do a fresh install with (v77) of opensolaris, which after discovering in the bug report has a kernel problem with mounting HSFS. This means that it can’t use an install cd properly (genius), the bug report suggests doing a NFS install which is fantastic(regarding the networking bug requires a reboot and the boot_archive updating).


bash-3.2# uname -a
SunOS host-a 5.11 snv_77 i86pc i386 i86xpv

Notes to follow

Bleeding edge sometimes makes too much mess

As my continual ‘tinkering’ I prefer to think of it as fannying around I decided that my half-baked/uncompleted solaris domU’s may as well be built from the latest version of open solaris, generally released quite regularly. Latest is 77, which as i managed to find after an hour of fiddling has a bug — YOU CANT INSTALL IT.

So luckily i’ve found version 75 which im downloading ever so slowly from a russian site, hopefully by tomorrow I can start to re-do my domU 🙂

Solaris install hang

Setting up Java, please wait... *HANG*

Not really investigated this issue fully, however thanks to baggy there is an easy solution. In the boot options select:

6. Single user mode

This will drop you straight to a shell where we can start the install process manually. The normal (console) install follows these steps:

1. mount cd-rom

2. dhcp network interfaces

3. do some java initialization

4. start the solaris install procedure

It is step 3 where the installation process hangs for some reason, so we simply manually do these steps. From the shell:

mount -F hsfs  /dev/dsk/c0d6p0 /cdrom

Should you wish to play with the java initialisation then:




For pretty much everything that needs doing i’d recommend using this link: here  

One of the primary reasons for the creation of the xen box would be the inclusion of solaris domU’s, these generally would be for testing and general usage etc..

Google Open solaris and again you’re faced with a plethora of scraps and tid-bits of information with regards to creating and more importantly troubleshooting a solaris domU. This was originally the most upto date and comprehensive guide with troubleshooting that I found –> here. However this would start to load and then present me with a kernel panic in the domU, this was a quick fix though as it was down to the kernel and ramdisk being 32bit.

‘A quick change in the extraction of the kernel and ramdisk, just make sure that you pop into the 64bit directory where the 32bit kernel is and grab that one if you’re on a 64bit box. ‘

On grabbing those you’ll find that starting your domU presents you with this:

Started domain solaris-a
SunOS Release 5.11 Version xen-nv66-2007-06-24 64-bit
Copyright 1983-2007 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
WARNING: Found xen v3.1.0 but need xen v3.0.4-1-sun
WARNING: The kernel may not function correctly

Ignore the version problem, doesn’t appear to cause any sort of issue. Once the domU is using the correct kernel etc. it will work as any normal native install does.

Networking and Bridges and such

Searching the internet for solutions for the strangeness created by xen’s networking solution really comes up with snippets from email chains or highly over complicated network diagrams, why? i’m not entirely sure. The default method for networking with Xen consists of a collection of ‘pokey’ scripts that seem to get (at least on my system) 90% of the way there.  I assume this again may be a ‘Gentoo’ issue however here are the steps (From a simplistic view) that are taken to create Xen networking:

  1. Original system consisted of eth0 and lo, eth0 has an ip of etc.
  2. Once the system comes up and xen starts its scripts using brctl create a network bridge, this is then used to bridge the physical interface (currently still eth0 and virtual interfaces, called vifs)
  3. xend, the xen daemon uses brctl to create a bridge called xenbr0 then things get a bit random.
  4. eth0 is renamed peth0 (peth = phystical ethernet)
  5. The ip information is taken from peth0 and peth0 is then added to xenbr0
  6. Once the peth0 is added to xenbr0 the ip information is taken from peth0 and applied to xenbr0
  7. Any xen domU that is created afterwards creates a vif which it uses, this vif is then added to the xenbr0 allowing it to communicate on the network

This is a very sparse/dumbed down version of events, however it gives an idea of whats happening. The problem that occured with Gentoo is that step 5. never happens.

What this results in is that Gentoo comes up, brings eth0 up and we have network activity for a few seconds until xen starts to get it’s claws into the network configuration. However the most simple method for repairing this involves a small configuration change in /etc/conf.d/local.start

# 00/00/00 -- IP Allocation to Xenbr0
ifconfig xenbr0 netmask
route add default gw

This is an example taken from mine, you’ll need to alter the gateway and ip address information, but put simply this will execute after every other service has been started, resulting on your domO being visible and network aware etc…

Virtualized.. Virtualised .. ?

Recently I managed to finally build and configure the Xen Hypervisor and put together a bunch of virtual machines all together. I forked out for a new box (quad core, 4Gb Ram) two months ago and getting to point that we’re at now has been a lot of tweaking, googling, patching and the odd bit of botching.

My original plan was a straight forward Vmware esx install, however it turns out that Vmware are obviously working tightly with harware vendors so it wouldnt use any old disk controller i.e. the one i have on my motherboard.

Then after a bit of deliberation it was decided that Xen would be best method to use to host virtual machines. Gentoo was selected as i decided that the ability to hand-tailor my kernel compile etc. would allow me that extra speed advantage, a bit over the top in hindsight. The build process is all well documented on the gentoo wiki -> here

Once completed the initial Gentoo install using the emerge tool took care of most of the xen install etc.. following the HOW-TO gentoo-xen guide in the wiki pretty much covers all -> here

Once the reboot was complete the box came up and then the shortfalls in the how-to became apparent:

  • the network-bridge script never worked correctly
  • finding alternate documentation is pretty hard work
  • networking once up needs tweaking for some domU’s (TCP checksums)
  • there is barely any documentation on enabling console anywhere

More to follow …