Openfiler (x86 & x86_64) iSCSI in Xen

A nice all round NAS/SAN storage system for small businesses fits well into a virtualised infrastructure, a seperate nic is recommended but for a small business a shared core and a reasonable allocation of memory is all that is required. There has been a disk image domU release for openfiler however this official release was version 1.0 and had only a x86 kernel (meaning that x64 xen builds would complain). Here is a simple how-to to having a complete NAS and SAN (using iSCSI) solution using xen.

1. Download domU filesystem image for openfiler (ensure correct architecture is downloaded) At current time openfiler-2.3-x86_64.tar.gz
2. Create a root filesystem for openfiler to live on:
– lvcreate -L 5G -n openfiler rootdg
– dd if=/dev/zero of=/export/openfiler.img seek=5G count=1

These will create either a 5GB logical volume or a 5G disk image to use as a disk for the openfiler filesystem
3. Create a filesystem on the disk/image. This is down with mkfs [path to file system/disk image]
4. Label the filesystem, which is required for the fstab for openfiler. This is done with e2label [path to file system/disk image]
5. Mount the new filesystem and extract the archive into it. If it is the logical volume then directly mount it (e.g. /mnt) or if a disk image a loop back mount will be required (e.g mount -o loop /export/home/openfiler.img /mnt). Copy the archive into the mount, then change directory to the mount and untar the archive (e.g. tar xvf openfiler-2.3-x86_64.tar.gz)
6. Unmount the filesystem and create a second 100mb filesystem. This second filesystem will be used as swap so needs a swap fs creating on it (e.g. mkswap /path/to/swap[fs/img])
7. Create xen config file (this example will need paths altering)

[code lang=”bash”]

name = ‘openfiler-xen’
memory = ‘256’
disk = [ ‘tap:aio:/dev/rootdg/openfiler,xvda1,w’, ‘tap:aio:/dev/rootdg/openfiler_swap,xvda1,w’ ]
vif = [ ” ]
root=”/dev/xvda1 ro”
#bootloader = ‘/usr/bin/pygrub’
on_reboot = ‘restart’
on_crash = ‘restart’

8. Start the xen image with a (e.g. xm create -c /path/to/configfile) and there you go, follow the install instructions from the relevant place on openfilers website to configure and finish the setup.

Note: Obviously this example has no extra disk space attached, it would make sense to add a large third disk to the xen config file. So that openfiler then can use that to allocate space as NFS/SMB or iSCSI etc..

Sun Cluster 3.2 in Vmware (Fusion)

Some quick notes regarding getting a cluster working in vmware on the mac
Each VM will require:
3 or more Nics
1 or more root disks
512MB or more ram 

On the first virtual machine also add a load of extra scsi disks these I created in the parent directory so that I could see them. The next step is to then right click on the Solaris Cluster Node A and “Show Package Contents”. Inside the Node ‘Package’ will be a .vmx file, this is a text file that contains the configuration information for this particular Virtual machine. Cluster File systemOpen the vmx file and look for the scsi information (specifically that relating to the extra scsi disks we added) this will need copying. Then open Node B and add this scsi information to it’s vmx file (sometimes absolute paths are required) 

[code lang=”bash”]scsi0:2.present = “TRUE”
scsi0:2.fileName = “../SCSI_Disk1.vmdk”
scsi0:3.present = “TRUE”
scsi0:3.fileName = “../SCSI_Disk2.vmdk”
scsi0:4.present = “TRUE”
scsi0:4.fileName = “../SCSI_Disk3.vmdk”
scsi0:5.present = “TRUE”
scsi0:5.fileName = “../SCSI_Disk4.vmdk

This will have both nodes pointing to the same scsi disks, in essence this would be shared scsi storage. When one of these nodes is starts it will create a bunch of .lck files that will mean that the scsi devices are locked to one particular virtual machine. Either manually delete these lck files or script something to remove them.
[code lang=”bash”]

# Script to remove locks on Disk Images
# Dan – 27/3/08
# ——————–

SCSI_LOCKS=( SCSI_Disk1.vmdk.lck SCSI_Disk2.vmdk.lck SCSI_Disk3.vmdk.lck SCSI_Disk4.vmdk.lck )

for (( i = 0 ; i < ${#SCSI_LOCKS[@]} ; i++ ))
    echo “Checking for” ${SCSI_LOCKS[$i]}
    if test -s ${SCSI_LOCKS[$i]}
        echo “found ” ${SCSI_LOCKS[$i]}
        rm -R ${SCSI_LOCKS[$i]}
        echo “” $i

xen hvm error -> (12, ‘Cannot allocate memory’)

Recently the xen server that hosts all my domains has been randomly crashing, completely powering itself off. This usually has happened when i’ve been nowhere near it, so by the time i’ve gotten back there is no real obvious reason as to why this happened. This weekend I managed to bring it up straight after a crash plugged a vga cable into the box and logged into HW monitoring on the bios. 

CPU TEMP: 110c/203F

So I immediately powered off the box and went and picked some thermal paste up. Cleaned the heat sink and cleaned off the told thermal paste, and replaced it with new paste etc.. System so far has been stable, however on reboot this time i’ve been affronted with a strange new error that appears only whilst booting hvm domains.  (12, ‘Cannot allocate memory’) However the xen virtualized domains booted fine, which led me to believe that this was something to do with the ‘balloon driver’

spike / # xm info | grep memory

total_memory           : 4093

free_memory            : 4

This output is straight from reboot, it appears the xen dom0 has taken all of the memory. There are two fixes to this either a  dom0_mem = mem_size in grub or  xm mem_set

This handbagged face lady can tell me nothing..

Spending her time sniffing poo, and making fat people cry.

From the outset (flakey accreditation)

  • The value of McKeith’s certified membership of the American Association of Nutritional Consultants = Guardian journalist Ben Goldacre managed to buy the same membership online for his dead cat for $60.
  • ‘Dr’ Gillian McKeith = a non-accredited correspondence-course PhD from the US

“If you contact the Australasian College of Health Sciences (Portland, US) where McKeith has a “pending diploma in herbal medicine”, they say they can’t tell you anything about their students. When you contact Clayton College of Natural Health to ask where you can read her PhD, they say you can’t. What kind of organisations are these? If I said I had a PhD from Cambridge, US or UK (I have neither), it would only take you a day to find it.”

Examples of her technical reasoning

  •  “She talks endlessly about chlorophyll, for example: how it’s “high in oxygen” and will “oxygenate your blood” – but chlorophyll will only make oxygen in the presence of light. It’s dark in your intestines, and even if you stuck a searchlight up your bum to prove a point, you probably wouldn’t absorb much oxygen in there”
  •  “She says DNA is an anti-ageing constituent: if you “do not have enough RNA/DNA”, in fact, you “may ultimately age prematurely”. Stress can deplete your DNA, but algae will increase it: and she reckons it’s only present in growing cells. Is my semen growing? Is a virus growing? Is chicken liver pate growing? All of these contain plenty of DNA”
  • McKeith argues that examining and smelling faeces can give clues to bodily misfunction. – According to Catherine Collins, chief dietician at St George’s Hospital, London, “it is impossible to diagnose medical conditions from looking at a normal brown stool.”

NetBSD quick fix

This is a quick list for bits and bobs on getting a ‘working’ netbsd box after the installation has completed. This will generally focus on getting the network up and then related services.

1. set up network editing the files /etc/resolv.conf, /etc/rc.conf and /etc/ifconfig.xxN. (xxN is your adapter name, e.g. xennet0 for a xen network interface)

[code lang=”bash”]
echo “nameserver x.x.x.x” >> /etc/resolv.conf
echo “inet netmask″ >> /etc/ifconfig.xxN

Open /etc/rc.conf with vi and add/change the following lines.

[code lang=”bash”]

This will now have you set up with a permenantly configured network interface. Next step is enabling remote logins either ssh or telnet, obviously ssh is the prefered method as it’s secure. Enabling ssh involves adding an overide to /etc/rc.conf, the /etc/defaults/rc.conf has the directive sshd=NO.

[code lang=”bash”]
echo “sshd=YES” >> /etc/rc.conf
/etc/init.d/sshd start

Enabling telnet is done through /etc/inetd.conf and consists of removing the hash from the beginning of the telnet directive. Also remove the ‘-a valid’ from the end to allow all types of logins.

A reboot will then allow remote users to log into the system, also remember to add a user to the ‘wheel’ group allowing them to su to root.


There have been a number of new toys appear at chateau de Dan recently, my new macbook, a new router and something i’ve been after for ages (an rs232 to ethernet).

 The macbook replaces my ageing powerbook 🙁 which i’ve had for a couple of years now and is starting to really show it’s age. The Macbook is complemented by 4Gb of ram which I ordered from crucial and arrived in no time what so ever. I could have ordered a macbook with the memory already in it, however it’s 500 quid for the memory from apple and 60 quid from crucial so only a nutter would buy the ram from apple. Also I had planned to use the discount that I get from BT. However with Kim with me it emerged that students again get discount from apple, a ridiculous amount of discount infact. The BT discount amounted to 6%, which isn’t to be sniffed at. However students get 14% discount on hardware and software and 70% discount on apple care. So I snapped there hands off when we were told this, and now i’ve a fully apple cared for laptop :).

The replacement router is a belkin ‘N’ which is utter dog wank, no snmp, no static routes, no telnet access, no snmp, crashes all the time and doesn’t work with my xBox. So I expect i’ll replace it soon enough.

 The rs232 to ethernet adapter on the other hand is fantastic, this is something that i’ve wanted for a very long time. The reason it’s taken so long to get one has been the fact that they cost a small fortune to buy new. I have trawled through ebay numerous times to find a cheaper one, but had so fair failed to find one. Luckily I found one on a nightshift a couple of days ago and bid a couple of quid on it, got outbid pretty sharpish and proceeded to forget about it. Then a couple of days later someone mentioned that they had been outbid on something else on eBay at which point I decided to see how much the adapter was. When I checked it hadn’t moved from three days ago and there was 4 minutes remaining on the auction. So I snapped that up and it appeared a day ago.

rs232 to ethernet

works like a charm 🙂

Netbsd 4.0 domU How-to

Not a 100% straight forward to install, and there were gaps in most FAQ’s and guides I found on the internet. However collating them gave me the information i need to have a fully working Netbsd4 install in my xen enviroment.

Step 1.  (find the kernel)

If you go to their website and read the xen install guide it pretty much will tell you that you need a xen kernel (of course) which are are in the binary kernel downloads. Looking on their download pages and through mirrors would only find me a i386 kernel which is no good. Under the amd64/ folder was only their general kernels and not a dom0 or dumU kernel. A bit of googling found be a link that didn’t work, however after traversing the link i worked out that the xen kernels are under the daily builds folder (my own fault for not reading things thoroughly enough). So click here and get yourself a xen kernel — > CLICK HERE

Step 2. (create domU)

I’m assuming (never a good thing) that if you’re googling or searching for such things as a netbsd domU install then you know the basics for making a domU. Creating you domU config file is straight forward, allocate memory, a name, a disk (file or physical path) and then give it a path to the kernel. You may have noticed that there were a couple of xen kernels on the ftp site, two are required:


To begin with use the install one (pointing out the obvious here) this contains a ramdisk which will configure your hardware (such as network cards) and allow you to do a step by step install. Be mindful of the path used in the ftp/http install as the default one is incorrect. Once the installation has completed *DONT REBOOT*

Step 3. (Fix domU)

Once installation takes you back to the main install screen, open the utilities and drop to the shell /bin/sh. The xen device files need moving over to the actual file system otherwise on reboot it wont find any disks, which can be irritating as it took me some messing around until i found this solution.

mount /dev/xbd0a /mnt 
cp -pR /dev/rxbd* /mnt/dev 
cp -pR /dev/xbd* /mnt/dev 
halt -p

This will mean that when you reboot the system will see the xen devices and actually have a disk to boot from. Once this is done and the system is halted you probably will need to control-5 out and do an xm destroy on the netbsd domU (doesn’t matter as the system is halted). The final step is to change the kernel in your domU config file from the install one over to the actual domU kernel.

Step 4. (boot your working netbsd domU)

xm create -c <your_config>    

Enjoy… oh and cheers for asistance

Netbsd 64bit domU

I’ve been wanting a bsd domU since i managed to originally get my xen server up and running, again googling the great wide web revealed past threads on forums and dead links. The effort to port the freeBSD kernel has been all but abandoned, and openBSD doesn’t appear to be moving toward that at all. So I looked into netBSD with their motto “Of course it runs NetBSD.”  surely I couldn’t go wrong. However when I was originally looking in Nov ’07 there was only a 32bit domU kernel and my xen domain is all 64bit so unless i went down the hvm route (which i didn’t) then that was that.

Yesterday I noticed that netbsd has been updated late december, and after a bit of searching found a note in the changelog saying that there was a 64bit kernel now for xen. However it’s hidden in the various depths of netbsd’s ftp site, which took a considerable bit of finding. However here is proof –>