Header image alt text

Kevin's Thoughts!

Maybe you agree, maybe you don't… find out!

As those that follow know, I briefly lived in the W7 Professional world in order to test out Microsoft Virtual PC.  I have found that, at least for my needs, VirtualBox is a much better solution.  I also found out that VirtualBox will die the moment Virtual PC is opened, so Virtual PC had to go.  With it gone, there was no need for Professional anymore, so I started the quest to downgrade.

Many a forum post will tell you its not possible.  Hogwash.  Yeah, its not exactly supported, but presuming you have a legal copy of Windows 7 Home Edition, and a valid product key, its doable.  You just need to insert the DVD and run an upgrade.  OK, OK… its not quite that simple, but close.  If you try and run the upgrade, it will complain that downgrading isn’t doable.  If you do the registry hacks to make your Professional Install look like Home Edition, it will get farther, but die from a message similar to “Installed version is more current”.

The trick (yes… there is always a trick isn’t there?) is to use the Microsoft Compatibility settings to trick the setup DVD into thinking your running Vista.  Its really that easy.  Just install the DVD, but don’t start it.  Open it.  Highlight “setup”, then tell Windows to run it in Vista SP 2 Compatibility mode.  Works like a charm.

Downside?  It takes FOREVER!  I’m talking like 12 hours on my machine.  But it worked.  I’m beginning to understand why my Windows Engineer colleagues hated to do upgrades and always preferred a fresh install.  During my “upgrade” over 800,000 files and settings were apparently ‘saved’ and later ‘restored’.

Of course, a fresh install is always an option (and is much, MUCH, quicker), but then you lose all your files and personal settings.  Personally, I store most of my files elsewhere, but “personal settings” include things like game data, Thunderbird account settings, etc.  Those e-mail settings will a killer since I use Thunderbird to consolidate a good half-dozen e-mail accounts.

MacOS – the adventure continues

Posted by Kevin on September 10, 2011
Posted in Tech Talk  | Tagged With: , , , , , | No Comments yet, please leave one

Its been an interesting couple of days.  As previously reported, I received my Mac OS X Lion thumb drive.  Unfortunately, it appears that one has to start with a Snow Leopard DVD install.  Did that.

First thing I wanted to try was getting Snow Leopard to run in a Microsoft Virtual PC.  That was fun, but unsuccessful.  I could get Iboot to load, and the Snow Leopard DVD would start to boot, but it rapidly crashed.

Going back to the Hackintosh pages (I rather like: http://lifehacker.com/321913/build-a-hackintosh-mac-for-under-800 ) it seemed like I could set myself up to dual boot.  I had plenty of disk space, so shrunk my Windows C drive by 128GB and started down the path.  This page http://tonymacx86.blogspot.com/2010/04/iboot-multibeast-install-mac-os-x-on.html seems to be the definitive reference.

Apparently the Mac install disk is pretty picky, and Apple of course does not support using just any old PC, but still, it seemed doable.  I yanked out half my memory (4GB max for Iboot), and started the process.  Iboot came up fine, but died when I selected it from the GUI.  Good reason:  your suppose to boot from the Iboot CD, then remove it, install the Mac installation disk, and hit F5.  Did that (after finding that little hint on page 3 of some post) and the Mac installation started!  Alas, it would only run for a few minutes before the screen went blank and the machine hung up.  Oddly, I would see a bit more DVD activity a few minutes after it went blank, but it never recovered.

Spent a morning learning about boot flags like “-v -x”, but never managed to get much past Mac OS loading its ram drives and network.  I did, however, find a reference to running Mac OS under VirtualBox.  There are several, but I’m currently comparing and contrasting http://lifehacker.com/5583650/run-mac-os-x-in-virtualbox-on-windows with others I have found.

VirtualBox is free.  Originally from Sun Microsystems and now offered free from Oracle.  It appears MUCH more advanced than Microsoft Virtual PC and in fact recognizes Mac OS and pre-configures a VM to be compatible.  Still playing with this, but I have high hopes.

Update:  Making good progress.  Going the IBoot route in a VirtualBox environment seems to be pretty clean.  Note you will need to register on TonyMacs site before that IBoot link works.  Its just a username/password thing, no commitment.  Using Tony’s Multibeast toolkit eliminates the need to boot from Iboot after the install, which is a good thing, since Iboot doesn’t support Lion yet (or so I’m told).  Anyhow, following the instruction at http://www.sysprobs.com/iboot-loader-virtualbox-install-snow-leopard (which also has links back to Tony’s site).  Make sure you continue reading after you get Snow Leopard to boot, there are more steps!

Three side notes: 1) If you appear to have lost your mouse control when running VirtualBox, just tap the RIGHT Ctrl key – it should appear.  2) If you happen to be playing with an old Windows XP license under VirtualBox, make sure you install the Guest Additions under the VirtualBox Devices tab for Windows XP.  Do it as soon as Windows XP finishes loading.  It will save you a LOT of mouse problems.  Mind you, this is a great use for an otherwise obsolete license. 3) If after Iboot sees your install disk, the install hangs, try typing “-x” before you select the install disk.  That will cause the boot to proceed in “safe mode”, which seems to often help.

While preparing my primary desktop for adventures wth MacOS I had to change my SATA setting from IDE to AHCI.  Should have done that when I installed Windows in the first place, since AHCI performs better – but didn’t know that then.

Anyhow, just changing that in BIOS will cause your Windows environment to crash from lack of it starting the AHCI driver.  There is some vodoo magic one can do with regedit to fix this after the fact, but my attempt at doing so broke my install.

So… last night I reinstalled Windows 7 from scratch and then used my friendly Windows Home Server to restore my missing files.  That went ok, its a simple drag and drop, but I was missing all my application data (things like all my Thunderbird accounts).  Played some more and found the hidden files needed to restore those and others (thank goodness), but when I was done the system was still “funky”.  I found old versions of windows installed which could not be removed (“no longer at same location”), and what was really weird:  I couldn’t open “My Computer” or “Recycle Bin” or any of the pictures on my desktop.  I could open Excel files, Acrobat files, and many other things.

While researching how to get around that, I came across an article on using Windows Home Server to restore and entire computer.  I had a great recent backup, so started down the adventure of trying this.  Finding and burning the current CD from Microsoft was straight forward.  When I booted it, it found my NIC and hard drives, but could not find my Home Server.  More googling suggested all types of weird things like changing the case of the server name, none of which worked.  One post indicated he installed a different NIC in his machine and all was well.  Others talked about loading the most recent disk and nic drivers on a USB thumb drive, and that the restore CD would scan and load the drivers.  The last sounded reasonable, so I loaded both the motherboard and NIC drivers straight from the motherboard manufactures web site onto my USB thumb drive.  Rebooted the restore CD, asked it to scan for drivers, and it did so, found some, and I thought all was well.  Alas, it still could not find the Home Server.

Having a bunch of computers at the house I keep a couple of spare NICs just in case.  It was time to unwrap one.  Did so, installed in the desktop, and tried the restore CD again.  No joy, it didn’t detect the new NIC.  Added the drivers from the CD to the USB thumb drive and tried again.  Booted the restore CD, told it to scan, and it found the new NIC.  Better yet, it quickly found the Home Server!

I’m reformatting the hard drive just to make sure everythign is clean as I key in this post, but I have high hopes.

Update:  The restore worked perfecting, taking just a bit less time than the format itself.  This placed the system back in “pre-AHCI” mode.  I took another look in the web and found that there were (2) registry entries I needed to change.  Did that and everything booted like a charm.

I now feel VERY good about having invested the time to place a Windows Home Server on my network.  Highly recommended if you have several Windows boxes or just want to have a nice way to recover one and provide yourself daily file restore capability on the side.

New adventure in technology: MacOS/IPad development

Posted by Kevin on August 31, 2011
Posted in Tech Talk  | Tagged With: , | No Comments yet, please leave one

Yesterday I started down the path of discovering what it takes to develop application for the IPad (and I suppose the IPhone and MacOS in general).  Step one was to register as an Apple Developer (http://developer.apple.com/programs/register/).  That was quick, painless, and free.

Today I went and looked for the Apple Software Development Kit (SDK).  Its free, and is designed to run under Mac OS (of course!).  So… how to proceed?  Would I need to buy a Mac to explore this?  Not an option right now.  A bit of we browsing has revealed a lot on how to create a “hackintosh” – a normal PC running Mac OS natively.  This was made technically possible when MacOS converted to running on the Intel platform a number of years ago.  One needs either a new PC machine (cheaper than a Mac equivalent) or a partition on an existing Windows box.  Fortunately, my Windows 7 machine has several hundred GBs free on its disk, so I just used the Windows disk management (control panel -> Administrative Tools -> Computer Management -> Storage -> Disk Management) to shrink the partition freeing up space for a Mac OS partition.  The hackintosh page indicates I need a Mac OS thumb-drive to boot from.  This can be created by downloading the Mac OS DVD image ($29 from the Mac Store) and placing on your own USB thumb drive, or buy buying the OS pre-installed on a thumb drive from the Mac store ($69).  Given that a 16GB thumb drive cost at least half the difference, and requires some effort to format and move the DVD image over, I opted to order the pre-installed device.  That should be here next week.

I was wondering if I could run it in a VM under Windows and found this link:


Apparently one can.  Seems to me like the legalities are questionable, but I’ll probably explore this while waiting for my thumb drive.  Note the above link is based on VMWare Workstation, which is free for 30 days, but cost $189 to purchase.

More as I progress!

9/1/2011:  Quick update.  I ordered the MAC OS thumb drive from the Apple store at 11:23am yesterday.  Standard delivery.   Order was shipped UPS at 4:46pm.  Delivered today at noon.  Pretty darn impressive!

KVM – libvirt

Posted by Kevin on August 31, 2011
Posted in Tech Talk  | Tagged With: , , | No Comments yet, please leave one

I think I’m done playing with Libvirt for the moment.  As a general purpose library management tool it looks good.  From an ease-of-use perspective I suspect its not bad if you are running in an X-windows environment.  From a “lets convert” standpoint?  Well, that is proving to be another issue.

For instance, I currently start up my KVMs with a command similar to:

kvm -net nic,model=virtio -net tap,ifname=tap6,script=no  -drive file=test2.img,if=virtio,boot=on -curses -no-reboot -m 2g -smp 2

This starts a fully paravirtualized instance using virtio for both networking and the disk image.  The resulting VM has 2gb of memory and 2 virtual CPUs allocated to it.

Libvirt uses XML files to control everything, an idea I fully support.  However getting started creating those can be a challenge.  I found two approaches.  The first was with a related tool called “virt-install”.  It is not part of the libvirt, but built using it and closely tied to it.  So after playing for a bit, I came up with:

virt-install –name “test2” –ram 2048 –vcpus=2  –cpu host –description “Test VM” –import –file test2.img –os-type=linux –os-variant=virtio26  –network bridge=br0,model=virtio –graphics=none   –noreboot

Using the virsh list command shows the VM has been created and is running!  Probing around a bit shows the file /etc/libvirt/qemu/test2.xml has been created.  Using virsh edit test2 puts me into a VI editing session on the XML.  All looks good.  Now to connect.  Connect?  Connect to what?  I try opening a ssh session and it times out.  Unlike when I issue the KVM command directly, there is no associated curses window to look it.  Looking at /var/log/libvirt/qemu shows a log of the command being issued… very useful!  Apparently a serial console connection has been opened on /dev/pts/2.  Ok, crank up screen /dev/pts/2 and it connects, but there is no output…  Looking at the log file shows a few concerning things:  1) KVM is being started up with a “-S” switch, which the developers tell me is normal, but the man-pages say disables the startup of the CPUs.  Its sure behaving like the man-pages are right.  2) the image file is being started with “if=none” instead of “if=virtio”.  A bit of command line testing verifies that “if=none” prevents the kernel from finding the root disk.  Not good.  Checked in with the user-list community and the generated XML looks appropriate, so this issue remains pending.  Finally, the network is not being started using tap6, my assigned network device for this VM.

The user community suggested I try an alternative method to generate the initial XML.  Apparently virsh amoungst its MANY options, has one for generating XML from a command line!  Cool.  Wish I had noticed that earlier.  The command:

virsh domxml-from-native qemu-argv /kvms/test2 >test2.xml

Takes the file test (which contains the KVM command at the top of this post) and converts it to equivalent xml for libvirt.  Very Cool!  Well, at least in concept.  It has a few very minor parsing problems, like the “2g” for memory must be specified as “2048” (in megabytes).  Also the kvm command must be fully qualified as /usr/bin/kvm.  Other than that, its works fine.  The generated xml looks very similar to that created by virt-install with a few minor differences like the instance not being named (ok, ok, its named “unnamed”) and the OS architecture and CPU information is generic X86 instead of the very specific AMD statements generated by virt-install.  There is also no serial port defined like virt-install did, but there is a “ethernet” device now that reflects my tap6 port.  A bit of cut & pasting and I managed to create a custom.xml with the best of both worlds.  Issuing virsh define /kvms/custom.xml imports the new xml file and again defines and tries to starts a running instance.

Notice the “tries” word above?  The network won’t start, so the instance is stopped.  The network won’t start because the “tap” device apparently needs to be a “tun” device.  Again looking at the log file shows the same if=none for the disk image as well.  Even if I reconfigured my network to use “tun” devices instead of “tap” devices, the VM would not get past booting the kernel.

I’ve spent a significant amount of time browsing the various man pages to try and figure out the above issues.  I did discover that libvirt appears to be more “cutting edge” than even the man pages for kvm.  e.g.  It uses some “newer” format parameters that I couldn’t find in the man pages (both local and on the official qemu site), which was a bit frustrating, but the format was pretty obvious. It also appears that the libvirt world really is built around VMs being behind a NAT subnet which libvirt wants to control.  It may be possible to get around that, but so far I haven’t had any luck.

So, for now, I’ll just manage my VMs from the command line, and start them via /etc/local.d/local.start.



Best Practices for KVM

Posted by Kevin on August 29, 2011
Posted in Tech Talk  | Tagged With: , , , | No Comments yet, please leave one

IBM is, and has been, a great supporter for KVM.  I was quite pleased today to find the following link:


Nice, easy to read, publication on how to optimize the performance of KVM, complete with a few test results to justify their recommendations.

Thank you IBM!

An update:

Being a bit of a computer efficiency enthusiast (read “speed freak”) I decided to migrate one of my VMs from a .img file base directly to an lvm – bypassing the Linux filesystem overhead as IBM suggests.  The process was relatively straight forward, given that I had no issues with the VM being down when I migrated it.  I decided to do this for my new Mail VM, since I was going to have to grow the image size anyhow to handle long term storage of e-mail.

1) Create a new logical disk volume:  lvcreate -L512G -nmail_vm vg

2) Boot my “recovery” minimal boot image passing it the old image file and the new logical volume

kvm  -drive file=mail.img,if=virtio -drive file=/dev/vg/mail_vm,if=virtio,format=raw -cdrom minimal.iso -boot d -curses -no-reboot -m 512 -smp 2

The VM then boots with 2 drives, /dev/vda (old mail.img) and /dev/vdb (new disk).  Within the minimal shell, you then:

A)  Format the new drive with fdisk /dev/vdb (I create 4 partitions, a 64mb /boot in partition   1, a 1GB swap area in partion 2, at 512mb /root in 3, and an LVM area with the rest of the disk in partition 4)

B)  Create a new physical volume group on it via pvcreate /dev/vdb4, a logical volume group on that with vgcreate vg00 /dev/vdb4  and divvy it up as desired with lvcreate commands.  Do note that the volume group name you use needs to be different than what is used on the .img drive.  Confirm everything looks good with vgs and lvs.

C)  Mount the old and new partitions onto new root trees like /or and /nr and use tar to move the files cd or; tar cf – . | (cd /nr; tar xvf -).

D)  Install grub on the new drive to make it bootable.  This was actually just a bit tricky, the key being to create symbolic links for /dev/hda, /dev/hda1, etc. to their /dev/vda counterparts after which the grub root (hd0,0) and setup (hd0) command worked as normal.

E)  Shutdown and start using your new /dev/vg/mail_vm image:

kvm -net nic,model=virtio -net tap,ifname=tap2,script=no  -drive file=/dev/vg/mail_vm,if=virtio,boot=on -curses -no-reboot -m 2g -smp 2 -runas kvm -name mail,process=mail-kvm


KVM Paravirtualization

Posted by Kevin on August 28, 2011
Posted in Tech Talk  | Tagged With: , , , | No Comments yet, please leave one

In previous post I talked building my KVM machine. This was semi straight forward, turn on visualization support in the kernel , build a image (see my KVM Virtualization post) and boot.  Pretty easy to get a user-based networked server up and running.  In its simplest form, the command:

kvm -hda base-vm.img -curses -no-reboot

Starts a virtual machine whose image is stored in the .img file, with a text based console, and will return to the command prompt when the virtual machine is shutdown or rebooted.  The virtual machine will see the image file as its /dev/sda disk.  An eth0 device will be presented (presuming the image is built with a supported driver, like rtl8139) along with dhcp and name services being provided by the kvm hypervisor (which is what the kvm command really is).  If you are logged into this machine, there would be little to give you any hint that you were running in a VM rather than on dedicated hardware.

That’s all good, but its somewhat inefficient.  Linux and KVM support a number of paravirtualization features which reduce overhead within the guest support – if the guest enables such support.  My guest have been configured to use as much of this support as possible, giving up most chances of running native, but benefiting from a reduced kernel size and more efficient execution.

There are several kernel configuration switches that need to be set to fully utilize this support:

Under “Paravirtualized guest support” select:
KVM paravirtualized clock (CONFIG_KVM_CLOCK=Y)
Enable Paravirtualization code (CONFIG_PARAVIRT=Y)
Under “Device Drivers”, “Block Devices”, select Virtio block driver (CONFIG_VIRTIO_BLK=Y)
Under “Device Drivers”, “Network Device Support”, select Virtio network driver (CONFIG_VIRTIO_NET=Y)

So far, nothing you have done will stop your kernel from booting on real hardware, although it will have some extra overhead with the Enable Paravirtualization code feature on.  However, the next step commits you to running as a guest only:

You can remove all SCSI disk, Serial ATA, and Parallel ATA support as well as all network device support (I leave Network Console logging support enabled by habit, although I’ve yet to use it).  Presuming you use LVM, you will need to leave Device Mapper Support enabled.

Since we have eliminated all ATA and SCSI devices, obviously /dev/sda will no longer exist.  It has been replaced by /dev/vda (virtual disk #1).  You will need to change your /etc/fstab file to reference /dev/vda instead of /dev/sda, and change grub’s menu.lst and device.map file via your favorite editor.  If you forget this step, just boot the minimal.iso – it supports virtio devices – and fix these entries.

Now change your startup command to look like:

kvm -drive file=base-vm.img,if=virtio,boot=on -net nic,model=virtio -curses -no-reboot

Notice when you boot that there are no references to “eth0”, but the device will be there, now 100% virtual without any emulation overhead.  Likewise for storage, everything will be presented without the overhead of emulating SCSI or ATA.  Your kernel will also take advantage of many paravirtualization improvements now that you gave it the opportunity to.

Happy efficient computing!


KVM Bridging

Posted by Kevin on August 28, 2011
Posted in Tech Talk  | Tagged With: , , , , , | No Comments yet, please leave one

Spent more than a little time banging my head against the wall and freaking out that Google couldn’t help.  What was the problem?  I couldn’t get virtual network bridging to work between my KVM based guest and the rest of the world.

If I started my a KVM with -net user (or let it default), all was fine.  By default, KVM will create a private network and provide dhcp and name services to the virtual instance.  You can either use a dhcpd client or just hardcode ip addresses in the defined 10.0.2.x space.  However, this “user” mode is designed as its named – it allows the VM to get out to the Internet, but doesn’t allow anything in.  So it works great for building your image, and is even fine for running things like BOINC, but is rather limited.  For instance, you can not SSH into the machine, so it better either run things automagically, or you will need to start it in a window.

For reasons unknown, if I started a KVM using a tun/tap (tap in this case) device it behaved oddly.  Once in awhile, I’d see an eth0 device appear during boot (seen via reviewing the boot log using “dmesg”).  As I tried with various “model=” options, I would see the corresponding driver load and sometimes a reference to eth0, but seldom anything that “ifconfig” could see.

Per most of the documentation I found online, the following startup command should have worked:

kvm -net nic,macaddr=00:00:00:00:01:00,model=virtio -net tap,ifname=tap0,script=no -drive file=test.img,if=virtio,boot=on -curses -no-reboot

And in fact, on those odd occasions when some variation of the above actually generated an eth0 device, it would appear with the macaddr specified.  Still, in those cases, I could not ping or web (via links) outside to the world.  (fyi:  ping using ICMP, webbing using TCP, so it was good to check both given KVMs blocking of ICMP when using -net user).   It was also freaky inconsistent.  At one point I had two KVMs with eth0’s showing up on one but not the other.  I did a side-by-side kernel build comparison and the kernels were identical.  I did several reboots of each KVM, several kernal builds including a “make clean” pass, and it would work on one but not the other.  Even when it did work, at the host level I often saw errors about the bridge receiving packets with its own address.  I thought those were referring to TCP/IP addresses (and went down a path trying to figure that out), but in hindsight it was referring to MAC addresses!

Finally I decided to ignore the web help, and just start at the beginning using the man pages.  Strange how that sometimes helps <smile>.

The breakthrough occured when I used the ever so slightly simplier command:

kvm -net nic,model=virtio -net tap,ifname=tap0,script=no -drive file=test.img,if=virtio,boot=on -curses -no-reboot

Note, the only difference is the lack of specifying the mac address.

Here are the relevant bits from /etc/conf.d/net:

tunctl_tap0=”-u root”
bridge_br0=”eth1 tap0 tap1 tap2 tap3 tap4 tap5 tap6 tap7″
routes_br0=”default gw″

A couple of notes:  Using a mac address of all zeros generates a boot time error about not being able to reassign the hardware address the mac address represents.  Any non-all-zero address works fine.  Also, the bridge will take on the mac address of the first tap device defined.  Also, this is a test environment, so I’ve given myself a lot of tap devices to play with.  So, at the host level, we see:

ifconfig br0
br0       Link encap:Ethernet  HWaddr 00:00:00:00:01:00
          inet addr:  Bcast:  Mask:
          RX packets:75628 errors:0 dropped:4930 overruns:0 frame:0
          TX packets:22841 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:54238310 (51.7 MiB)  TX bytes:8638206 (8.2 MiB)

ifconfig tap0
tap0      Link encap:Ethernet  HWaddr 00:00:00:00:01:00
          RX packets:28733 errors:0 dropped:0 overruns:0 frame:0
          TX packets:53601 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:35590127 (33.9 MiB)  TX bytes:5638527 (5.3 MiB)

And within the KVM we see:

ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56
          inet addr:  Bcast:  Mask:
          RX packets:44833 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28551 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3881750 (3.7 MiB)  TX bytes:35412015 (33.7 MiB)

Much to my delight – this worked!  KVM apparently uses the tap device, but assigns it a unique MAC address (the HWaddr shown in the ifconfig output).  Now all generated IP packets have different MAC source addresses from the bridge and everything works as expected!  My guest KVM can ping, web, and do anything else it wishes with the outside world (at least until I start configuring iptables, but that is another post).  One glitch remained, the outside world didn’t know how to get the virtual machine.  A quick route add -host gw allowed me to ssh into the vm.  Still couldn’t get to it from a windows machine though.  Emerging and starting mrouted seem to have fixed that.  Not 100% sure that was needed, but its working, and at this point I’m going with “if its not broke, don’t fix it” as a philosophy.

With VMWare recently announcing a new pricing structure based on total virtual memory size instead of server size, I decided that perhaps another look at KVM was appropriate.  KVM is open-source and built into the Linux kernel (presuming you enable it).

Like most projects this one has spun off several diversions.  But first, perhaps a bit of background:

I love the Gentoo distribution.  I like its simplicity, cleanness, and the fact that’s its one step off the beaten path so less likely to be a hack target.  All my Gentoo based systems are command line driven, no X-windows or the like.  This keeps them very clean with a fully functional base system typically having about 250 packages installed.  My test system, which I’m trying to keep squeaky clean, is at 173 packages with everything needed to run a guest KVM machine, VI, mdadm, lvm, libvirt, links, grub, etc.

Oh, I really needed a separate system for this so have breadboarded together an AMD Phenom II X6 (6-core) based system with 8GB of memory and (6) 500GB disk.  Suffice it to say that I REALLY like Raid-6ing my disk and 500GB, although not the smallest disk available these days, is close and cost just a few dollars more than the 320GB low-cost SATA drives I could find.

I simple did a minimal install following the instructions at the Gentoo site below.  Yes, I’ve been doing Gentoo for, oh goodness, perhaps a decade now, and yes, I still follow the instructions.  Its just easier and you tend not to forget things, like setting the root password!


An hour or so later all was functional, and I did a couple of full “emerge -e world” just to make sure all everything was clean and compiled specifically for my processor.  Oh, in case you were not aware, Gentoo is a source based distribution.  You can start with pre-compiled things if you like (and most do – its called a stage 3 install) and then recompile like I did, or slug your way through bootstrapping up a stage 1 install and be clean from the start.  I recommend the stage 3 install personally, its just not worth the extra effort to do a stage 1, but you can if you want to.

Learned one new trick doing this:  Rather than using a -march=i686 (for a very generic build that will run on virtually any Intel like system) or -march=athlon64 (for a system like mine), just use -march=native and GCC will figure it out, building things to use every bell and whistle that your processor offers.  Just don’t try and port the system to a different CPU type.  Yes, this will bite you if you decide later to change your motherboard to the “other side”, but you system will run optimally until then.

The first side project was to explore LVM (logical volume manager – a way to manage your disk space rather virtually) a bit.  I understood it in concept, the HP systems we bought decades ago used it, but I hadn’t personally played with it much.  The install link highlighted above gave me the basic commands I needed, but chickened out and only used LVM for the non-root partitions.  Turns out this is easy, you don’t even need to use an initramfs if you don’t want to (just add lvm to you boot runlevel).

Alas, if I was going to use LVM, I wanted to use it for everything (shy the /boot partition, which simply must be a real partition or at most a Raid 1 one – I’m using a 6 disk Raid 1.  Yes, overkill, but what else should I use that space for on each drive?).  Using it for the root partition mean I need to load the virtual device files for the volume groups prior to trying to mount root.  Cute trick?  Not really, things like that are why Linux supports initramfs systems.  They are RAM based file systems that you run an initial “init” process from (thus its name!).  Kernel support is trivial, just enable “initial RAM filesystem and RAM disk (initramfs/initrd) support” in your kernal configuration file.  The filesystem itself is just a gzipped cpio archive of a very simple root filesystem.  Mine contains statically linked copies of busybox, nano, ld-linux-x86-64.so and libc.so (for nano), a mdadm.conf file, statically linked copies of lvm, mdadm, and symbolic links for vgscan and vgchange to that lvm binary.  Oh, and a “init” file which is just the script to execute.  The init script is almost trivial, it mounts proc and devtmpfs, issues a mdadm assemble command to build the raid 6 devices and start them, displays /proc/mdstat for 5 seconds just as a “Look!  At least at this point everything looks good” comfort, does the vgscan and vgchange to enable all the lvm devices, mounts the lvm root partition read-only, unmounts proc and dev, then invokes busybox’s “switch_root” function to clean everything up, chroot to the real root system, and turn things over to init there.

Took a bit to figure that out, but as always, Google was my friend and my lvm based root system was functional!

Next was to build a KVM machines and get it to boot.  Again, Google and Gentoo documentation came to my aid.  I followed this link to get started:


Now one minor complication arose.  It seems the world loves to run graphics interfaces.  I like command line!  I can use Putty to access my systems from anywhere, and I don’t have to installed X-Windows, Gnome or KDE, and the hundreds of packages necessary to get them all up and running.  Took awhile, but like most things, its easy once you know how.  The key was simply to use the “-curses” flag when starting up the virtual machine.  I keep a “README” file in my KVM directory for reference.  I’ve cut and pasted that for your quick reference:

Base install created via:

qemu-img create -f qcow2 -o preallocation=metadata base-vm.img 20G

To boot from an ISO image:

kvm -hda base-vm.img -cdrom minimal.iso -boot d -curses -no-reboot

This tells KVM to use the .img file as a disk drive, use the .iso as the cdrom, boot
from the cdrom (“a” = Floppy, “c” = harddrive, “d” = cdrom, “n” = network)
and to use curses/ncurses for display.  That last is the key to being able to run KVM
in a text window.  The “-no-reboot” at the end causes the VM to exit back to the command
prompt when a reboot is attempted.  Handy when you want to change startup parameters.

Note this starts the system with only 128MB of memory.  Enough to boot.

Other good switches:

-m 512m  /* or whatever size you want */
-smp n /* sets the number of CPUs to n */
-echr chr  /* set terminal escape character instead of ctrl-a */
-runas user  /* nice for sorting  processes at hypervisor level */
-net user  /* auto sets up a firewalled virtual network */
-net tap,ifname=tap0,script=no,downscript=no
-net nic,macaddr=00:00:00:00:00:01 (edit: see my next post on KVM Bridging, this wasn’t needed)
-name string1[,process=string2] string1 sets the window title and string2 the process name

Using this with the small gentoo minimal install ISO image (144mb available from http://www.gentoo.org in the download section) brought me up a 24×80 text based window with the normal install boot running!  From that point I just did a normal install, formatting the virtual 20gb file created with qemu-img as if it was a 20gb disk drive, creating an lvm for /usr, /var, and /home, and doing the rest of the install in much the same way as the original machine install.  Note I did not LVM root in the virtual machine, I wanted to keep them as clean and simple as possible.  I gave /boot (/dev/sda1) 50mb of space, created a 1gb swap partition (/dev/sda2), a 256mb root (/dev/sda3), and allocated the rest to /dev/sda4 which I let LVM manage.  LVM in my base image has a 4GB /usr, a 8GB /var, and a 1GB /home.  /usr ended up being a bit tight, but that was mostly solved by nfs mounting /usr/portage/distfiles up from the host.  Makes sense to only have one copy of those files lying around anyhow – I’ll probably export them to my other Linux boxes one of these days.

One note that wasn’t real obvious.  KVM defaults to -net user.  This creates a firewalled user environment that can access the outside world using TCP and UDP, but can’t be accessed from it.  Think of it like a PC connected behind a typical router/firewall box.  Technically, it creates an IP address range of and provides a DHCP service to assign an IP address and route to the outside world.   From the minimal image boot, just do a “net-setup eth0”, select “wired”, and then “dhcp” and you will be alive network wise.  Note it does NOT support ICMP, so pings won’t work, but its easy to verify it working – just do a “links www.google.com” and make sure it responds (links is a text mode browser – a weird concept, but it works and is included in the minimal build).

Ah, finally!  I had a functional KVM based virtual machine.  Now, like all my computers, I like to have BOINC running (http://boinc.berkeley.edu).  All my spare CPU cycles go to a couple of projects under BOINC – mostly SETI@home and World Community Grid, but also Virtual Prairie whenever that project has work to be done.  So what could be better than a KVM running Boinc?  Sounded perfect to me.  All I did was:  cp base_vm.img bonic.img, issue the KVM command:

kvm -hda boinc.img -boot d -curses -no-reboot -m 6g -smp 6

and I had a clone of that original base install up and running.  I changed the IP address (which I had hardcoded in the base image) so it wouldn’t conflict, changed the root password, emerged boinc, attached to my projects, and watched everything run!  Took maybe 20 minutes, perhaps less, to create a fully operational KVM customized to the app I wanted to run.

Playing around a bit this morning, I now invoke it with:

nice -n 19 kvm -hda boinc.img -curses -no-reboot -m 6g -smp 6 -runas boinc_kvm -name boinc,process=kvm

This runs the KVM at the lowest priority, so I can do anything else I want without impact, gives it 75% of the memory if it wants to use it, runs on all the idle CPUs, runs it as user “boinc_kvm” so that I can easily track it usage, and called the long “qemu-kvm” command simply “kvm”.  Top shows it as:

19721 boinc_kv  39  19 6304m 1.1g 2596 S  599 14.0  10:26.20 kvm

Oh, one thing I noticed while building the KVM image and is shown above:  Even though we started the KVM with 6GB of memory (shown under the VIRT label), its only using 1.1G as unique memory for the process.  Thus, like with all VM environments, I should be able to “over subscribe” the memory and get away with it.

That pretty much brings this entry to a close.  I do have several more things I want to work on (and have been as those nasty side projects):

LibVirt looks like a cool interface for managing my virtual machines.  Took me a frustrating forever to figure out how to get it to behave.  One step was obvious, after emerging the code itself:  Start libvirtd in /etc/init.d.  The second was not:  an undocumented USE flag needed to be set when building libvirtd:  USE=qemu.  I added to it my /etc/make.conf USE variable and once libvirtd was rebuilt, it FINALLY started to work.  Prior to that I kept getting “unable to connect to hypervisor” errors.  There is also a /etc/libvirt/libvirtd.conf file that needs very minor and well internally commented tweaking.

The “-net user” works fine for my boinc KVM, but would not suffice for say an Apache KVM since the outside world would need access to it.  For that, a virtually bridged Tun/Tap environment will be needed.  I’ve played with that a bit (remember those nasty side projects?  Tun/Tap was one of them) but don’t quite have it, or do, and just need to test with something other than Ping?  Not sure.

I’d also like to play around with resizing the kvm image file.  Suspect that will be the easy one.

One final note for fellow Putty users (http://www.putty.org):  I find it handy to run putty in a slightly stretched window (add an inch of screen space top and to the sides).  Linux adapts to this and just gives you more screen space, but more importantly curses does NOT.  So if I duplicate my larger than default screen to create a window to run KVM in, and start as described above, the KVM machine console will appear with a blank border around it.  Sounds trivial, but really helps when your bouncing between windows to give you a visual reminder of what level your running at.




weekend crash

Posted by Kevin on August 21, 2011
Posted in Tech Talk  | No Comments yet, please leave one

Well… we are off to a rocky start.  Left for the weekend and come home to MySQL problems.  Apparently log files filled one of my filesystems that MySQL uses and MySQL takes great offense to that.  Believe I have everything repaired (Thank you Google!), but best I can tell, we were offline from around 4am to a bit after 10pm.

Going to see what I can do to put the logs in their own partition.

FWIW – I’ve run this (or previous generations of this) server since around 1993 and this is the first time I’ve had a filesystem fill on me.