Header image alt text

Kevin's Thoughts!

Maybe you agree, maybe you don't… find out!

KVM – libvirt

Posted by Kevin on August 31, 2011
Posted in Tech Talk  | Tagged With: , , | No Comments yet, please leave one

I think I’m done playing with Libvirt for the moment.  As a general purpose library management tool it looks good.  From an ease-of-use perspective I suspect its not bad if you are running in an X-windows environment.  From a “lets convert” standpoint?  Well, that is proving to be another issue.

For instance, I currently start up my KVMs with a command similar to:

kvm -net nic,model=virtio -net tap,ifname=tap6,script=no  -drive file=test2.img,if=virtio,boot=on -curses -no-reboot -m 2g -smp 2

This starts a fully paravirtualized instance using virtio for both networking and the disk image.  The resulting VM has 2gb of memory and 2 virtual CPUs allocated to it.

Libvirt uses XML files to control everything, an idea I fully support.  However getting started creating those can be a challenge.  I found two approaches.  The first was with a related tool called “virt-install”.  It is not part of the libvirt, but built using it and closely tied to it.  So after playing for a bit, I came up with:

virt-install –name “test2” –ram 2048 –vcpus=2  –cpu host –description “Test VM” –import –file test2.img –os-type=linux –os-variant=virtio26  –network bridge=br0,model=virtio –graphics=none   –noreboot

Using the virsh list command shows the VM has been created and is running!  Probing around a bit shows the file /etc/libvirt/qemu/test2.xml has been created.  Using virsh edit test2 puts me into a VI editing session on the XML.  All looks good.  Now to connect.  Connect?  Connect to what?  I try opening a ssh session and it times out.  Unlike when I issue the KVM command directly, there is no associated curses window to look it.  Looking at /var/log/libvirt/qemu shows a log of the command being issued… very useful!  Apparently a serial console connection has been opened on /dev/pts/2.  Ok, crank up screen /dev/pts/2 and it connects, but there is no output…  Looking at the log file shows a few concerning things:  1) KVM is being started up with a “-S” switch, which the developers tell me is normal, but the man-pages say disables the startup of the CPUs.  Its sure behaving like the man-pages are right.  2) the image file is being started with “if=none” instead of “if=virtio”.  A bit of command line testing verifies that “if=none” prevents the kernel from finding the root disk.  Not good.  Checked in with the user-list community and the generated XML looks appropriate, so this issue remains pending.  Finally, the network is not being started using tap6, my assigned network device for this VM.

The user community suggested I try an alternative method to generate the initial XML.  Apparently virsh amoungst its MANY options, has one for generating XML from a command line!  Cool.  Wish I had noticed that earlier.  The command:

virsh domxml-from-native qemu-argv /kvms/test2 >test2.xml

Takes the file test (which contains the KVM command at the top of this post) and converts it to equivalent xml for libvirt.  Very Cool!  Well, at least in concept.  It has a few very minor parsing problems, like the “2g” for memory must be specified as “2048” (in megabytes).  Also the kvm command must be fully qualified as /usr/bin/kvm.  Other than that, its works fine.  The generated xml looks very similar to that created by virt-install with a few minor differences like the instance not being named (ok, ok, its named “unnamed”) and the OS architecture and CPU information is generic X86 instead of the very specific AMD statements generated by virt-install.  There is also no serial port defined like virt-install did, but there is a “ethernet” device now that reflects my tap6 port.  A bit of cut & pasting and I managed to create a custom.xml with the best of both worlds.  Issuing virsh define /kvms/custom.xml imports the new xml file and again defines and tries to starts a running instance.

Notice the “tries” word above?  The network won’t start, so the instance is stopped.  The network won’t start because the “tap” device apparently needs to be a “tun” device.  Again looking at the log file shows the same if=none for the disk image as well.  Even if I reconfigured my network to use “tun” devices instead of “tap” devices, the VM would not get past booting the kernel.

I’ve spent a significant amount of time browsing the various man pages to try and figure out the above issues.  I did discover that libvirt appears to be more “cutting edge” than even the man pages for kvm.  e.g.  It uses some “newer” format parameters that I couldn’t find in the man pages (both local and on the official qemu site), which was a bit frustrating, but the format was pretty obvious. It also appears that the libvirt world really is built around VMs being behind a NAT subnet which libvirt wants to control.  It may be possible to get around that, but so far I haven’t had any luck.

So, for now, I’ll just manage my VMs from the command line, and start them via /etc/local.d/local.start.



KVM Management – Libvirt

Posted by Kevin on August 29, 2011
Posted in Technology  | Tagged With: , , , | No Comments yet, please leave one

Hi all.

I’ve been playing around with KVM for a bit now and have several virtual machines up and running.  Having passed what I considered “first look” stage, I’m ready to start using libvirt as a manager.  For reference, pre-libvirt, I have started this VM with:

kvm -net nic,model=virtio -net tap,ifname=tap5,script=no  -drive file=test1.img,if=virtio,boot=on -curses -no-reboot -m 1g -smp 2

I have a test environment I like to play with, and created the initial domain XML using virt-install:

virt-install –name “Test” –ram 1024 –vcpus=2  –cpu host –description “Test VM” –import –file test1.img –os-type=linux –os-variant=virtio26  –network bridge=br0,model=virtio –graphics=none –autostart –prompt

When I look in /var/log/libvirt/qemu/Test.log  I see the VM is started with the command (edited down to the relevant parts):

/usr/bin/qemu-kvm -S -m 1024 -smp 2 -name Test,process=qemu:Test -nographics -boot c -drive file=/kvms/test1.img,if=none,boot=on

This has two problems:

1) The “-S” is keeping the system from booting
2)  the “if=none” needs to be “if=virtio”

I’ve confirmed both by changing -nographics to -curses and watching what happens.  For #1, its a big fat nothing (no surprise since -S turns off the CPUs), for #2 grub loads and the kernel boots, but my /dev/vda3 isn’t found so “init” can’t start.

I’m anticipating a 3rd problem:  I don’t see where to specify that I want the VM to use “tap5” of my bridged network.

I have to imagine these things can be corrected by editing the Libvirt XML file for the domain Test.  I’m asking the Libvirt mailing list https://www.redhat.com/mailman/listinfo/libvirt-users for help with this one!

With VMWare recently announcing a new pricing structure based on total virtual memory size instead of server size, I decided that perhaps another look at KVM was appropriate.  KVM is open-source and built into the Linux kernel (presuming you enable it).

Like most projects this one has spun off several diversions.  But first, perhaps a bit of background:

I love the Gentoo distribution.  I like its simplicity, cleanness, and the fact that’s its one step off the beaten path so less likely to be a hack target.  All my Gentoo based systems are command line driven, no X-windows or the like.  This keeps them very clean with a fully functional base system typically having about 250 packages installed.  My test system, which I’m trying to keep squeaky clean, is at 173 packages with everything needed to run a guest KVM machine, VI, mdadm, lvm, libvirt, links, grub, etc.

Oh, I really needed a separate system for this so have breadboarded together an AMD Phenom II X6 (6-core) based system with 8GB of memory and (6) 500GB disk.  Suffice it to say that I REALLY like Raid-6ing my disk and 500GB, although not the smallest disk available these days, is close and cost just a few dollars more than the 320GB low-cost SATA drives I could find.

I simple did a minimal install following the instructions at the Gentoo site below.  Yes, I’ve been doing Gentoo for, oh goodness, perhaps a decade now, and yes, I still follow the instructions.  Its just easier and you tend not to forget things, like setting the root password!


An hour or so later all was functional, and I did a couple of full “emerge -e world” just to make sure all everything was clean and compiled specifically for my processor.  Oh, in case you were not aware, Gentoo is a source based distribution.  You can start with pre-compiled things if you like (and most do – its called a stage 3 install) and then recompile like I did, or slug your way through bootstrapping up a stage 1 install and be clean from the start.  I recommend the stage 3 install personally, its just not worth the extra effort to do a stage 1, but you can if you want to.

Learned one new trick doing this:  Rather than using a -march=i686 (for a very generic build that will run on virtually any Intel like system) or -march=athlon64 (for a system like mine), just use -march=native and GCC will figure it out, building things to use every bell and whistle that your processor offers.  Just don’t try and port the system to a different CPU type.  Yes, this will bite you if you decide later to change your motherboard to the “other side”, but you system will run optimally until then.

The first side project was to explore LVM (logical volume manager – a way to manage your disk space rather virtually) a bit.  I understood it in concept, the HP systems we bought decades ago used it, but I hadn’t personally played with it much.  The install link highlighted above gave me the basic commands I needed, but chickened out and only used LVM for the non-root partitions.  Turns out this is easy, you don’t even need to use an initramfs if you don’t want to (just add lvm to you boot runlevel).

Alas, if I was going to use LVM, I wanted to use it for everything (shy the /boot partition, which simply must be a real partition or at most a Raid 1 one – I’m using a 6 disk Raid 1.  Yes, overkill, but what else should I use that space for on each drive?).  Using it for the root partition mean I need to load the virtual device files for the volume groups prior to trying to mount root.  Cute trick?  Not really, things like that are why Linux supports initramfs systems.  They are RAM based file systems that you run an initial “init” process from (thus its name!).  Kernel support is trivial, just enable “initial RAM filesystem and RAM disk (initramfs/initrd) support” in your kernal configuration file.  The filesystem itself is just a gzipped cpio archive of a very simple root filesystem.  Mine contains statically linked copies of busybox, nano, ld-linux-x86-64.so and libc.so (for nano), a mdadm.conf file, statically linked copies of lvm, mdadm, and symbolic links for vgscan and vgchange to that lvm binary.  Oh, and a “init” file which is just the script to execute.  The init script is almost trivial, it mounts proc and devtmpfs, issues a mdadm assemble command to build the raid 6 devices and start them, displays /proc/mdstat for 5 seconds just as a “Look!  At least at this point everything looks good” comfort, does the vgscan and vgchange to enable all the lvm devices, mounts the lvm root partition read-only, unmounts proc and dev, then invokes busybox’s “switch_root” function to clean everything up, chroot to the real root system, and turn things over to init there.

Took a bit to figure that out, but as always, Google was my friend and my lvm based root system was functional!

Next was to build a KVM machines and get it to boot.  Again, Google and Gentoo documentation came to my aid.  I followed this link to get started:


Now one minor complication arose.  It seems the world loves to run graphics interfaces.  I like command line!  I can use Putty to access my systems from anywhere, and I don’t have to installed X-Windows, Gnome or KDE, and the hundreds of packages necessary to get them all up and running.  Took awhile, but like most things, its easy once you know how.  The key was simply to use the “-curses” flag when starting up the virtual machine.  I keep a “README” file in my KVM directory for reference.  I’ve cut and pasted that for your quick reference:

Base install created via:

qemu-img create -f qcow2 -o preallocation=metadata base-vm.img 20G

To boot from an ISO image:

kvm -hda base-vm.img -cdrom minimal.iso -boot d -curses -no-reboot

This tells KVM to use the .img file as a disk drive, use the .iso as the cdrom, boot
from the cdrom (“a” = Floppy, “c” = harddrive, “d” = cdrom, “n” = network)
and to use curses/ncurses for display.  That last is the key to being able to run KVM
in a text window.  The “-no-reboot” at the end causes the VM to exit back to the command
prompt when a reboot is attempted.  Handy when you want to change startup parameters.

Note this starts the system with only 128MB of memory.  Enough to boot.

Other good switches:

-m 512m  /* or whatever size you want */
-smp n /* sets the number of CPUs to n */
-echr chr  /* set terminal escape character instead of ctrl-a */
-runas user  /* nice for sorting  processes at hypervisor level */
-net user  /* auto sets up a firewalled virtual network */
-net tap,ifname=tap0,script=no,downscript=no
-net nic,macaddr=00:00:00:00:00:01 (edit: see my next post on KVM Bridging, this wasn’t needed)
-name string1[,process=string2] string1 sets the window title and string2 the process name

Using this with the small gentoo minimal install ISO image (144mb available from http://www.gentoo.org in the download section) brought me up a 24×80 text based window with the normal install boot running!  From that point I just did a normal install, formatting the virtual 20gb file created with qemu-img as if it was a 20gb disk drive, creating an lvm for /usr, /var, and /home, and doing the rest of the install in much the same way as the original machine install.  Note I did not LVM root in the virtual machine, I wanted to keep them as clean and simple as possible.  I gave /boot (/dev/sda1) 50mb of space, created a 1gb swap partition (/dev/sda2), a 256mb root (/dev/sda3), and allocated the rest to /dev/sda4 which I let LVM manage.  LVM in my base image has a 4GB /usr, a 8GB /var, and a 1GB /home.  /usr ended up being a bit tight, but that was mostly solved by nfs mounting /usr/portage/distfiles up from the host.  Makes sense to only have one copy of those files lying around anyhow – I’ll probably export them to my other Linux boxes one of these days.

One note that wasn’t real obvious.  KVM defaults to -net user.  This creates a firewalled user environment that can access the outside world using TCP and UDP, but can’t be accessed from it.  Think of it like a PC connected behind a typical router/firewall box.  Technically, it creates an IP address range of and provides a DHCP service to assign an IP address and route to the outside world.   From the minimal image boot, just do a “net-setup eth0”, select “wired”, and then “dhcp” and you will be alive network wise.  Note it does NOT support ICMP, so pings won’t work, but its easy to verify it working – just do a “links www.google.com” and make sure it responds (links is a text mode browser – a weird concept, but it works and is included in the minimal build).

Ah, finally!  I had a functional KVM based virtual machine.  Now, like all my computers, I like to have BOINC running (http://boinc.berkeley.edu).  All my spare CPU cycles go to a couple of projects under BOINC – mostly SETI@home and World Community Grid, but also Virtual Prairie whenever that project has work to be done.  So what could be better than a KVM running Boinc?  Sounded perfect to me.  All I did was:  cp base_vm.img bonic.img, issue the KVM command:

kvm -hda boinc.img -boot d -curses -no-reboot -m 6g -smp 6

and I had a clone of that original base install up and running.  I changed the IP address (which I had hardcoded in the base image) so it wouldn’t conflict, changed the root password, emerged boinc, attached to my projects, and watched everything run!  Took maybe 20 minutes, perhaps less, to create a fully operational KVM customized to the app I wanted to run.

Playing around a bit this morning, I now invoke it with:

nice -n 19 kvm -hda boinc.img -curses -no-reboot -m 6g -smp 6 -runas boinc_kvm -name boinc,process=kvm

This runs the KVM at the lowest priority, so I can do anything else I want without impact, gives it 75% of the memory if it wants to use it, runs on all the idle CPUs, runs it as user “boinc_kvm” so that I can easily track it usage, and called the long “qemu-kvm” command simply “kvm”.  Top shows it as:

19721 boinc_kv  39  19 6304m 1.1g 2596 S  599 14.0  10:26.20 kvm

Oh, one thing I noticed while building the KVM image and is shown above:  Even though we started the KVM with 6GB of memory (shown under the VIRT label), its only using 1.1G as unique memory for the process.  Thus, like with all VM environments, I should be able to “over subscribe” the memory and get away with it.

That pretty much brings this entry to a close.  I do have several more things I want to work on (and have been as those nasty side projects):

LibVirt looks like a cool interface for managing my virtual machines.  Took me a frustrating forever to figure out how to get it to behave.  One step was obvious, after emerging the code itself:  Start libvirtd in /etc/init.d.  The second was not:  an undocumented USE flag needed to be set when building libvirtd:  USE=qemu.  I added to it my /etc/make.conf USE variable and once libvirtd was rebuilt, it FINALLY started to work.  Prior to that I kept getting “unable to connect to hypervisor” errors.  There is also a /etc/libvirt/libvirtd.conf file that needs very minor and well internally commented tweaking.

The “-net user” works fine for my boinc KVM, but would not suffice for say an Apache KVM since the outside world would need access to it.  For that, a virtually bridged Tun/Tap environment will be needed.  I’ve played with that a bit (remember those nasty side projects?  Tun/Tap was one of them) but don’t quite have it, or do, and just need to test with something other than Ping?  Not sure.

I’d also like to play around with resizing the kvm image file.  Suspect that will be the easy one.

One final note for fellow Putty users (http://www.putty.org):  I find it handy to run putty in a slightly stretched window (add an inch of screen space top and to the sides).  Linux adapts to this and just gives you more screen space, but more importantly curses does NOT.  So if I duplicate my larger than default screen to create a window to run KVM in, and start as described above, the KVM machine console will appear with a blank border around it.  Sounds trivial, but really helps when your bouncing between windows to give you a visual reminder of what level your running at.