Spent more than a little time banging my head against the wall and freaking out that Google couldn’t help. What was the problem? I couldn’t get virtual network bridging to work between my KVM based guest and the rest of the world.
If I started my a KVM with -net user (or let it default), all was fine. By default, KVM will create a private network and provide dhcp and name services to the virtual instance. You can either use a dhcpd client or just hardcode ip addresses in the defined 10.0.2.x space. However, this “user” mode is designed as its named – it allows the VM to get out to the Internet, but doesn’t allow anything in. So it works great for building your image, and is even fine for running things like BOINC, but is rather limited. For instance, you can not SSH into the machine, so it better either run things automagically, or you will need to start it in a window.
For reasons unknown, if I started a KVM using a tun/tap (tap in this case) device it behaved oddly. Once in awhile, I’d see an eth0 device appear during boot (seen via reviewing the boot log using “dmesg”). As I tried with various “model=” options, I would see the corresponding driver load and sometimes a reference to eth0, but seldom anything that “ifconfig” could see.
Per most of the documentation I found online, the following startup command should have worked:
kvm -net nic,macaddr=00:00:00:00:01:00,model=virtio -net tap,ifname=tap0,script=no -drive file=test.img,if=virtio,boot=on -curses -no-reboot
And in fact, on those odd occasions when some variation of the above actually generated an eth0 device, it would appear with the macaddr specified. Still, in those cases, I could not ping or web (via links) outside to the world. (fyi: ping using ICMP, webbing using TCP, so it was good to check both given KVMs blocking of ICMP when using -net user). It was also freaky inconsistent. At one point I had two KVMs with eth0’s showing up on one but not the other. I did a side-by-side kernel build comparison and the kernels were identical. I did several reboots of each KVM, several kernal builds including a “make clean” pass, and it would work on one but not the other. Even when it did work, at the host level I often saw errors about the bridge receiving packets with its own address. I thought those were referring to TCP/IP addresses (and went down a path trying to figure that out), but in hindsight it was referring to MAC addresses!
Finally I decided to ignore the web help, and just start at the beginning using the man pages. Strange how that sometimes helps <smile>.
The breakthrough occured when I used the ever so slightly simplier command:
kvm -net nic,model=virtio -net tap,ifname=tap0,script=no -drive file=test.img,if=virtio,boot=on -curses -no-reboot
Note, the only difference is the lack of specifying the mac address.
Here are the relevant bits from /etc/conf.d/net:
tuntap_tap0=”tap”
config_tap0=”null”
tunctl_tap0=”-u root”
mac_tap0=”00:00:00:00:01:00″
bridge_br0=”eth1 tap0 tap1 tap2 tap3 tap4 tap5 tap6 tap7″
config_br0=”10.184.155.29/24″
routes_br0=”default gw 10.184.155.1″
A couple of notes: Using a mac address of all zeros generates a boot time error about not being able to reassign the hardware address the mac address represents. Any non-all-zero address works fine. Also, the bridge will take on the mac address of the first tap device defined. Also, this is a test environment, so I’ve given myself a lot of tap devices to play with. So, at the host level, we see:
ifconfig br0
br0 Link encap:Ethernet HWaddr 00:00:00:00:01:00
inet addr:10.184.155.29 Bcast:10.184.155.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:75628 errors:0 dropped:4930 overruns:0 frame:0
TX packets:22841 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:54238310 (51.7 MiB) TX bytes:8638206 (8.2 MiB)
ifconfig tap0
tap0 Link encap:Ethernet HWaddr 00:00:00:00:01:00
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:28733 errors:0 dropped:0 overruns:0 frame:0
TX packets:53601 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:35590127 (33.9 MiB) TX bytes:5638527 (5.3 MiB)
And within the KVM we see:
ifconfig eth0
eth0 Link encap:Ethernet HWaddr 52:54:00:12:34:56
inet addr:10.184.155.30 Bcast:10.184.155.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:44833 errors:0 dropped:0 overruns:0 frame:0
TX packets:28551 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3881750 (3.7 MiB) TX bytes:35412015 (33.7 MiB)
Much to my delight – this worked! KVM apparently uses the tap device, but assigns it a unique MAC address (the HWaddr shown in the ifconfig output). Now all generated IP packets have different MAC source addresses from the bridge and everything works as expected! My guest KVM can ping, web, and do anything else it wishes with the outside world (at least until I start configuring iptables, but that is another post). One glitch remained, the outside world didn’t know how to get the virtual machine. A quick route add -host 10.184.155.30 gw 10.184.155.29 allowed me to ssh into the vm. Still couldn’t get to it from a windows machine though. Emerging and starting mrouted seem to have fixed that. Not 100% sure that was needed, but its working, and at this point I’m going with “if its not broke, don’t fix it” as a philosophy.