Loopback Filesystems Revisited

In a post quite some time ago, I discussed loopback filesystems.  Well, lo and behold, a few weeks ago, I had the opportunity to play with them again.  Xen, a Linux VM Hypervisor (http://www.xen.org) mounts up its disk images as loopbacks.

I found out rather quickly, that the default number of loopback devices available to the Kernel is 8.  When you have one image for disk, and one for swap on each domU (virtual guest), that means you can only cram 4 machines onto a single hypervisor by default...  That's just unacceptable.

I found two methods that worked, although one is a bit more painful than the other:

Hypervisor with Guests running in Production:

When you can't reboot the machine for whatever reason (in my case, it was running VM's that I didn't have any space to migrate, and couldn't shut down), you can modprobe the loop module again and pass it a parameter:

# modprobe loop max_loop=128

This is all fine and good, really, except for the fact that you don't get nodes created for the loop devices in /dev automatically, so you have to do some lovely work with mknod before you can use your new devices, even though the kernel is aware of them:

# mknod -m660 /dev/loop8 b 7 8

Loop devices have a major number of 7, so make sure to change both 8's to the appropriate value for your system.  The defaults on the systems I've been working with (Debian Lenny, Debian Squeeze, and Ubuntu Precise (12.04 LTS)) are to have 8 loop devices, loop0 - loop7.

Of course making 120 new loop devices by hand isn't exactly fun... Time for bash-fu!

# for x in `seq 8 128`; do mknod -m660 /dev/loop$x b 7 $x; done;

This gives us our 128 loopback devices (assuming you had 0-7 OOTB), with the nodes created in /dev.  If you reboot, they're gone, so make sure to add the options to /etc/modules.conf or wherever your flavor decided to keep the module options.

Hypervisor without Guests (or can be rebooted):

If the machine can be rebooted, the situation is much easier.  My solution was to pass the parameter to the kernel at boot time via GRUB.

Simply set the following in /etc/default/grub (for Debian based distros):

GRUB_CMDLINE_LINUX_DEFAULT="max_loop=128 quiet"

Then rebuild your GRUB config and reboot:

# update-grub && shutdown -r now

When the machine comes back up, GRUB will have passed the max_loop option to the kernel at boot, and the kernel will have automatically created all of your loopback devices for you.

Obviously, this is a significantly simpler operation than creating the nodes in place, and if you have the option to reboot, I would say go for it.

Part I

Victory!

After switching virtual network cards in my hypervisor, I was able to get RH7.1 online.  Textutils compiled as statically linked binaries just as expected, and ran like a mo-f'ing CHAMP on the dettuxx machine.  This allowed me to download coreutils-5.0 and compile/install it.

I ran into another circular dependency with awk (which was a pre-req of upgrading Bash), and was able to cross compile the latest gawk and get it installed just fine via the RH7.1 machine.  Now I'm going to try compiling binutils, as another pre-req of bash, and see what happens... Hopefully I won't have to cross compile too much more.

Several failed attempts...

Apparently, in order to compile textutils, you need to have textutils installed...  Naturally (to me anyway), my first thought was Busybox.  I tried to compile it, no joy (unmet dependancies), I tried the appropriate binaries for my VM, also no joy... insta-segfault.

Fine, let's get the real sort util, and copy it over, perfect!  So I spend the next 20 minutes configuring another (32bit) VM to support SSH1 on the internal network (since, of course, the version of SSH with Dettuxx is ancient), and get scp working.  Copy the binary over, and it crashes complaining about libraries. File confirmed that it was dynamically linked, with libs that weren't installed in Dettuxx, no doubt.  Fine...

export CFLAGS="-static -O2 -g"
./configure --with-prefix=/home/peter/textutils_build
make
make install

This process went along nicely, and generated a statically linked set of textutils bins (the old textutils package wasn't exactly fun to find, mind you).  I scp these binaries over, and get:  "FATAL:  Kernel too old"

As we speak, I have RH7.1 installing in a VM, running a 2.4 kernel out of the box...  Wish me luck...

Back for more...?

So, all these years later, I decided to give DettuxX another try (I mean... we have VM's now... why not), and was able to get it installed, bootable, and on the interwebs last night...  Thats when I remembered just how much of a bitch it is.  I, by some act of God, was able to get gcc compiled, but that's about as far as I got before I started running into... problems.  In order to compile most things, you need to have sort (which was a part of textutils when the distro was released... now textutils has been rolled into coreutils).  Unfortunately, there's a circular dependancy there.  Textutils requires sort to successfully configure, and sort is a part of Textutils...  Guess it's time to evaluate busy box, and the prospect of cross compiling textutils on another machine and copying it over...  Joy...