I found out rather quickly, that the default number of loopback devices available to the Kernel is 8. When you have one image for disk, and one for swap on each domU (virtual guest), that means you can only cram 4 machines onto a single hypervisor by default... That's just unacceptable.
I found two methods that worked, although one is a bit more painful than the other:
Hypervisor with Guests running in Production:
When you can't reboot the machine for whatever reason (in my case, it was running VM's that I didn't have any space to migrate, and couldn't shut down), you can modprobe the loop module again and pass it a parameter:
# modprobe loop max_loop=128
This is all fine and good, really, except for the fact that you don't get nodes created for the loop devices in /dev automatically, so you have to do some lovely work with mknod before you can use your new devices, even though the kernel is aware of them:
# mknod -m660 /dev/loop8 b 7 8
Loop devices have a major number of 7, so make sure to change both 8's to the appropriate value for your system. The defaults on the systems I've been working with (Debian Lenny, Debian Squeeze, and Ubuntu Precise (12.04 LTS)) are to have 8 loop devices, loop0 - loop7.
Of course making 120 new loop devices by hand isn't exactly fun... Time for bash-fu!
# for x in `seq 8 128`; do mknod -m660 /dev/loop$x b 7 $x; done;
This gives us our 128 loopback devices (assuming you had 0-7 OOTB), with the nodes created in /dev. If you reboot, they're gone, so make sure to add the options to /etc/modules.conf or wherever your flavor decided to keep the module options.
Hypervisor without Guests (or can be rebooted):
If the machine can be rebooted, the situation is much easier. My solution was to pass the parameter to the kernel at boot time via GRUB.
Simply set the following in /etc/default/grub (for Debian based distros):
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=128 quiet"
Then rebuild your GRUB config and reboot:
# update-grub && shutdown -r now
When the machine comes back up, GRUB will have passed the max_loop option to the kernel at boot, and the kernel will have automatically created all of your loopback devices for you.
Obviously, this is a significantly simpler operation than creating the nodes in place, and if you have the option to reboot, I would say go for it.
