1. Adam Megacz's Opinion
Look, I've been burned a billion times by the insanity of kernel modules and "initial ramdisks" (or whatever they're called nowadays). I am of the opinion that the main purpose of kernel modules is to make life easier for kernel developers, so they can try new code without rebooting. I still remain unconvinced that kernel modules are a good idea for normal users.
Life is massively simpler if you just compile your own kernel and put everything in statically. The beautiful thing about this is, if you compile the kernel statically, your bootloader is not broken, the root fs mounts and the root fs has /bin/bash, /dev/console, and /dev/null, then you can always recover from any damage. Note that if the kernel has boot-from-nfs or boot-from-afs compiled in, then all you need is a working bootloader and a kernel! If you rely on kernel modules, magical initial ramdisks, etc, the situation becomes much, much more fragile -- exponentially so if you need kernel modules (such as the md driver) to mount the root filesystem.
The above paragraph is not 100% true any longer. It used to be the case back in the good old days of devfs. However, devfs was retired for entirely nontechnical reasons (completely political/manpower-based issues) and now we have udev. Unfortunately, udev "overmounts" its own tempfs over the root filesystem's /dev, so once the system is up there's really no easy way to make sure that /dev/console and /dev/null files are in place "beneath" udev's overmount. This is a pain, but now that the world has decided on udev, there is no longer any alternative (I am personally of the opinion that a kernel hack to make these two devices "seem to" exist even if they aren't there would be a great idea). Devfs was much better because you could have it mounted automatically before process 0 (init) launched and it required no daemons or configuration files.
1.1. How to Do It
Grab the source tarball for the kernel. Do a make menuconfig, disable support for kernel modules, DO NOTHING ELSE, save, and exit. This will change all the "M" options to "*" -- it will change the default kernel config to use the same options, but to simply compile them statically rather than as modules. Unfortunately then you will need to go back in and re-enable module support because (for licensing reasons) the openafs code cannot be statically compiled into the kernel. But everything else can be.
It might make sense for hcoop to maintain a base kernel .config file, and keep a copy of the actual .config file used to build each of our kernels in some canonical place in /afs.
Also, it's a really good idea to enable access to the kernel .config file via /proc... there's an option for this, but last time I checked it was off-by-default. This is really useful, though.