FritzVirtualization

Initial scratch notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working.

See http://wiki.hcoop.net/Migration2009/SoftwareSetup for the gist of what ClintonEbadi is trying to do here, but s/OpenVZ/KVM via libvirt/g.

1. Misc Setup Notes

Nothing in particular order since it's all quite fuzzy

Account clinton_admin has been added to the libvirt group (permits ClintonEbadi to manage kvms as his user remotely using virt-manager
Investigated bridging and firewalling: https://bugzilla.redhat.com/show_bug.cgi?id=512206
- This also implies that using a separate bridge per VM is ideal
  - Need to test things locally and create appropriate bridges (${host}_br atop eth0?, br0:{1,2,3,...}?)
- As advised in the bug, we have disabled netfilter on the bridge
Installed and configured: less sudo vim emacs23-nox etckeeper changetrack openssh-server debsums logcheck bzip2 denyhosts rkhunter openafs-client ntp nscd krb5-user libpam-krb5 ssmtp libpam-afs-session openafs-krb5 (not true, see the preseed for the real list)
To build domtool, required packages: mlton ml-nlffigen libssl-dev libpcre3-dev

1.1. Tasks

= done, = not done, = possibly done, awaiting verification, = gave up or died trying

Set up network bridge
Create test KVM to discover preseed values and other config bits
Preseed Debian Install
- Generate basic preseed file where login + kinit && aklog work
- Custom partman config
  - /, /boot, /tmp, and /var/cache/openafs/
- Preseed that installs libnss-afs and hcoop-nsswitch-config
- Add hcoop archive to installed sources.list
Create local Debian archive
- gnupg keyring etc. for verified package builds
- Archive key for secure apt installs
Package nsswitch.conf changes and generate preseed for a machine that recognizes pts users (ssh $hcoop-user@machine should work at this point)
- Update DebianPackaging with information on creating config packages and using dput to push them to the archive
Update FirewallRules closed.conf for the modern age
- Add hostname field fwtool firewall config (so that users / services can have different ports on different machines)
- Codify universal afs / kerberos / etc. ports that always have to be open in firewall config (can probably mostly yank this info from fritz)
- Create hcoop-firewall-config package with default restrictive firewall for all nodes
  - Create empty system rule files as conffiles
  - Create per-machine system services conffiles package (alternative: hcoop-$service-config includes ferm.d rules)
Package essential configuration
- sudoers + login.restrict
  - Change: list all admins explictly; adding admins to wheel on each machine is prone to decoherence. Less than ideal having to update two packages with per-node admin access info, but better than the situation before, and it can be improved.
- pam files to check login.restrict via pam_listfile on admin-only nodes
  - I think there is a pam config framework that we should use for this (it doesn't help us for restricting logins but not auth in general)
- sshd (GSSAPI support)
  - restart sshd after install (punting for now, not really neccessary for preeseeding)
- krb5.conf (admin_server)
  - Fun fact: the SRV record based way of locating the admin server does not actually work (even in kerberos 1.10) so you still have to distribute a custom krb5.conf (listing admin servers in the admin_server clause of libdefaults). Feh!
  - Alias kerberos-adm.hcoop.net to whichever machine is the master KDC (hack until MitKerberos can use SRV records)
  - Set domain_realm. Default mapping logic seems like it would work, but it doesn't, feh!
Essential custom Debian packages
- Move libnss-afs sources from ~clinton_admin to repo source directory
- Check packaging of mod_waklog and include in repository
Install domtool (automate as much as possible)
- Backport mlton-tools from Wheezy
  - Through some wizardry ml-nlffigen was in lenny, didn't make it into squeeze, but got back into testing afterward?
- Update domtool init scripts to work with insserv since non-dependency-based init is deprecated and will be removed in wheezy
- Synchronize keytabs (just updating the existing script to sync to kvm is fine for the next year...)
- Fix aklog failure in domtool-admin-sudo so it actually works
  - or not: for some reason it works if you first nix all of your tokens...
Apply advanced wine making techniques to carefully blend the Apache configurations on fritz and mire
- Package the result
  - Remember apache-sync-logs cron job
- Add new phpVersion 53 to DomTool and (hopefully this can be done) make phpVersion support checking if the host supports that version (easy check: if the node is mire, support 4/5, if the node is fritz only support 5.3)
  - Can't be done -- we can only differentiate between 4 and 5 :-\
- Change defaultPhpVersion to 5
  - Check all user domtool configs and explicitly set phpVersion = 4 if needed
- Metapackage for php, basic cgi libraries, etc. (enough to bring up HCoop services)
  - OK if this is an ugly package
Name machines (HostnameSuggestions)
- User server: bog
- Web server: navajos (decided to use mccarthy for future admin node)
Integrate into infrastructure
- Sync keytabs
- DomTool configuration
  - Add as slave
  - Add as user webnode
  - Generate node certificate
- Route outgoing mail through deleuze
Automate post-install
- Extract host keytab
- Deploy domtool slave
- Create domtool node directories
  - Not sure of the best way to do this ... only nodes that run particular services ought to have the required directories and it seems better handled during domtool installation. For now we can live with a navajos-specific postinst (in addition to a generic one)
- Install packages needed for web server
  - Package hcoop-firewall-navajos-config with http + https
Spin up the fancy new Apache KVM and pray that it works
- Move gitweb and git hosting over
- Set up rcube
- Set up squirrelmail (harder than rcube: we have to point mail. at the KVM, and then use MX records... punt on this for the time being)
- Turn off fritz's Apache (it's the KVM host and KDC ... change of plans, eh)
- Point hcoop.net at the new machine (also a huge reconfiguration PITA)
- Start assisting the first brave users with "moving" to new machine (i.e. webAt "newNode", or adding an env var to Easy_Domain` to change the default web node for everything)
  - After sure of everything working, inspect all user DomTool configs and make the needed changes for the users to switch their hosting to new node (in trivial cases e.g. mod_proxy to app on mire, static file serving)
Using lessons from above tasks, spin up new user shell machine
Harrass any users who refuse to leave mire
- Remove php4 support from domtool
Turn mire off, remove from rack, set on fire

1.1.1. Other Tasks

These need to be done, but aren't going to kill anyone if they go undone until after the new machine is up. A lot of them were surfaced through the setup process, but we don't have a year to right every wrong...

Package configurations
- debarchiver
Fix local user vs ptdb user for system services UID mismatch
Repackage libnss-afs for the current Debian standards version
- Through some magic the package builds fine for the time being

1.2. Packages Config

Things not mentioned on SetupNewMachines that had to have their default debconf values changed.

ssmtp
- forward all mail for UID < 1000 to logs
- Masquerade as hcoop.net
PAM
- Newfangled pam-config framework for a fresh squeeze install looks quite promising... (enabled kerberos + unix + afs session)

1.3. Major Open issues

Exim setup (have to add to forwardable domains on deleuze)
Figuring out what to do wrt local users for system services that need to access afs
- e.g. Apache, Exim, debarchiver, domtool, impad, spamd, ...
- AFAICT, it makes more sense to just have afs users -- if the ptdb is gone, the services will not operate in a correct way anyway
- Removes issues with keeping UIDs in sync
- How does this interact with Debian automatically creating system users for packages?
- A few system users were created using create-user -- mail is routed to them and they are subscribed to mailing lists and whatnot which is ... probably bad. i.e. We probably want to split create-user into the portions to just create an afs/kerberos user and then to do the fancy stuff an actual factual human user needs.
Integration with package requests
- Preseeding means we can kill/respin web node images with ease -- but not restore packages users have requested to support their cgi programs
- If the portal stores this info, we need a package to reinstall user packages
Is the keytab situation messy? Having the domtool keytab at the toplevel seems out of place
- Reading old -sysadmin posts revealed that only having $user.daemon keys was supposed to be temporary -- we really should have a few standard principles for hcoop services to access data (e.g. $user.mail for mail delivery), and have a portal interface (and domtool integration) allowing users to request additional principles if they want (e.g. $user.$webapp-they-run ... or just $user.cgi in the default Just Works (tm) configuration)

1.3.1. fwtool

Making FirewallRules support all of the needed functionality for a user machine is proving difficult

Want to store per-node firewall config for system services (apache, exim, imap, etc.)
- Ideally, also store common port config (afs, kerberos, domtool, etc.)
Need to easily grant certain users additional permissions (e.g. all admins should be allowed outgoing ssh, normal users should not) on multiple nodes
Don't want to tie configuration to physical nodes (e.g. moving to a new shell server)
Restrict users to having rules on certain nodes (statically enforced)

Conclusion: the current fwtool implementation would require duplicating a lot of functionality already present in the support machinery for the domtool domain type. A new syntax for user rule files would need to be created (or tons of hackish supporting code) so ...

The only (in)sane way forward is to create a domtool node type and firewall plugin to manage rules. This has distinct advantages:

Takes advantage of current domtool infrastructure for pushing configs
The domtool language is quite nice and has the needed functionality for abstracting groups of users, common config, etc.
Lays the groundwork for using domtool to perform node management in addition to domain management

And a few distinct disadvantages:

Need to solve the uid mismatch between nodes for system users problem beforehand
- ClintonEbadi says: openafs pts shall be the only source for users (except those needed before afs can come up)
Pretty hefty time/code investment
- ClintonEbadi's SML-fu and type theory are weak (but knowledge is power™)
- The portal will need updating (but its interface to firewall rules sucks anyway)

Interim solution:

Getting a user shell machine online is slightly less important than shifting cgi hosting off of mire (load average is usually high, software is outdated). Users can live with for another month logging into an etch system but running their php and whatnot on a new machine... Therefore:

Common ferm config distributed using Debian package
- conffiles for local system ports created
Local ports config distributed using another Debian package (divert of site-wide conffiles)

This will force codification of the open ports for the web server machine, and will be easy to undo when domtool support is in place. A slightly hacked together FirewallRules may need to be used for the user node (time, what is time?) -- but a restrictive firewall must be used (it's impossible to implement one on a box that didn't have one before with breaking things).

2. Debian Mirror

See DebianPackaging (look ma, I kept the docs up to date)

2.1. Setup Notes

Moved common.debian volume from deleuze to fritz (ack!)
- Incidentally moved common.app and common.etc to fritz which might improve bugzilla/whatnot performance (despite network overhead, it's probably faster to serve from fritz to deleuze)
Kept everything in one volume -- it's not too hard to split common.debian up later
Created and destroyed afs user debarchiver
- Issues with UID mismatches and nscd bit me again, created an afs only debian-archive user instead
Gave debian-archive.daemon write permissions to /afs/hcoop.net/common/debian/archive
Granted hcoop domtool perms to debian archive dir and set http://debian.hcoop.net/ to serve it (from fritz, but soon the www server...)

3. TODO

Create an afs group that can write to the incoming directory of the archive
See why libnss-afs can no longer build with openafs 1.6 (not an issue at the moment, but it will be sooner than we expect...)
Debug debarchiver cron job (it isn't working, feh!)

3.1. Config

As part of standardizing the config ... these should be put into hcoop-debarchiver-config and hcoop-dput-config

/etc/debarchiver.conf: see hopper, too long to include

/etc/cron.d/debarchiver: Unfortunately not quite working -- for some reasons this has to be done twice before Packages is updated (this happens with my local debarchiver so I ... have no idea)

#
# Regular cron jobs for the debarchiver package
#
# Run the archiver every five minutes.
*/5 * * * *     debian-archive  test -x /usr/bin/debarchiver && k5start -f /etc/keytabs/user.daemon/debian-archive -t -U -- debarchiver --autoscanall --addoverride | logger -t debarchiver -p daemon.info

4. Debian Based Package Config

Most info updated at DebianPackaging

Packages needing customization on all machines:

mdadm, rkhunter, tripwire, et al: This will need to be done as a general "CleaningUpOurAtrociouslyNoisyLoggingConfiguration" project (hint, hint).

Packages that need customization if installed:

whatever imapd we use on the new machines
exim
ejabberd
apache

Ideas:

virtual packages hcoop-user-node-config and hcoop-services-node-config that conflict and depend on the appropriate basic config settings (e.g. for setting up login.restrict, default ulimits, etc.) This is trivial using equiv.
If we want to use runit for services, we might include the service files and init.d overrides
What copyright policy should we take on conffiles (are they copyrightable? ... at least disclaim copyright? Does basing them on debian base files mean they have to take the license of the package?)

4.1. Setup Notes

Creating new archive section hcoop-config (MIT's athena uses debathena-config ... seems sane to me)
DEB_AUTO_UPDATE_DEBIAN_CONTROL=yes fakeroot debian/rules clean does something magic
Created initial hcoop-nsswitch-config, need to import into git and upload (see /afs/hcoop.net/user/c/cl/clinton_admin/hcoop-nsswitch-config-0/ for now)