Initial scratch notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working.

See http://wiki.hcoop.net/Migration2009/SoftwareSetup for the gist of what ClintonEbadi is trying to do here, but s/OpenVZ/KVM via libvirt/g.

1. Test Setup Notes

Nothing in particular order since it's all quite fuzzy

• Account clinton_admin has been added to the libvirt group (permits ClintonEbadi to manage kvms as his user remotely using virt-manager

• Investigated bridging and firewalling: https://bugzilla.redhat.com/show_bug.cgi?id=512206

  • This also implies that using a separate bridge per VM is ideal
  • As advised in the bug, we have disabled netfilter on the bridge

• Installed and configured: less sudo vim emacs23-nox etckeeper changetrack  openssh-server debsums logcheck bzip2 denyhosts rkhunter openafs-client ntp nscd krb5-user libpam-krb5 ssmtp libpam-afs-session openafs-krb5

1.1. Tasks

= done, = not done, = gave up or died trying

• Set up network bridge

• Create test KVM to discover preseed values and other config bits

• Generate basic preseed file where login + kinit && aklog work

• Create local Debian archive for libnss-afs

  • gnupg keyring etc. for verified package builds

  • Archive key for secure apt installs

  • Repackage libnss-afs for the current Debian standards versions (version is 3.7 which is unsupported; debuild promotes it to 4, but levels before 5 are deprecated...)

• Package nsswitch.conf changes and generate preseed for a machine that recognizes pts users (ssh $hcoop-user@machine should work at this point)

  • Update DebianPackaging with information on creating config packages and using dput to push them to the archive

• Preseed that installs libnss-afs and hcoop-nsswitch-config

• Update domtool init scripts to work with insserv since non-dependency-based init is deprecated and will be removed in wheezy

• Update FirewallRules closed.conf for the modern age and package

  • Add hostname field fwtool firewall config (so that users / services can have different ports on different machines)

  • Codify universal afs / kerberos / etc. ports that always have to be open in firewall config (can probably mostly yank this info from fritz)

  • Create hcoop-ferm-config package with default restrictive firewall for all nodes

• Apply advanced wine making techniques to carefully blend the Apache configurations on fritz and mire

  • Package the result

  • Add new phpVersion 53 to DomTool and (hopefully this can be done) make phpVersion support checking if the host supports that version (easy check: if the node is mire, support 4/5, if the node is fritz only support 5.3)

• Fix local user vs ptdb user for system services UID mismatch

• Spin up the fancy new Apache KVM and pray that it works

  • Move gitweb and git hosting over

  • Set up rcube

  • Set up squirrelmail (harder than rcube: we have to point mail. at the KVM, and then use MX records... punt on this for the time being)

  • Turn off fritz's Apache (it's the KVM host and KDC ... change of plans, eh)

  • Point hcoop.net at the new machine (also a huge reconfiguration PITA)

  • Start assisting the first brave users with "moving" to new machine (i.e. webAt "newNode", or adding an env var to Easy_Domain` to change the default web node for everything)

    • After sure of everything working, inspect all user DomTool configs and make the needed changes for the users to switch their hosting to new node (in trivial cases e.g. mod_proxy to app on mire, static file serving)

• Using lessons from above tasks, spin up new user shell machine

• Harrass any users who refuse to leave mire

• Turn mire off, remove from rack, set on fire

1.1.1. Other Tasks

These need to be done, but aren't going to kill anyone if they go undone until after the new machine is up.

• Package configurations

  • debarchiver

1.2. Packages Config

Things not mentioned on SetupNewMachines that had to have their default debconf values changed.

• ssmtp

  • forward all mail for UID < 1000 to logs

  • Masquerade as hcoop.net

• PAM
  • Newfangled pam-config framework for a fresh squeeze install looks quite promising... (enabled kerberos + unix + afs session)

1.3. Major Open issues

• Exim setup (have to add to forwardable domains on deleuze)
• Automated partitioning (looks like I might have to manually craft the partman template instead of dumping it from d-i)

• HostnameSuggestions (the results are eagerly awaited)

• Figuring out what to do wrt local users for system services that need to access afs
  • e.g. Apache, Exim, debarchiver, domtool, impad, spamd, ...
  • AFAICT, it makes more sense to just have afs users -- if the ptdb is gone, the services will not operate in a correct way anyway
  • Removes issues with keeping UIDs in sync
  • How does this interact with Debian automatically creating system users for packages?

  • A few system users were created using create-user -- mail is routed to them and they are subscribed to mailing lists and whatnot which is ... probably bad. i.e. We probably want to split create-user into the portions to just create an afs/kerberos user and then to do the fancy stuff an actual factual human user needs.

• Integration with package requests
  • Preseeding means we can kill/respin web node images with ease -- but not restore packages users have requested to support their cgi programs
  • If the portal stores this info, we need a package to reinstall user packages

2. Debian Mirror

See DebianPackaging (look ma, I kept the docs up to date)

2.1. Setup Notes

• Moved common.debian volume from deleuze to fritz (ack!)

  • Incidentally moved common.app and common.etc to fritz which might improve bugzilla/whatnot performance (despite network overhead, it's probably faster to serve from fritz to deleuze)

• Kept everything in one volume -- it's not too hard to split common.debian up later

• Created and destroyed afs user debarchiver

  • Issues with UID mismatches and nscd bit me again, created an afs only debian-archive user instead

• Gave debian-archive.daemon write permissions to /afs/hcoop.net/common/debian/archive

• Granted hcoop domtool perms to debian archive dir and set http://debian.hcoop.net/ to serve it (from fritz, but soon the www server...)

3. TODO

• Create an afs group that can write to the incoming directory of the archive

• Fix libnss-afs package for current Debian standards

• See why libnss-afs can no longer build with openafs 1.6 (not an issue at the moment, but it will be sooner than we expect...)

3.1. Config

As part of standardizing the config ... these should be put into hcoop-debarchiver-config and hcoop-dput-config

/etc/debarchiver.conf: see hopper, too long to include

/etc/cron.d/debarchiver: Unfortunately not quite working -- for some reasons this has to be done twice before Packages is updated (this happens with my local debarchiver so I ... have no idea)

#
# Regular cron jobs for the debarchiver package
#
# Run the archiver every five minutes.
*/5 * * * *     debian-archive  test -x /usr/bin/debarchiver && k5start -f /etc/keytabs/user.daemon/debian-archive -t -U -- debarchiver --autoscanall --addoverride | logger -t debarchiver -p daemon.info

4. Debian Based Package Config

Most info updated at DebianPackaging

Packages needing customization on all machines:

• ferm (closed.conf)

• nsswitch.conf

• sudoers + login.restrict

  • Change: list all admins explictly; adding admins to wheel on each machine is prone to decoherence. Less than ideal having to update two packages with per-node admin access info, but better than the situation before, and it can be improved.

• pam files to check login.restrict via pam_listfile on admin-only nodes

  • I think there is a pam config framework that we should use for this

• sshd (GSSAPI support)

• mdadm, rkhunter, tripwire, et al: This will need to be done as a general "CleaningUpOurAtrociouslyNoisyLoggingConfiguration" project (hint, hint).

Packages that need customization if installed:

• whatever imapd we use on the new machines
• exim
• ejabberd
• apache

Ideas:

• virtual packages hcoop-user-node-config and hcoop-services-node-config that conflict and depend on the appropriate basic config settings (e.g. for setting up login.restrict, default ulimits, etc.) This is trivial using equiv.

• If we want to use runit for services, we might include the service files and init.d overrides

• What copyright policy should we take on conffiles (are they copyrightable? ... at least disclaim copyright? Does basing them on debian base files mean they have to take the license of the package?)

4.1. Setup Notes

• Creating new archive section hcoop-config (MIT's athena uses debathena-config ... seems sane to me)

• DEB_AUTO_UPDATE_DEBIAN_CONTROL=yes fakeroot debian/rules clean does something magic

• Created initial hcoop-nsswitch-config, need to import into git and upload (see /afs/hcoop.net/user/c/cl/clinton_admin/hcoop-nsswitch-config-0/ for now)

5. Installer Preseeding

http://wiki.debian.org/DebianInstaller/Preseed

http://git.hcoop.net/?p=hcoop/machine-template.git;a=summary

Pretty useful, need to document more.

Installer command line: auto url=http://hcoop.net/preseed-test-0.cfg

Proof this is worth it (enter network info -> hot damn any afs user can login to the kvm)

CategorySystemAdministration CategoryWorkInProgress