Initial scratch notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working.

See http://wiki.hcoop.net/Migration2009/SoftwareSetup for the gist of what ClintonEbadi is trying to do here, but s/OpenVZ/KVM via libvirt/g.

1. Tasks

= done, = not done, = possibly done, awaiting verification, = gave up or died trying

• Apply advanced wine making techniques to carefully blend the Apache configurations on fritz and mire

  • Per machine NameVirtualHost config

    • It turns out that both the NameVirtualHost and VirtualHost directive must use * or an explicit IP. For the sake of correctness, keeping the IP in VirtualHost directives seems like a Good Idea (tm), so we need to have domtool install /etc/apache2/conf.d/hcoop-namevhost-$machine for every web serving node.

  • Per-machine apache default vhost

  • Change defaultPhpVersion to 5

    • Check all user domtool configs and explicitly set phpVersion = 4 if needed

  • Domtool mod_proxy support to machines other than localhost

  • Check DomTool for all used modules and sure they are enabled

• Spin up the fancy new Apache KVM and pray that it works

  • Reinstall using preseed + postinst

  • Make openafs start earlier in boot process

    • Things like apache need to resolve pts users; it's easier to divert/transform the openafs script than to divert every script that needs access to afs uids.

    • This was really hard. insserv in squeeze has weird problems.

  • Move gitweb and git hosting over

  • Set up rcube

  • Install squirrelmail at webmail.hcoop.net temporarily (mod_proxy from deleuze later)

  • Turn off fritz's Apache (it's the KVM host and KDC ... change of plans, eh)

    • Deal with http://debian.hcoop.net (if we move it to navajos, we can't reinstall navajos... but we probably should move it to navajos)

  • Move mire web services

    • phpmyadmin

    • ajaxterm

  • Move unknownlamer.org onto navajos (what better a guinea pig)

  • Point hcoop.net at the new machine (also a huge reconfiguration PITA)

  • Start assisting the first brave users with "moving" to new machine (i.e. webAt "newNode", or adding an env var to Easy_Domain to change the default web node for everything)

    • DavorOcelic

      • Fix SSL problems

    • SteveKillen

      • Fix moinmoin-install

    • BtTempleton

    • After sure of everything working, inspect all user DomTool configs and make the needed changes for the users to switch their hosting to new node (in trivial cases e.g. mod_proxy to app on mire, static file serving)

• Databases

  • Set up postgresql 9.1

    • And we're even using kerberos instead of ident for auth!

  • Install postgres/mysql client libraries

  • Firewall rules for connecting to database server

• Using lessons from above tasks, spin up new user shell machine

• Harrass any users who refuse to leave mire

  • Remove php4 support from domtool

• Turn mire off, remove from rack, set on fire

  • Move secondary DNS to hopper, update ns aliases

• Migrate web services from deleuze

  • Migrate and upgrade hcoop wiki

    • Ideally into afs space, owned by the hcoop account.
    • Or not? One reason for it being on local fs space is to remain accessible in case kerberos/afs are gone. This is less of an issue now (and will be even less so once we have multiple volservers and ensure the hcoop volume is on all of them), so might not be worth it. Alternative: set up automatic moin mirroring to outpost using a trivial lighttpd setup.

  • Migrate Portal

    • Move all data storage into afs

    • Hack needed 32-bit libraries

    • Update SMLSQL for libpq5

Other tasks, lower priority:

• Configuration package nits

  • ssh: restart sshd after installation

    • Actually easy: just setup a trivial postinst/prerm ala hcoop-apache2-config

  • firewall: restart ferm after installation

2. Bog Initial Setup Notes

The preseed in theory is enough to have a user node; the admin config package restricts logins. A few open questions:

• Do user nodes need to run domtool-slave?
  • Yes -- for fwtool at the very least

• Add /afs/hcoop.net/common/bin to default path using /etc/profile.d/

  • Should we use @sys? We have no binaries installed right now, can probably skip.

Check preseed for installation of admin node specific packages and move to navajos postinst. We need some preseed templating system eventually, but for now...

Automatic re-installability is officially a low priority for this release cycle... so don't worry about post-installing or preseeding the various packages a fully featured user node needs (db clients, perl, etc.).

3. Packages Config

Things not mentioned on SetupNewMachines that had to have their default debconf values changed.

• ssmtp

  • forward all mail for UID < 1000 to logs

  • Masquerade as hcoop.net

• PAM
  • Newfangled pam-config framework for a fresh squeeze install looks quite promising... (enabled kerberos + unix + afs session)

4. Major Open issues

• Exim setup (have to add to forwardable domains on deleuze)
• Figuring out what to do wrt local users for system services that need to access afs
  • e.g. Apache, Exim, debarchiver, domtool, impad, spamd, ...
  • AFAICT, it makes more sense to just have afs users -- if the ptdb is gone, the services will not operate in a correct way anyway
  • Removes issues with keeping UIDs in sync
  • How does this interact with Debian automatically creating system users for packages?

  • A few system users were created using create-user -- mail is routed to them and they are subscribed to mailing lists and whatnot which is ... probably bad. i.e. We probably want to split create-user into the portions to just create an afs/kerberos user and then to do the fancy stuff an actual factual human user needs.

• Integration with package requests
  • Preseeding means we can kill/respin web node images with ease -- but not restore packages users have requested to support their cgi programs
  • If the portal stores this info, we need a package to reinstall user packages
• Is the keytab situation messy? Having the domtool keytab at the toplevel seems out of place
  • Reading old -sysadmin posts revealed that only having $user.daemon keys was supposed to be temporary -- we really should have a few standard principles for hcoop services to access data (e.g. $user.mail for mail delivery), and have a portal interface (and domtool integration) allowing users to request additional principles if they want (e.g. $user.$webapp-they-run ... or just $user.cgi in the default Just Works (tm) configuration)

4.1. fwtool

Making FirewallRules support all of the needed functionality for a user machine is proving difficult

• Want to store per-node firewall config for system services (apache, exim, imap, etc.)
  • Ideally, also store common port config (afs, kerberos, domtool, etc.)
• Need to easily grant certain users additional permissions (e.g. all admins should be allowed outgoing ssh, normal users should not) on multiple nodes
• Don't want to tie configuration to physical nodes (e.g. moving to a new shell server)
• Restrict users to having rules on certain nodes (statically enforced)

Conclusion: the current fwtool implementation would require duplicating a lot of functionality already present in the support machinery for the domtool domain type. A new syntax for user rule files would need to be created (or tons of hackish supporting code) so ...

The only (in)sane way forward is to create a domtool node type and firewall plugin to manage rules. This has distinct advantages:

• Takes advantage of current domtool infrastructure for pushing configs
• The domtool language is quite nice and has the needed functionality for abstracting groups of users, common config, etc.
• Lays the groundwork for using domtool to perform node management in addition to domain management

And a few distinct disadvantages:

• Need to solve the uid mismatch between nodes for system users problem beforehand

  • ClintonEbadi says: openafs pts shall be the only source for users (except those needed before afs can come up)

• Pretty hefty time/code investment

  • ClintonEbadi's SML-fu and type theory are weak (but knowledge is power™)

  • The portal will need updating (but its interface to firewall rules sucks anyway)

Interim solution:

Getting a user shell machine online is slightly less important than shifting cgi hosting off of mire (load average is usually high, software is outdated). Users can live with for another month logging into an etch system but running their php and whatnot on a new machine... Therefore:

• Common ferm config distributed using Debian package
  • conffiles for local system ports created
• Local ports config distributed using another Debian package (divert of site-wide conffiles)

This will force codification of the open ports for the web server machine, and will be easy to undo when domtool support is in place. A slightly hacked together FirewallRules may need to be used for the user node (time, what is time?) -- but a restrictive firewall must be used (it's impossible to implement one on a box that didn't have one before with breaking things). Unfortunately, without SELinux, we can't restrict what ports members listen on, so input firewall rules will be less useful than they could be for now.

5. Installer Preseeding

http://wiki.debian.org/DebianInstaller/Preseed

http://git.hcoop.net/?p=hcoop/machine-template.git;a=summary

Pretty useful, need to document more.

Installer command line: auto url=http://hcoop.net/~clinton_admin/preseed-test-0.cfg

Proof this is worth it (enter network info -> hot damn any afs user can login to the kvm)

CategorySystemAdministration CategoryWorkInProgress