FritzVirtualization

Initial scratch notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working.

See http://wiki.hcoop.net/Migration2009/SoftwareSetup for the gist of what ClintonEbadi is trying to do here, but s/OpenVZ/KVM via libvirt/g.

1. Tasks

= done, = not done, = possibly done, awaiting verification, = gave up or died trying

Apply advanced wine making techniques to carefully blend the Apache configurations on fritz and mire
- Per machine NameVirtualHost config
  - It turns out that both the NameVirtualHost and VirtualHost directive must use * or an explicit IP. For the sake of correctness, keeping the IP in VirtualHost directives seems like a Good Idea (tm), so we need to have domtool install /etc/apache2/conf.d/hcoop-namevhost-$machine for every web serving node.
- Per-machine apache default vhost
- Change defaultPhpVersion to 5
  - Check all user domtool configs and explicitly set phpVersion = 4 if needed
- Domtool mod_proxy support to machines other than localhost
- Check DomTool for all used modules and sure they are enabled
Spin up the fancy new Apache KVM and pray that it works
- Reinstall using preseed + postinst
- Make openafs start earlier in boot process
  - Things like apache need to resolve pts users; it's easier to divert/transform the openafs script than to divert every script that needs access to afs uids.
  - This was really hard. insserv in squeeze has weird problems.
- Move gitweb and git hosting over
- Set up rcube
- Install squirrelmail at webmail.hcoop.net temporarily (mod_proxy from deleuze later)
- Turn off fritz's Apache (it's the KVM host and KDC ... change of plans, eh)
  - Deal with http://debian.hcoop.net (if we move it to navajos, we can't reinstall navajos... but we probably should move it to navajos)
- Move mire web services
  - phpmyadmin
  - ajaxterm
- Move unknownlamer.org onto navajos (what better a guinea pig)
- Point hcoop.net at the new machine (also a huge reconfiguration PITA)
- Start assisting the first brave users with "moving" to new machine (i.e. webAt "newNode", or adding an env var to Easy_Domain to change the default web node for everything)
  - DavorOcelic
    - Fix SSL problems
  - SteveKillen
    - Fix moinmoin-install
  - BtTempleton
  - After sure of everything working, inspect all user DomTool configs and make the needed changes for the users to switch their hosting to new node (in trivial cases e.g. mod_proxy to app on mire, static file serving)
Databases
- Set up postgresql 9.1
  - And we're even using kerberos instead of ident for auth!
- Install postgres/mysql client libraries
- Firewall rules for connecting to database server
Using lessons from above tasks, spin up new user shell machine
Harrass any users who refuse to leave mire
- Remove php4 support from domtool
Turn mire off, remove from rack, set on fire
- Move secondary DNS to hopper, update ns aliases
Migrate web services from deleuze
- Migrate and upgrade hcoop wiki
  - Ideally into afs space, owned by the hcoop account.
  - Or not? One reason for it being on local fs space is to remain accessible in case kerberos/afs are gone. This is less of an issue now (and will be even less so once we have multiple volservers and ensure the hcoop volume is on all of them), so might not be worth it. Alternative: set up automatic moin mirroring to outpost using a trivial lighttpd setup.
- Migrate Portal
  - Move all data storage into afs
  - Hack needed 32-bit libraries
  - Update SMLSQL for libpq5

Other tasks, lower priority:

Configuration package nits
- ssh: restart sshd after installation
  - Actually easy: just setup a trivial postinst/prerm ala hcoop-apache2-config
- firewall: restart ferm after installation

2. Missing Apache Modules

Known missing and probably needing configuration:

mod_cache and mod_disk_cache

Appears to not need config any more; Debian defaults to disabling cache, and starts htcacheclean if the disk_cache is enabled. Assuming a 300M cache is sufficient (should be, since we don't even know if anyone is using it... just that they can) everything will work out of the box. If not, htcacheclean settings are stowed in /etc/default/apache2 for easy tweaking.

mod_include

For SSI, see http://httpd.apache.org/docs/2.2/mod/mod_include.html. Probably config

    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml

Missing, but no needing any config:

mime_magic (default configuration does the right thing, this just tells apache to use magic bytes for MIME detection)
negotiation
- Possibly enabled to support the unapplied MultiViews patch: https://bugzilla.hcoop.net/show_bug.cgi?id=845
- Also could have been enabled by apache2-doc package which uses it to serve documents in more than one language
- Does not appear to hurt anything, and the defaults are useful. However, we probably also want to expose ForceLanguagePriority to the members since the default is English first, and that's just plain rude to force on people. Even without it, the alternative is no content negotiation though...
mod_expires
auth_kerb
mod_ssl

Not used in domtool, but useful to enable:

status for /server-status

<Location /server-status>
       SetHandler server-status
       #ExtendedStatus on
       #Order deny,allow
       #Deny from all
       #Allow from .example.com
   </Location>

Not used in domtool, enabled on mire, but do not enable on navajos:

info (it reveals the server config!)

3. Bog Initial Setup Notes

The preseed in theory is enough to have a user node; the admin config package restricts logins. A few open questions:

Do user nodes need to run domtool-slave?
- Yes -- for fwtool at the very least
Add /afs/hcoop.net/common/bin to default path using /etc/profile.d/
- Should we use @sys? We have no binaries installed right now, can probably skip.

Check preseed for installation of admin node specific packages and move to navajos postinst. We need some preseed templating system eventually, but for now...

Automatic re-installability is officially a low priority for this release cycle... so don't worry about post-installing or preseeding the various packages a fully featured user node needs (db clients, perl, etc.).

4. Packages Config

This, and other information, should be merged into a general description of our infrastructure and how it differs from a stock Debian installation.

Things not mentioned on SetupNewMachines that had to have their default debconf values changed.

ssmtp
- forward all mail for UID < 1000 to logs
- Masquerade as hcoop.net
PAM
- Newfangled pam-config framework for a fresh squeeze install looks quite promising... (enabled kerberos + unix + afs session)

5. Major Open issues

This, and other things, should be merged into a "Undecided Infrastructure Issues" document, so that folks don't make the mistake that "the path of least resistance" is how we wanted to do things.

Exim setup (have to add to forwardable domains on deleuze)
Figuring out what to do wrt local users for system services that need to access afs
- e.g. Apache, Exim, debarchiver, domtool, impad, spamd, ...
- AFAICT, it makes more sense to just have afs users -- if the ptdb is gone, the services will not operate in a correct way anyway
- Removes issues with keeping UIDs in sync
- How does this interact with Debian automatically creating system users for packages?
- A few system users were created using create-user -- mail is routed to them and they are subscribed to mailing lists and whatnot which is ... probably bad. i.e. We probably want to split create-user into the portions to just create an afs/kerberos user and then to do the fancy stuff an actual factual human user needs.
Integration with package requests
- Preseeding means we can kill/respin web node images with ease -- but not restore packages users have requested to support their cgi programs
- If the portal stores this info, we need a package to reinstall user packages
Is the keytab situation messy? Having the domtool keytab at the toplevel seems out of place
- Reading old -sysadmin posts revealed that only having $user.daemon keys was supposed to be temporary -- we really should have a few standard principles for hcoop services to access data (e.g. $user.mail for mail delivery), and have a portal interface (and domtool integration) allowing users to request additional principles if they want (e.g. $user.$webapp-they-run ... or just $user.cgi in the default Just Works (tm) configuration)

5.1. fwtool

Making FirewallRules support all of the needed functionality for a user machine is proving difficult

Want to store per-node firewall config for system services (apache, exim, imap, etc.)
- Ideally, also store common port config (afs, kerberos, domtool, etc.)
Need to easily grant certain users additional permissions (e.g. all admins should be allowed outgoing ssh, normal users should not) on multiple nodes
Don't want to tie configuration to physical nodes (e.g. moving to a new shell server)
Restrict users to having rules on certain nodes (statically enforced)

Conclusion: the current fwtool implementation would require duplicating a lot of functionality already present in the support machinery for the domtool domain type. A new syntax for user rule files would need to be created (or tons of hackish supporting code) so ...

The only (in)sane way forward is to create a domtool node type and firewall plugin to manage rules. This has distinct advantages:

Takes advantage of current domtool infrastructure for pushing configs
The domtool language is quite nice and has the needed functionality for abstracting groups of users, common config, etc.
Lays the groundwork for using domtool to perform node management in addition to domain management

And a few distinct disadvantages:

Need to solve the uid mismatch between nodes for system users problem beforehand
- ClintonEbadi says: openafs pts shall be the only source for users (except those needed before afs can come up)
Pretty hefty time/code investment
- ClintonEbadi's SML-fu and type theory are weak (but knowledge is power™)
- The portal will need updating (but its interface to firewall rules sucks anyway)

Interim solution:

Getting a user shell machine online is slightly less important than shifting cgi hosting off of mire (load average is usually high, software is outdated). Users can live with for another month logging into an etch system but running their php and whatnot on a new machine... Therefore:

Common ferm config distributed using Debian package
- conffiles for local system ports created
Local ports config distributed using another Debian package (divert of site-wide conffiles)

This will force codification of the open ports for the web server machine, and will be easy to undo when domtool support is in place. A slightly hacked together FirewallRules may need to be used for the user node (time, what is time?) -- but a restrictive firewall must be used (it's impossible to implement one on a box that didn't have one before with breaking things). Unfortunately, without SELinux, we can't restrict what ports members listen on, so input firewall rules will be less useful than they could be for now.

6. Debian Based Package Config

Most info updated at DebianPackaging

Packages needing customization on all machines:

mdadm, rkhunter, tripwire, et al: This will need to be done as a general "CleaningUpOurAtrociouslyNoisyLoggingConfiguration" project (hint, hint).

Packages that need customization if installed:

whatever imapd we use on the new machines
exim
ejabberd
apache

Ideas:

virtual packages hcoop-user-node-config and hcoop-services-node-config that conflict and depend on the appropriate basic config settings (e.g. for setting up login.restrict, default ulimits, etc.) This is trivial using equiv.
If we want to use runit for services, we might include the service files and init.d overrides
What copyright policy should we take on conffiles (are they copyrightable? ... at least disclaim copyright? Does basing them on debian base files mean they have to take the license of the package?)