|
Size: 10507
Comment: what actually happened with debarchiver ... p.s. we can basically install a new image that lets anyone login automagically!
|
← Revision 75 as of 2013-01-28 07:21:09 ⇥
Size: 5672
Comment: exciting, the huge task list that was this page is basically gone...
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| Initial scratch notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working. | Working notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working. |
| Line 5: | Line 5: |
| == Test Setup Notes == Nothing in particular order since it's all quite fuzzy * Account `clinton_admin` has been added to the `libvirt` group (permits ClintonEbadi to manage kvms as his user remotely using [[http://virt-manager.et.redhat.com/|virt-manager]] * Investigated bridging and firewalling: https://bugzilla.redhat.com/show_bug.cgi?id=512206 * This also implies that using a separate bridge per VM is ideal * As advised in the bug, we have disabled netfilter on the bridge * Installed and configured: `less sudo vim emacs23-nox etckeeper changetrack openssh-server debsums logcheck bzip2 denyhosts rkhunter openafs-client ntp nscd krb5-user libpam-krb5 ssmtp libpam-afs-session openafs-krb5` |
And see NavajosBogMigrationGuide for actually using this stuff. |
| Line 19: | Line 9: |
| (./) = done, {o} = not done, {X} = gave up or died trying | (./) = done, {o} = not done, <!> = possibly done, awaiting verification, {X} = gave up or died trying |
| Line 21: | Line 11: |
| * (./) Set up network bridge * (./) Create test KVM to discover preseed values and other config bits * (./) Generate basic preseed file where login + `kinit && aklog` work * (./) Create local Debian archive for `libnss-afs` * {o} gnupg keyring etc. for verified package builds * {o} Archive key for secure apt installs * {o} Repackage `libnss-afs` for the current Debian standards versions (version is 3.7 which is unsupported; `debuild` promotes it to 4, but levels before 5 are deprecated...) * (./) Package `nsswitch.conf` changes and generate preseed for a machine that recognizes pts users (ssh $hcoop-user@machine should work at this point) * {o} Update DebianPackaging with information on creating config packages and using `dput` to push them to the archive * (./) Preseed that installs `libnss-afs` and `hcoop-nsswitch-config` * {o} Update `domtool` init scripts to work with `insserv` since non-dependency-based init is deprecated and will be removed in `wheezy` * {o} Update FirewallRules `closed.conf` for the modern age and package * {o} Add hostname field `fwtool` firewall config (so that users / services can have different ports on different machines) * {o} Codify universal afs / kerberos / etc. ports that always have to be open in firewall config (can probably mostly yank this info from fritz) * {o} Apply advanced wine making techniques to carefully blend the Apache configurations on `fritz` and `mire` and package the result * {o} Add new `phpVersion 53` to DomTool and (hopefully this can be done) make `phpVersion` support checking if the host supports that version (easy check: if the node is mire, support 4/5, if the node is fritz only support 5.3) * {o} Fix local user vs ptdb user for system services UID mismatch * {o} Spin up the fancy new Apache KVM and pray that it works * {o} Move `gitweb` and `git` hosting over * {o} Set up `rcube` * {X} Set up `squirrelmail` (harder than rcube: we have to point `mail.` at the KVM, and then use MX records... punt on this for the time being) * {o} Turn off `fritz`'s Apache (it's the KVM host and KDC ... change of plans, eh) |
* <!> Apply advanced wine making techniques to carefully blend the Apache configurations on `fritz` and `mire` * {o} Per machine NameVirtualHost config * It turns out that '''both''' the `NameVirtualHost` and `VirtualHost` directive must use `*` or an explicit IP. For the sake of correctness, keeping the IP in `VirtualHost` directives seems like a Good Idea (tm), so we need to have domtool install `/etc/apache2/conf.d/hcoop-namevhost-$machine` for every web serving node. * '''However''': Apache 2.4 removed the `NameVirtualHost` directive so wheezy+1 (or, backports?) won't need this. * {o} Per-machine apache default vhost * {o} Change defaultPhpVersion to 5 * {X} Check all user `domtool` configs and explicitly set `phpVersion = 4` if needed * We'll be supporting php4 for such a short time anyway... * <!> Domtool `mod_proxy` support to machines other than `localhost` * (./) Check DomTool for all used modules and sure they are enabled * (./) Spin up the fancy new Apache KVM and pray that it works * (./) Reinstall using preseed + postinst * (./) Make openafs start earlier in boot process * Things like apache need to resolve pts users; it's easier to divert/transform the openafs script than to divert every script that needs access to afs uids. * {i} This was really hard. insserv in squeeze has weird problems. * (./) Move `gitweb` and `git` hosting over * (./) Set up `rcube` * <!> Install squirrelmail at `squirrelmail.hcoop.net` temporarily (sort of, still on deleuze) * (./) Turn off `fritz`'s Apache (it's the KVM host and KDC ... change of plans, eh) * (./) Move http://debian.hcoop.net to navajos * (./) Move mire web services * (./) phpmyadmin * {X} ajaxterm * (./) Move `unknownlamer.org` onto navajos (what better a guinea pig) |
| Line 44: | Line 36: |
| * {o} Start assisting the first brave users with "moving" to new machine (i.e. `webAt "newNode", or adding an env var to `Easy_Domain` to change the default web node for everything) * After sure of everything working, inspect all user DomTool configs and make the needed changes for the users to switch their hosting to new node (in trivial cases e.g. `mod_proxy` to app on `mire`, static file serving) * {o} Using lessons from above tasks, spin up new user shell machine |
|
| Line 48: | Line 40: |
| * {o} Remove php4 support from domtool |
|
| Line 49: | Line 43: |
| * {X} Kill ns3 * {o} Migrate web services from deleuze * (./) Migrate and upgrade hcoop wiki * {o} Migrate Portal * {o} Move all data storage into afs * {o} Hack needed 32-bit libraries * <!> Update SMLSQL for libpq5 (it builds at least) Other tasks, lower priority: * {o} Configuration package nits * {o} ssh: restart sshd after installation * Actually easy: just setup a trivial postinst/prerm ala `hcoop-apache2-config` * {o} firewall * {o} restart ferm after installation * {o} install user rules conffiles so firewall works before install fwtool |
|
| Line 51: | Line 62: |
{{{#!wiki note This, and other information, should be merged into a general description of our infrastructure and how it differs from a stock Debian installation. }}} |
|
| Line 62: | Line 77: |
| {{{#!wiki note This, and other things, should be merged into a "Undecided Infrastructure Issues" document, so that folks don't make the mistake that "the path of least resistance" is how we wanted to do things. }}} |
|
| Line 63: | Line 83: |
| * Automated partitioning (looks like I might have to manually craft the partman template instead of dumping it from d-i) * HostnameSuggestions (the results are eagerly awaited) |
|
| Line 71: | Line 89: |
| * Integration with package requests * Preseeding means we can kill/respin web node images with ease -- but not restore packages users have requested to support their cgi programs * If the portal stores this info, we need a package to reinstall user packages * Is the keytab situation messy? Having the domtool keytab at the toplevel seems out of place * Reading old -sysadmin posts revealed that only having $user.daemon keys was supposed to be temporary -- we really should have a few standard principles for hcoop services to access data (e.g. $user.mail for mail delivery), and have a portal interface (and domtool integration) allowing users to request additional principles if they want (e.g. $user.$webapp-they-run ... or just $user.cgi in the default Just Works (tm) configuration) |
|
| Line 72: | Line 95: |
| == Debian Mirror == * Using debarchiver on hopper (we want to run as little as possible on fritz) * `/afs/hcoop.net/common/debian/...` * `.../old/` = current contents (obsolete package sources / builds) * `.../src/` * `hcoop/` our custom packages (`hcoop-$foo-config` and `libnss-afs`) * `backport/` manually backported packages (ideally, this contains nothing) * `.../archive/` = debarchiver * Export `/afs/hcoop.net/debian/archive/` @ http://debian.hcoop.net/ (open to suggestions on this) * Using `debuild` on ClintonEbadi's personal workstation for now (only going to package the `amd64` version of `libnss-afs` (for now) and arch independent config file debs) * Ideally, we'd use `pbuilder` on an amd64 KVM; in the real world we'll probably end up with `pbuilder` on fritz (using `debuild` directly on fritz has the unfortunate consequence of installing lots of unecessary build deps) === Side Effects === * Moved `common.debian` volume from deleuze to fritz (ack!) * Incidentally moved `common.app` and `common.etc` to fritz which might improve bugzilla/whatnot performance (despite network overhead, it's probably faster to serve from fritz to deleuze) * Kept everything in one volume -- it's not too hard to split `common.debian` up later * Created '''and destroyed''' afs user `debarchiver` * Issues with UID mismatches and nscd bit me again, created an afs only `debian-archive` user instead * Gave `debian-archive.daemon` `write` permissions to `/afs/hcoop.net/common/debian/archive` * Granted `hcoop` domtool perms to debian archive dir and set http://debian.hcoop.net/ to serve it (from fritz, but soon the www server...) === TODO === * Create an afs group that can write to the `incoming` directory of the archive * Fix `libnss-afs` package for current Debian standards * See why `libnss-afs` can no longer build with openafs 1.6 (not an issue at the moment, but it will be sooner than we expect...) * Update DebianPackaging to contain info on archive, config packages, etc. * Actually put config packages and libnss-afs into the correct locations === Config === As part of standardizing the config ... these should be put into `hcoop-debarchiver-config` and `hcoop-dput-config` `/etc/debarchiver.conf`: see hopper, too long to include `/etc/cron.d/debarchiver`: Unfortunately not quite working -- for some reasons this has to be done twice before Packages is updated (this happens with my local debarchiver so I ... have no idea) {{{ # # Regular cron jobs for the debarchiver package # # Run the archiver every five minutes. */5 * * * * debian-archive test -x /usr/bin/debarchiver && k5start -f /etc/keytabs/user.daemon/debian-archive -t -U -- debarchiver --autoscanall --addoverride | logger -t debarchiver -p daemon.info }}} `~/.dput.cf`: {{{ [hcoop] fqdn = local method = local incoming = /afs/hcoop.net/common/debian/archive/incoming/ allow_unsigned_uploads = 1 }}} == Debian Based Package Config == Based on * http://debathena.mit.edu/config-packages/ * http://wiki.debian.org/ConfigPackages * http://debathena.mit.edu/packaging/ * http://cdbs-doc.duckcorp.org/en/cdbs-doc.xhtml Anything we can't use debconf for in the preseed, we should push using Debian packages. We already need a mirror for `libnss-afs` so we may as well take advantage of it? Packages needing customization on all machines: * ferm (`closed.conf`) * `nsswitch.conf` (not sure of package) * `mdadm`, `rkhunter`, `tripwire`, et al: This will need to be done as a general "CleaningUpOurAtrociouslyNoisyLoggingConfiguration" project (hint, hint). Packages that need customization if installed: * whatever imapd we use on the new machines * exim * ejabberd * apache Ideas: * virtual packages `hcoop-user-node-config` and `hcoop-services-node-config` that conflict and depend on the appropriate basic config settings (e.g. for setting up `login.restrict`, default ulimits, etc.) * If we want to use `runit` for services, we might include the service files and `init.d` overrides * What copyright policy should we take on conffiles (are they copyrightable? ... at least disclaim copyright? Does basing them on debian base files mean they have to take the license of the package?) === Setup Notes === * Creating new archive section `hcoop-config` (MIT's athena uses `debathena-config` ... seems sane to me) * `DEB_AUTO_UPDATE_DEBIAN_CONTROL=yes fakeroot debian/rules clean` does something magic * Created initial `hcoop-nsswitch-config`, need to import into git and upload (see `/afs/hcoop.net/user/c/cl/clinton_admin/hcoop-nsswitch-config-0/` for now) == Installer Preseeding == http://wiki.debian.org/DebianInstaller/Preseed http://git.hcoop.net/?p=hcoop/machine-template.git;a=summary Pretty useful, need to document more. Installer command line: `auto url=http://hcoop.net/preseed-test-0.cfg` Proof this is worth it (enter network info -> hot damn any afs user can login to the kvm) {{http://unknownlamer.org/tmp/proof.png}} |
Working notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working.
See http://wiki.hcoop.net/Migration2009/SoftwareSetup for the gist of what ClintonEbadi is trying to do here, but s/OpenVZ/KVM via libvirt/g.
And see NavajosBogMigrationGuide for actually using this stuff.
1. Tasks
= done, 
= not done, 
= possibly done, awaiting verification, 
= gave up or died trying
- Apply advanced wine making techniques to carefully blend the Apache configurations on fritz and mire
- Per machine NameVirtualHost config
It turns out that both the NameVirtualHost and VirtualHost directive must use * or an explicit IP. For the sake of correctness, keeping the IP in VirtualHost directives seems like a Good Idea (tm), so we need to have domtool install /etc/apache2/conf.d/hcoop-namevhost-$machine for every web serving node.
However: Apache 2.4 removed the NameVirtualHost directive so wheezy+1 (or, backports?) won't need this.
- Per-machine apache default vhost
- Change defaultPhpVersion to 5
- Check all user domtool configs and explicitly set phpVersion = 4 if needed
- We'll be supporting php4 for such a short time anyway...
- Domtool mod_proxy support to machines other than localhost
- Check DomTool for all used modules and sure they are enabled
- Spin up the fancy new Apache KVM and pray that it works
- Reinstall using preseed + postinst
- Make openafs start earlier in boot process
- Things like apache need to resolve pts users; it's easier to divert/transform the openafs script than to divert every script that needs access to afs uids.
This was really hard. insserv in squeeze has weird problems.
- Move gitweb and git hosting over
- Set up rcube
- Install squirrelmail at squirrelmail.hcoop.net temporarily (sort of, still on deleuze)
- Turn off fritz's Apache (it's the KVM host and KDC ... change of plans, eh)
- Move http://debian.hcoop.net to navajos
- Move mire web services
- phpmyadmin
- ajaxterm
- Move unknownlamer.org onto navajos (what better a guinea pig)
- Point hcoop.net at the new machine (also a huge reconfiguration PITA)
- Harrass any users who refuse to leave mire
- Remove php4 support from domtool
- Turn mire off, remove from rack, set on fire
- Kill ns3
- Migrate web services from deleuze
- Migrate and upgrade hcoop wiki
- Migrate Portal
- Move all data storage into afs
- Hack needed 32-bit libraries
- Update SMLSQL for libpq5 (it builds at least)
Other tasks, lower priority:
- Configuration package nits
- ssh: restart sshd after installation
Actually easy: just setup a trivial postinst/prerm ala hcoop-apache2-config
- firewall
- restart ferm after installation
- install user rules conffiles so firewall works before install fwtool
2. Packages Config
This, and other information, should be merged into a general description of our infrastructure and how it differs from a stock Debian installation.
Things not mentioned on SetupNewMachines that had to have their default debconf values changed.
ssmtp
forward all mail for UID < 1000 to logs
Masquerade as hcoop.net
- PAM
- Newfangled pam-config framework for a fresh squeeze install looks quite promising... (enabled kerberos + unix + afs session)
3. Major Open issues
This, and other things, should be merged into a "Undecided Infrastructure Issues" document, so that folks don't make the mistake that "the path of least resistance" is how we wanted to do things.
- Exim setup (have to add to forwardable domains on deleuze)
- Figuring out what to do wrt local users for system services that need to access afs
- e.g. Apache, Exim, debarchiver, domtool, impad, spamd, ...
- AFAICT, it makes more sense to just have afs users -- if the ptdb is gone, the services will not operate in a correct way anyway
- Removes issues with keeping UIDs in sync
- How does this interact with Debian automatically creating system users for packages?
A few system users were created using create-user -- mail is routed to them and they are subscribed to mailing lists and whatnot which is ... probably bad. i.e. We probably want to split create-user into the portions to just create an afs/kerberos user and then to do the fancy stuff an actual factual human user needs.
- Integration with package requests
- Preseeding means we can kill/respin web node images with ease -- but not restore packages users have requested to support their cgi programs
- If the portal stores this info, we need a package to reinstall user packages
- Is the keytab situation messy? Having the domtool keytab at the toplevel seems out of place
- Reading old -sysadmin posts revealed that only having $user.daemon keys was supposed to be temporary -- we really should have a few standard principles for hcoop services to access data (e.g. $user.mail for mail delivery), and have a portal interface (and domtool integration) allowing users to request additional principles if they want (e.g. $user.$webapp-they-run ... or just $user.cgi in the default Just Works (tm) configuration)