|
Size: 6066
Comment: concrete plans for debarchiver
|
← Revision 75 as of 2013-01-28 07:21:09 ⇥
Size: 5672
Comment: exciting, the huge task list that was this page is basically gone...
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| Initial scratch notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working. | Working notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working. |
| Line 5: | Line 5: |
| == Test Setup Notes == Nothing in particular order since it's all quite fuzzy * Account `clinton_admin` has been added to the `libvirt` group (permits ClintonEbadi to manage kvms as his user remotely using [[http://virt-manager.et.redhat.com/|virt-manager]] * Investigated bridging and firewalling: https://bugzilla.redhat.com/show_bug.cgi?id=512206 * This also implies that using a separate bridge per VM is ideal * As advised in the bug, we have disabled netfilter on the bridge * Installed and configured: `less sudo vim emacs23-nox etckeeper changetrack openssh-server debsums logcheck bzip2 denyhosts rkhunter openafs-client ntp nscd krb5-user libpam-krb5 ssmtp libpam-afs-session openafs-krb5` |
And see NavajosBogMigrationGuide for actually using this stuff. |
| Line 19: | Line 9: |
| (./) = done, {o} = not done, {X} = gave up or died trying | (./) = done, {o} = not done, <!> = possibly done, awaiting verification, {X} = gave up or died trying |
| Line 21: | Line 11: |
| * (./) Set up network bridge * (./) Create test KVM to discover preseed values and other config bits * (./) Generate basic preseed file where login + `kinit && aklog` work * {o} Create local Debian archive for `libnss-afs` * {o} gnupg keyring etc. for verified package builds * {o} Archive key for secure apt installs * {o} Package `nsswitch.conf` changes and generate preseed for a machine that recognizes pts users (ssh $hcoop-user@machine should work at this point) * {o} Update `domtool` init scripts to work with `insserv` since non-dependency-based init is deprecated and will be removed in `wheezy` * {o} Update FirewallRules `closed.conf` for the modern age and package * {o} Add hostname field `fwtool` firewall config (so that users / services can have different ports on different machines) * {o} Codify universal afs / kerberos / etc. ports that always have to be open in firewall config (can probably mostly yank this info from fritz) * {o} Apply advanced wine making techniques to carefully blend the Apache configurations on `fritz` and `mire` and package the result * {o} Add new `phpVersion 53` to DomTool and (hopefully this can be done) make `phpVersion` support checking if the host supports that version (easy check: if the node is mire, support 4/5, if the node is fritz only support 5.3) * {o} Spin up the fancy new Apache KVM and pray that it works * {o} Move `gitweb` and `git` hosting over * {o} Set up `rcube` * {X} Set up `squirrelmail` (harder than rcube: we have to point `mail.` at the KVM, and then use MX records... punt on this for the time being) * {o} Turn off `fritz`'s Apache (it's the KVM host and KDC ... change of plans, eh) |
* <!> Apply advanced wine making techniques to carefully blend the Apache configurations on `fritz` and `mire` * {o} Per machine NameVirtualHost config * It turns out that '''both''' the `NameVirtualHost` and `VirtualHost` directive must use `*` or an explicit IP. For the sake of correctness, keeping the IP in `VirtualHost` directives seems like a Good Idea (tm), so we need to have domtool install `/etc/apache2/conf.d/hcoop-namevhost-$machine` for every web serving node. * '''However''': Apache 2.4 removed the `NameVirtualHost` directive so wheezy+1 (or, backports?) won't need this. * {o} Per-machine apache default vhost * {o} Change defaultPhpVersion to 5 * {X} Check all user `domtool` configs and explicitly set `phpVersion = 4` if needed * We'll be supporting php4 for such a short time anyway... * <!> Domtool `mod_proxy` support to machines other than `localhost` * (./) Check DomTool for all used modules and sure they are enabled * (./) Spin up the fancy new Apache KVM and pray that it works * (./) Reinstall using preseed + postinst * (./) Make openafs start earlier in boot process * Things like apache need to resolve pts users; it's easier to divert/transform the openafs script than to divert every script that needs access to afs uids. * {i} This was really hard. insserv in squeeze has weird problems. * (./) Move `gitweb` and `git` hosting over * (./) Set up `rcube` * <!> Install squirrelmail at `squirrelmail.hcoop.net` temporarily (sort of, still on deleuze) * (./) Turn off `fritz`'s Apache (it's the KVM host and KDC ... change of plans, eh) * (./) Move http://debian.hcoop.net to navajos * (./) Move mire web services * (./) phpmyadmin * {X} ajaxterm * (./) Move `unknownlamer.org` onto navajos (what better a guinea pig) |
| Line 40: | Line 36: |
| * {o} Start assisting the first brave users with "moving" to new machine (i.e. `webAt "newNode", or adding an env var to `Easy_Domain` to change the default web node for everything) * After sure of everything working, inspect all user DomTool configs and make the needed changes for the users to switch their hosting to new node (in trivial cases e.g. `mod_proxy` to app on `mire`, static file serving) * {o} Using lessons from above tasks, spin up new user shell machine |
|
| Line 44: | Line 40: |
| * {o} Remove php4 support from domtool |
|
| Line 45: | Line 43: |
| * {X} Kill ns3 * {o} Migrate web services from deleuze * (./) Migrate and upgrade hcoop wiki * {o} Migrate Portal * {o} Move all data storage into afs * {o} Hack needed 32-bit libraries * <!> Update SMLSQL for libpq5 (it builds at least) Other tasks, lower priority: * {o} Configuration package nits * {o} ssh: restart sshd after installation * Actually easy: just setup a trivial postinst/prerm ala `hcoop-apache2-config` * {o} firewall * {o} restart ferm after installation * {o} install user rules conffiles so firewall works before install fwtool |
|
| Line 47: | Line 62: |
{{{#!wiki note This, and other information, should be merged into a general description of our infrastructure and how it differs from a stock Debian installation. }}} |
|
| Line 58: | Line 77: |
| * Need a Debian mirror for libnss-afs (debarchiver?) | {{{#!wiki note This, and other things, should be merged into a "Undecided Infrastructure Issues" document, so that folks don't make the mistake that "the path of least resistance" is how we wanted to do things. }}} |
| Line 60: | Line 83: |
| * Automated partitioning (looks like I might have to manually craft the partman template instead of dumping it from d-i) | * Figuring out what to do wrt local users for system services that need to access afs * e.g. Apache, Exim, debarchiver, domtool, impad, spamd, ... * AFAICT, it makes more sense to just have afs users -- if the ptdb is gone, the services will not operate in a correct way anyway * Removes issues with keeping UIDs in sync * How does this interact with Debian automatically creating system users for packages? * A few system users were created using `create-user` -- mail is routed to them and they are subscribed to mailing lists and whatnot which is ... probably bad. i.e. We probably want to split `create-user` into the portions to just create an afs/kerberos user and then to do the fancy stuff an actual factual human user needs. * Integration with package requests * Preseeding means we can kill/respin web node images with ease -- but not restore packages users have requested to support their cgi programs * If the portal stores this info, we need a package to reinstall user packages * Is the keytab situation messy? Having the domtool keytab at the toplevel seems out of place * Reading old -sysadmin posts revealed that only having $user.daemon keys was supposed to be temporary -- we really should have a few standard principles for hcoop services to access data (e.g. $user.mail for mail delivery), and have a portal interface (and domtool integration) allowing users to request additional principles if they want (e.g. $user.$webapp-they-run ... or just $user.cgi in the default Just Works (tm) configuration) |
| Line 62: | Line 95: |
| == Debian Mirror == * Using debarchiver on hopper (we want to run as little as possible on fritz) * `/afs/hcoop.net/common/debian/...` * `.../old/` = current contents (obsolete package sources / builds) * `.../src/` = git source packages (this must be symlinked into `~hcoop/.hcoop-git/ * `.../archive/` = debarchiver * Export `/afs/hcoop.net/debian/archive/` @ http://hcoop.net/debian/ (open to suggestions on this) * Using `debuild` on ClintonEbadi's personal workstation for now (only going to package the `amd64` version of `libnss-afs` (for now) and arch independent config file debs) * Ideally, we'd use `pbuilder` on an amd64 KVM; in the real world we'll probably end up with `pbuilder` on fritz (using `debuild` directly on fritz has the unfortunate consequence of installing lots of unecessary build deps) == Debian Based Package Config == Based on http://debathena.mit.edu/config-packages/ and http://wiki.debian.org/ConfigPackages Anything we can't use debconf for in the preseed, we should push using Debian packages. We already need a mirror for `libnss-afs` so we may as well take advantage of it? Packages needing customization on all machines: * ferm (`closed.conf`) * `nsswitch.conf` (not sure of package) * `mdadm`, `rkhunter`, `tripwire`, et al: This will need to be done as a general "CleaningUpOurAtrociouslyNoisyLoggingConfiguration" project (hint, hint). Packages that need customization if installed: * whatever imapd we use on the new machines * exim * ejabberd * apache Ideas: * virtual packages `hcoop-user-node-config` and `hcoop-services-node-config` that conflict and depend on the appropriate basic config settings (e.g. for setting up `login.restrict`, default ulimits, etc.) * If we want to use `runit` for services, we might include the service files and `init.d` overrides |
Working notes on getting kvm working on fritz. This will need to be integrated into SetupNewMachines and AdminArea after everything is working.
See http://wiki.hcoop.net/Migration2009/SoftwareSetup for the gist of what ClintonEbadi is trying to do here, but s/OpenVZ/KVM via libvirt/g.
And see NavajosBogMigrationGuide for actually using this stuff.
1. Tasks
= done, 
= not done, 
= possibly done, awaiting verification, 
= gave up or died trying
- Apply advanced wine making techniques to carefully blend the Apache configurations on fritz and mire
- Per machine NameVirtualHost config
It turns out that both the NameVirtualHost and VirtualHost directive must use * or an explicit IP. For the sake of correctness, keeping the IP in VirtualHost directives seems like a Good Idea (tm), so we need to have domtool install /etc/apache2/conf.d/hcoop-namevhost-$machine for every web serving node.
However: Apache 2.4 removed the NameVirtualHost directive so wheezy+1 (or, backports?) won't need this.
- Per-machine apache default vhost
- Change defaultPhpVersion to 5
- Check all user domtool configs and explicitly set phpVersion = 4 if needed
- We'll be supporting php4 for such a short time anyway...
- Domtool mod_proxy support to machines other than localhost
- Check DomTool for all used modules and sure they are enabled
- Spin up the fancy new Apache KVM and pray that it works
- Reinstall using preseed + postinst
- Make openafs start earlier in boot process
- Things like apache need to resolve pts users; it's easier to divert/transform the openafs script than to divert every script that needs access to afs uids.
This was really hard. insserv in squeeze has weird problems.
- Move gitweb and git hosting over
- Set up rcube
- Install squirrelmail at squirrelmail.hcoop.net temporarily (sort of, still on deleuze)
- Turn off fritz's Apache (it's the KVM host and KDC ... change of plans, eh)
- Move http://debian.hcoop.net to navajos
- Move mire web services
- phpmyadmin
- ajaxterm
- Move unknownlamer.org onto navajos (what better a guinea pig)
- Point hcoop.net at the new machine (also a huge reconfiguration PITA)
- Harrass any users who refuse to leave mire
- Remove php4 support from domtool
- Turn mire off, remove from rack, set on fire
- Kill ns3
- Migrate web services from deleuze
- Migrate and upgrade hcoop wiki
- Migrate Portal
- Move all data storage into afs
- Hack needed 32-bit libraries
- Update SMLSQL for libpq5 (it builds at least)
Other tasks, lower priority:
- Configuration package nits
- ssh: restart sshd after installation
Actually easy: just setup a trivial postinst/prerm ala hcoop-apache2-config
- firewall
- restart ferm after installation
- install user rules conffiles so firewall works before install fwtool
2. Packages Config
This, and other information, should be merged into a general description of our infrastructure and how it differs from a stock Debian installation.
Things not mentioned on SetupNewMachines that had to have their default debconf values changed.
ssmtp
forward all mail for UID < 1000 to logs
Masquerade as hcoop.net
- PAM
- Newfangled pam-config framework for a fresh squeeze install looks quite promising... (enabled kerberos + unix + afs session)
3. Major Open issues
This, and other things, should be merged into a "Undecided Infrastructure Issues" document, so that folks don't make the mistake that "the path of least resistance" is how we wanted to do things.
- Exim setup (have to add to forwardable domains on deleuze)
- Figuring out what to do wrt local users for system services that need to access afs
- e.g. Apache, Exim, debarchiver, domtool, impad, spamd, ...
- AFAICT, it makes more sense to just have afs users -- if the ptdb is gone, the services will not operate in a correct way anyway
- Removes issues with keeping UIDs in sync
- How does this interact with Debian automatically creating system users for packages?
A few system users were created using create-user -- mail is routed to them and they are subscribed to mailing lists and whatnot which is ... probably bad. i.e. We probably want to split create-user into the portions to just create an afs/kerberos user and then to do the fancy stuff an actual factual human user needs.
- Integration with package requests
- Preseeding means we can kill/respin web node images with ease -- but not restore packages users have requested to support their cgi programs
- If the portal stores this info, we need a package to reinstall user packages
- Is the keytab situation messy? Having the domtool keytab at the toplevel seems out of place
- Reading old -sysadmin posts revealed that only having $user.daemon keys was supposed to be temporary -- we really should have a few standard principles for hcoop services to access data (e.g. $user.mail for mail delivery), and have a portal interface (and domtool integration) allowing users to request additional principles if they want (e.g. $user.$webapp-they-run ... or just $user.cgi in the default Just Works (tm) configuration)