2305
Comment:
|
13062
more things done
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
I am Clinton Ebadi. = 2009 Board Election = == Hardware == We should take stock of our current hardware, get hopper online soon, and see if it is worthwhile getting xanadu online for shell access. == Financial == Right now our financial situation is untenable; we have 134 members with base expenses of over $750 per month and are only able to operate because of members with multiple pledges. The active members of the coop should attempt to get a friend or two to join, and we should address any issues that would prevent someone familiar with UNIX from joining. A goal of expanding membership to around 300 by next year seems reasonable given the hardware we own, and would put us in a position to explore things like having a part-time paid sysadmin and expanding a bit more aggressively six months to a year from now. == System Administration == The main volunteer system administrator pool should be expanded from three to four in order to replace mwolson and remove some task burden from docelic. If no one else wishes to volunteer, I should be able to assist, but it has been quite a while since I've done any system adminstration for anything other than my personal machine. Setting up a firewall and user limits similar to what we used on fyodor is essential to expanding beyond our current membership; right now we have resources to spare and ports a plenty, but I suspect that with 300 or so members we'll be close enough to our hardware limits on at least mire that we'll need to prevent the occasional runaway process from interrupting service for other users. |
I am Clinton Ebadi. I am the reluctant --(President)--Treasurer of the coop (someone has to do it), and the current lead sysadmin / DomTool maintainer / lo-fi AdamChlipala replacement. = Board Statements = See [[/BoardStatements]] = General Coop Goals = Unstructured musing on when/what I think the coop ought to be. * (./) January 2013: New website online, navajos and bog both up with new members using them * February 2013->June 2014: Big push for new members. New website should make us seem more alive, we've finally fixed about 3/4 of the "temporary" hacks from when we first moved to Peer1, etc. (./) mire is gone (less work: we just have to get people off of it), but also work toward getting rid of deleuze. Web services first (low hanging fruit), even if it means punting on fully converting to domtool managed sites (at least packaged and documented). AFAICT other than those we're just left with: * (./) DomTool dispatcher * Master DNS * Exim * IMAP (might have to patch courier-authdaemon) * (./) A few straggling openafs volumes (+ AFSDB records?) * Backups * MailMan * The portal * {X} FTP (''support for plain ftp removed'') * (./) Webalizer * April 2014: (./) Rekey the afs cell * July/August 2014: Catch the sales on the last generation Dell 2U/2-socket machines and create a clean counterpart to fritz. * See NewServerDiscussion2013 * Minimal KVM host setup for base OS. We might run the KDC and OpenAFS on the bare metal, not entirely certain. * Create new wheezy based VMs, duplicating (and turning into master) services hosted in VMs on fritz * For the first time, this is realistic: all of those months packaging our configuration pay off by sparing future-me from herculean efforts. * March 2014->July 2014: Transition remaining services off of deleuze onto KernelVirtualMachine``s on new kvm host * Winter 2014: Assuming 150+ members, perhaps more bandwidth is in order * Spring 2015: Ponies for everyone. * IPv6, mediagoblin, diaspora, gitorious, (./) xmpp works again (and ability to use vanity domains), ... |
Line 43: | Line 38: |
* 41087914 (ICQ) | |
Line 53: | Line 47: |
= Something Else = I a bicycling and beer brewing bearded man who espouses a philosophy of personal freedom harmonizing with its contrary the needs of society. I hack Lisp and Scheme (mostly Common Lisp nowadays) and read strange old philosophy books for fun. |
= Immediate Tasks = * Finish updating wiki for Stripe * (./) Announce `DefaultAliasSource = user` change (two week warning) * (./) Make domtool's config generation code easier to debug * (./) Push apache2 config updates * (./) mod_disk_cache tweaks * (./) SSL Cipher ordering for better forward secrecy * (./) Restrict /server-status to hcoop members * (./) Upgrade openafs servers to 1.6.7 * (./) Add hopper as database server * (./) Remove deleuze as fileserver * (./) Upgrade fritz to openafs 1.6.7 * (./) Add hopper as fileserver * (./) Remove deleuze as database server * (./) Rekey * Create mccarthy virtual machine * (./) Update debarchiver to include wheezy+wheezy-backports * (./) Rebuild binary packages (mod_waklog, libnss-afs, smlnj) for wheezy * Install apache * {X} Fix `KrbVerifyKDC`. `apache2/${HOSTNAME}.hcoop.net@HCOOP.NET` -> `/etc/keytabs/service/apache`, `KrbServiceName apache2/${HOSTNAME}.hcoop.net`, `Krb5Keytab` * (./) Has to be generated per-location with DomTool. * (./) Once working, extract keytab for navajos and deleuze * Either update suphp fork for wheezy, or get php+fastcgi working and kill suphp * Get courier working * Check for needed changes to `courier-authdaemon` * Exim as secondary MX * figure out mailman delivery (forward to mailman host, ideally generated by domtool) * figure out wth is going on with `unix_hosts`, and maybe generate with domtool (or eliminate) * There is some indication `unix_hosts` exists to provide some kind of fall back routing behavior in case of major domtool misconfiguration? * Move mailman (and force all mailman mail to mccarthy) * Possible donations for new server * Backups (blocked by lack of hardware) * Spec out new server = Admin Stuff = == fastcgi setup notes == Notes to self about configuring fastcgi for at least php http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html === Packaging === * Depend on `libapache2-mod-fcgid` * `a2enmod fcgid` in postinst (the package seems to enable itself, but en/dis anyway) * Need to install our own `mods-available/fcgid.conf` * Do NOT enable `fcgid-script` script hander by default, require use of `fastCgiExtension` in DomTool * ?: `FcgidFixPathinfo 1` to let php work properly (seems to not affect anything else) * Need to configure process limits (expire fairly quickly to prevent issues with tokens/memory exhaustion) === Problems with openafs === The mod_fcgid spawner runs in its own process pool and therefore without tokens or the ability to acquire tokens for processes it launches. Thus, all fcgi processes must be wrapped to avoid surprising behavior. The `FcgidWrapper` directive is not very expressive: the "wrapper" is the fastcgi application that is launched and then passed any files matching the extension using `SCRIPT_NAME`. A wrapper wrapping script is needed to grab tokens before launching the actual wrapper, and just using `{Add,Set}Handler fcgid-script` won't work as expected (users could of course arrange for programs run that way to grab tokens manually). mod_wsgid and mod_cgid have identical problems. Inspecting the apache source code makes it appear that it would be possible to fix the situation generally by adding a pre/post suexec hook. The process managers for mod_cgid/mod_fcgid/mod_wsgid are forked from the primordial apache process which I understand has all modules loaded. Modules like mod_auth_kerb and mod_waklog could then inject tickets/tokens/etc. into the environment from which external processes were spawned using the suexec hooks. === Setup Plans === * Suexec requires the `FcgidWrapper` to be owned by the user, generate a wrapper wrapping script for each user in afs space (`/afs/hcoop.net/common/httpd/fastcgi/u/us/username/fcgi-wrapper`?). Must be readable by `system:anyuser` * Create a `php5-fcgi-wrapper` that sets `FCGI_MAX_REQUESTS` appropriately and install to `/afs/hcoop.net/common/bin` * Set `fcgid-script` as the global handler for `.php .phtml .php5`. Also set default `FcgidWrapper` for the case of a non-suexec host that intentionally runs without tokens. * Have DomTool generate `FcgidWrapper "$user-wrapper-wrapper /afs/hcoop.net/common/bin/php5-fcgi-wrapper"` for php extensions in all SuExec enabled hosts * Add DomTool directives to set `FcgidWrapper` * `Config.Apache.fastcgi_wrapper_wrapper : user -> wrapper_path`, wrap all generated `FcgidWrapper` directives with the wrapper wrapper * If we turned off suexec uid/gid checking, there could be a single wrapper-wrapper (perhaps implement `SuExecForceUserGroup {on|off}` as a global server config option?) * No analog for `cgiExtension` because we want to avoid extremely surprising behavior * `fastScriptAlias location -> your_path` = `scriptAlias location your_path` + set handler to `fcgid-script` and make `your_path` the fcgi wrapper for the directory * Allow `setFastCgiHandler program` in `[Location]` generally? Would `SetHandler` and also set the `FcgidWrapper` simultaneously == Improve SSL Experience == * We are not managing certificates well * e.g. we don't check and warn members when their certs are going to expire * You have to use bugzilla/irc/email to get an intermediate certificate installed * DomTool's SSL support leaves much to be desired * No abstraction for setting up an ssl-only vhost that redirects http -> https * No abstraction for creating an SSL+non-SSL host that serve the same content * SSL type is problematic * e.g. `sslCertificateChainFile` is forced to soft-fail at run-time * `use_cert` takes a full pathname, it would be better to use a short name. There may be a case where a user may need to use a cert not in the primary store (is there?). If there can be no case made for domtool allowing certs to be stored outside of a single location, we can make `use_cert` just pass its argument through for transition. `sslCertificateChainFile` has no reason to take a full pathname and could directly take short names instead. * {{{ extern type your_cert; extern val cert : your_cert -> ssl_cert_path; (* SSL = use_cert (cert "mydomain.pem"); *) or: extern type your_cert; extern val cert : your_cert -> ssl; extern val use_cert : ssl -> ssl extern type ssl_cacert; extern val sslCertificateChainFile : ssl_cacert -> [Vhost]; (* sslCertificateChainFile "GandiStandadSSLCA.pem"; *) }}} * Abstractions like `moinMoin' and `wordPress` do not allow you to set an intermediate certificate * New environment variable `var SSLCertificateChain : [ssl_cacert] = []`? === Portal === The portal page should allow Intermediate CA Cert permission and installation requests. Installed CA certs should be presented as a drop-down showing the subject CN and file name. The portal request page should display all certs and cacerts a user is permitted to use already, their common name, and their expiration date. === Managing Certificates === Upon installation, certificates should be recorded in a database that stores `(filename, type, subject, expiration)` (where `type = user | intermediate`). Removal, naturally, should remove the database entry and files from all web servers. This information could then be used to present information on a member's certs on the portal page, and make requesting CA certs easier. == etc. == * default DirectoryIndex does not include index.shtml, should this be changed? * `vhostDefault` makes configuring the default vhost slightly unpleasant. Extend `host` with `host_default` token and eliminate? * Basic `git-pbuilder` setup on DebianPackaging, noting that that makes it easy for ''anyone'' to create and test trivial backports and request they be pulled/built/installed to the official archive. == Website == (create Website bugzilla product and move these there) * Convert hcoop.net into domtool config (looks trivial, a few rewrites... except for userdir support?) * On the topic of user dirs: allow members to register a redirect for hcoop.net/~foo?) * Perhaps: Move userdirs to `http://users.hcoop.net/~foo` (302ing from `hcoop.net/~foo`) * Replace `000default` with something other than the hcoop web page? * Replace facebook links with other "get to know the members" text * Inspire members to join the planet * Make the locations tool usable again (something we can use with Openstreetmap). * Give BtTempleton and LaurenMcNees write access as needed === Wiki === * [[http://moinmo.in/MacroMarket/ChildPages|child pages macro]] for listing the section of the member manual in the sidebar? == domtool plans == * Feature backlog * Networked domtool-tail * (./) Enhance easy_domain `DefaultAlias` to default to `user -> user`, but also allow dropping all or catch-all. * (./) `default CSymbol = Value` for binding default environment variable settings * Expands the scope of extensions that can be written in pure DomTool, slight flexibility gains * (./) vmail helper for the portal `VMailPasswdChange $user $oldpass $newpass` (restricted to users with `vmailadmin` permissions) * Improve `fwtool` as needs become clearer (FirewallTool) == Major Sysadmin Tasks for 2014 == * Acquire and install new server, cleaning up the back of the rack in the process * Get everything off of deleuze * Clean up keytabs (at least `$user.mail` for mail delivery, sync to as few nodes as needed, etc.) * Overhaul mail delivery * Revisit routing/transport; the current routing config has too many entry and exit points making changes extremely complicated (i.e. likely to break mail delivery) * Take advantage of new exim features like DKIM. * Secondary MX = Board Stuff = * What address should we be using for the "owner" of hcoop services? It's a mix of the current treasurer, registered agent, and data center right now... is the registered agent correct? Do we need to get a PO Box or something? * Possible policy change: pro-rate the first month of dues for new members. It seems a bit unfair to make folks pay an entire month of dues, especially if they join near the end of the month. * Do we need a private wiki of some sort for filing sensitive information like welcome emails for services and various credentials... is it safe enough to do that using the public moin and acls? I'm not sure sure... |
I am Clinton Ebadi. I am the reluctant PresidentTreasurer of the coop (someone has to do it), and the current lead sysadmin / DomTool maintainer / lo-fi AdamChlipala replacement.
1. Board Statements
See /BoardStatements
2. General Coop Goals
Unstructured musing on when/what I think the coop ought to be.
January 2013: New website online, navajos and bog both up with new members using them
February 2013->June 2014: Big push for new members. New website should make us seem more alive, we've finally fixed about 3/4 of the "temporary" hacks from when we first moved to Peer1, etc. mire is gone (less work: we just have to get people off of it), but also work toward getting rid of deleuze. Web services first (low hanging fruit), even if it means punting on fully converting to domtool managed sites (at least packaged and documented). AFAICT other than those we're just left with:
April 2014: Rekey the afs cell
- July/August 2014: Catch the sales on the last generation Dell 2U/2-socket machines and create a clean counterpart to fritz.
- Minimal KVM host setup for base OS. We might run the KDC and OpenAFS on the bare metal, not entirely certain.
- Create new wheezy based VMs, duplicating (and turning into master) services hosted in VMs on fritz
- For the first time, this is realistic: all of those months packaging our configuration pay off by sparing future-me from herculean efforts.
March 2014->July 2014: Transition remaining services off of deleuze onto KernelVirtualMachines on new kvm host
- Winter 2014: Assuming 150+ members, perhaps more bandwidth is in order
- Spring 2015: Ponies for everyone.
IPv6, mediagoblin, diaspora, gitorious, xmpp works again (and ability to use vanity domains), ...
3. Contact
<clinton at unknownlamer dot org> (Email)
- unknownlamer (AOL Instant Messenger)
<clinton at hcoop dot net> (Jabber)
unknown_lamer on freenode (IRC) in #hcoop
- +1 443 538 8058 (Phone, SMS preffered)
4. Websites
http://unknownlamer.org Personal Homepage
5. Immediate Tasks
- Finish updating wiki for Stripe
Announce DefaultAliasSource = user change (two week warning)
Make domtool's config generation code easier to debug
Push apache2 config updates
mod_disk_cache tweaks
SSL Cipher ordering for better forward secrecy
Restrict /server-status to hcoop members
Upgrade openafs servers to 1.6.7
Add hopper as database server
Remove deleuze as fileserver
Upgrade fritz to openafs 1.6.7
Add hopper as fileserver
Remove deleuze as database server
Rekey
- Create mccarthy virtual machine
Update debarchiver to include wheezy+wheezy-backports
Rebuild binary packages (mod_waklog, libnss-afs, smlnj) for wheezy
- Install apache
Fix KrbVerifyKDC. apache2/${HOSTNAME}.hcoop.net@HCOOP.NET -> /etc/keytabs/service/apache, KrbServiceName apache2/${HOSTNAME}.hcoop.net, Krb5Keytab
Has to be generated per-location with DomTool.
Once working, extract keytab for navajos and deleuze
- Either update suphp fork for wheezy, or get php+fastcgi working and kill suphp
- Get courier working
Check for needed changes to courier-authdaemon
- Exim as secondary MX
- figure out mailman delivery (forward to mailman host, ideally generated by domtool)
figure out wth is going on with unix_hosts, and maybe generate with domtool (or eliminate)
There is some indication unix_hosts exists to provide some kind of fall back routing behavior in case of major domtool misconfiguration?
- Move mailman (and force all mailman mail to mccarthy)
- Possible donations for new server
- Backups (blocked by lack of hardware)
- Spec out new server
6. Admin Stuff
6.1. fastcgi setup notes
Notes to self about configuring fastcgi for at least php
http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html
6.1.1. Packaging
Depend on libapache2-mod-fcgid
a2enmod fcgid in postinst (the package seems to enable itself, but en/dis anyway)
Need to install our own mods-available/fcgid.conf
Do NOT enable fcgid-script script hander by default, require use of fastCgiExtension in DomTool
?: FcgidFixPathinfo 1 to let php work properly (seems to not affect anything else)
- Need to configure process limits (expire fairly quickly to prevent issues with tokens/memory exhaustion)
6.1.2. Problems with openafs
The mod_fcgid spawner runs in its own process pool and therefore without tokens or the ability to acquire tokens for processes it launches. Thus, all fcgi processes must be wrapped to avoid surprising behavior. The FcgidWrapper directive is not very expressive: the "wrapper" is the fastcgi application that is launched and then passed any files matching the extension using SCRIPT_NAME. A wrapper wrapping script is needed to grab tokens before launching the actual wrapper, and just using {Add,Set}Handler fcgid-script won't work as expected (users could of course arrange for programs run that way to grab tokens manually).
mod_wsgid and mod_cgid have identical problems. Inspecting the apache source code makes it appear that it would be possible to fix the situation generally by adding a pre/post suexec hook. The process managers for mod_cgid/mod_fcgid/mod_wsgid are forked from the primordial apache process which I understand has all modules loaded. Modules like mod_auth_kerb and mod_waklog could then inject tickets/tokens/etc. into the environment from which external processes were spawned using the suexec hooks.
6.1.3. Setup Plans
Suexec requires the FcgidWrapper to be owned by the user, generate a wrapper wrapping script for each user in afs space (/afs/hcoop.net/common/httpd/fastcgi/u/us/username/fcgi-wrapper?). Must be readable by system:anyuser
Create a php5-fcgi-wrapper that sets FCGI_MAX_REQUESTS appropriately and install to /afs/hcoop.net/common/bin
Set fcgid-script as the global handler for .php .phtml .php5. Also set default FcgidWrapper for the case of a non-suexec host that intentionally runs without tokens.
Have DomTool generate FcgidWrapper "$user-wrapper-wrapper /afs/hcoop.net/common/bin/php5-fcgi-wrapper" for php extensions in all SuExec enabled hosts
Add DomTool directives to set FcgidWrapper
Config.Apache.fastcgi_wrapper_wrapper : user -> wrapper_path, wrap all generated FcgidWrapper directives with the wrapper wrapper
If we turned off suexec uid/gid checking, there could be a single wrapper-wrapper (perhaps implement SuExecForceUserGroup {on|off} as a global server config option?)
No analog for cgiExtension because we want to avoid extremely surprising behavior
fastScriptAlias location -> your_path = scriptAlias location your_path + set handler to fcgid-script and make your_path the fcgi wrapper for the directory
Allow setFastCgiHandler program in [Location] generally? Would SetHandler and also set the FcgidWrapper simultaneously
6.2. Improve SSL Experience
- We are not managing certificates well
- e.g. we don't check and warn members when their certs are going to expire
- You have to use bugzilla/irc/email to get an intermediate certificate installed
DomTool's SSL support leaves much to be desired
No abstraction for setting up an ssl-only vhost that redirects http -> https
- No abstraction for creating an SSL+non-SSL host that serve the same content
- SSL type is problematic
e.g. sslCertificateChainFile is forced to soft-fail at run-time
use_cert takes a full pathname, it would be better to use a short name. There may be a case where a user may need to use a cert not in the primary store (is there?). If there can be no case made for domtool allowing certs to be stored outside of a single location, we can make use_cert just pass its argument through for transition. sslCertificateChainFile has no reason to take a full pathname and could directly take short names instead.
extern type your_cert; extern val cert : your_cert -> ssl_cert_path; (* SSL = use_cert (cert "mydomain.pem"); *) or: extern type your_cert; extern val cert : your_cert -> ssl; extern val use_cert : ssl -> ssl extern type ssl_cacert; extern val sslCertificateChainFile : ssl_cacert -> [Vhost]; (* sslCertificateChainFile "GandiStandadSSLCA.pem"; *)
Abstractions like moinMoin' and wordPress` do not allow you to set an intermediate certificate
New environment variable var SSLCertificateChain : [ssl_cacert] = []?
6.2.1. Portal
The portal page should allow Intermediate CA Cert permission and installation requests. Installed CA certs should be presented as a drop-down showing the subject CN and file name.
The portal request page should display all certs and cacerts a user is permitted to use already, their common name, and their expiration date.
6.2.2. Managing Certificates
Upon installation, certificates should be recorded in a database that stores (filename, type, subject, expiration) (where type = user | intermediate). Removal, naturally, should remove the database entry and files from all web servers.
This information could then be used to present information on a member's certs on the portal page, and make requesting CA certs easier.
6.3. etc.
default DirectoryIndex does not include index.shtml, should this be changed?
vhostDefault makes configuring the default vhost slightly unpleasant. Extend host with host_default token and eliminate?
Basic git-pbuilder setup on DebianPackaging, noting that that makes it easy for anyone to create and test trivial backports and request they be pulled/built/installed to the official archive.
6.4. Website
(create Website bugzilla product and move these there)
- Convert hcoop.net into domtool config (looks trivial, a few rewrites... except for userdir support?)
- On the topic of user dirs: allow members to register a redirect for hcoop.net/~foo?)
Perhaps: Move userdirs to http://users.hcoop.net/~foo (302ing from hcoop.net/~foo)
Replace 000default with something other than the hcoop web page?
- Replace facebook links with other "get to know the members" text
- Inspire members to join the planet
- Make the locations tool usable again (something we can use with Openstreetmap).
Give BtTempleton and LaurenMcNees write access as needed
6.4.1. Wiki
child pages macro for listing the section of the member manual in the sidebar?
6.5. domtool plans
- Feature backlog
- Networked domtool-tail
Enhance easy_domain DefaultAlias to default to user -> user, but also allow dropping all or catch-all.
default CSymbol = Value for binding default environment variable settings
Expands the scope of extensions that can be written in pure DomTool, slight flexibility gains
vmail helper for the portal VMailPasswdChange $user $oldpass $newpass (restricted to users with vmailadmin permissions)
Improve fwtool as needs become clearer (FirewallTool)
6.6. Major Sysadmin Tasks for 2014
- Acquire and install new server, cleaning up the back of the rack in the process
- Get everything off of deleuze
Clean up keytabs (at least $user.mail for mail delivery, sync to as few nodes as needed, etc.)
- Overhaul mail delivery
- Revisit routing/transport; the current routing config has too many entry and exit points making changes extremely complicated (i.e. likely to break mail delivery)
- Take advantage of new exim features like DKIM.
- Secondary MX
7. Board Stuff
- What address should we be using for the "owner" of hcoop services? It's a mix of the current treasurer, registered agent, and data center right now... is the registered agent correct? Do we need to get a PO Box or something?
- Possible policy change: pro-rate the first month of dues for new members. It seems a bit unfair to make folks pay an entire month of dues, especially if they join near the end of the month.
- Do we need a private wiki of some sort for filing sensitive information like welcome emails for services and various credentials... is it safe enough to do that using the public moin and acls? I'm not sure sure...