welcome: please sign in

Diff for "ClintonEbadi"

Differences between revisions 65 and 95 (spanning 30 versions)
Revision 65 as of 2015-03-20 19:21:51
Size: 17275
Editor: ClintonEbadi
Comment: thinking about permissions and modules with dangerous features and domtool
Revision 95 as of 2020-09-02 05:11:03
Size: 18730
Editor: ClintonEbadi
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
I am Clinton Ebadi. I am the reluctant --(President)--Treasurer of the coop (someone has to do it), and the current lead sysadmin / DomTool maintainer / lo-fi AdamChlipala replacement. I am Clinton Ebadi. I am the Treasurer of the coop (someone has to do it), and the current lead sysadmin / DomTool maintainer / lo-fi AdamChlipala replacement.   <<TableOfContents>>
Line 11: Line 13:
 * (./) January 2013: New website online, navajos and bog both up with new members using them
 * February 2013->June 2014: Big push for new members. New website should make us seem more alive, we've finally fixed about 3/4 of the "temporary" hacks from when we first moved to Peer1, etc. (./) mire is gone (less work: we just have to get people off of it), but also work toward getting rid of deleuze. Web services first (low hanging fruit), even if it means punting on fully converting to domtool managed sites (at least packaged and documented). AFAICT other than those we're just left with:
   * (./) DomTool dispatcher
   * Master DNS
   * Exim
   * IMAP (might have to patch courier-authdaemon)
   * (./) A few straggling openafs volumes (+ AFSDB records?)
   * Backups
   * MailMan
   * The portal
   * {X} FTP (''support for plain ftp removed'')
   * (./) Webalizer
 * April 2014: (./) Rekey the afs cell
 * July/August 2014: Catch the sales on the last generation Dell 2U/2-socket machines and create a clean counterpart to fritz.
   * See NewServerDiscussion2013
   * Minimal KVM host setup for base OS. We might run the KDC and OpenAFS on the bare metal, not entirely certain.
   * Create new wheezy based VMs, duplicating (and turning into master) services hosted in VMs on fritz
     * For the first time, this is realistic: all of those months packaging our configuration pay off by sparing future-me from herculean efforts.
 * March 2014->July 2014: Transition remaining services off of deleuze onto KernelVirtualMachine``s on new kvm host
 * Winter 2014: Assuming 150+ members, perhaps more bandwidth is in order
 * Spring 2015: Ponies for everyone.
   * IPv6, mediagoblin, diaspora, gitorious, (./) xmpp works again (and ability to use vanity domains), ...
 * Summer/Fall 2020: Debian buster across the board, OpenAFS 1.8.6, [[https://compliance.conversations.im/|full compliance with modern XMPP requirements]], maybe [[/JitsiMeet]]
Line 37: Line 18:
 * unknownlamer (AOL Instant Messenger)
* <<MailTo(clinton at hcoop dot net)>> (Jabber)
 * <<MailTo(clinton at hcoop dot net)>> (XMPP)
Line 40: Line 20:
 * +1 443 538 8058 (Phone, SMS preffered)  * +1 443 538 8058 (Phone, SMS preffered as I will ignore your call if I don't have your number already and check voicemail approximately once per century)
Line 44: Line 24:
 * http://unknownlamer.org '''Personal Homepage'''  * https://unknownlamer.org '''Personal Homepage'''
   * [[https://journal.unknownlamer.org|My Journal]] is a bit less dusty
 * http://demoncat.org
Line 47: Line 29:
= Immediate Tasks =

 * (./) Finish updating wiki for Stripe
 * (./) Announce `DefaultAliasSource = user` change (two week warning)
 * (./) Make domtool's config generation code easier to debug
 * (./) Push apache2 config updates
   * (./) mod_disk_cache tweaks
   * (./) SSL Cipher ordering for better forward secrecy
   * (./) Restrict /server-status to hcoop members
 * (./) Upgrade openafs servers to 1.6.7
   * (./) Add hopper as database server
   * (./) Remove deleuze as fileserver
   * (./) Upgrade fritz to openafs 1.6.7
   * (./) Add hopper as fileserver
   * (./) Remove deleuze as database server
   * (./) Rekey
 * Create mccarthy virtual machine
   * (./) Update debarchiver to include wheezy+wheezy-backports
   * (./) Rebuild binary packages (mod_waklog, libnss-afs, smlnj) for wheezy
   * Install apache
     * {X} Fix `KrbVerifyKDC`. `apache2/${HOSTNAME}.hcoop.net@HCOOP.NET` -> `/etc/keytabs/service/apache`, `KrbServiceName apache2/${HOSTNAME}.hcoop.net`, `Krb5Keytab`
       * (./) Has to be generated per-location with DomTool.
       * (./) Once working, extract keytab for navajos and deleuze
     * Either update suphp fork for wheezy, or get php+fastcgi working and kill suphp
   * Get courier working
     * Check for needed changes to `courier-authdaemon`
   * Exim as secondary MX
     * figure out mailman delivery (forward to mailman host, ideally generated by domtool)
     * figure out wth is going on with `unix_hosts`, and maybe generate with domtool (or eliminate)
       * There is some indication `unix_hosts` exists to provide some kind of fall back routing behavior in case of major domtool misconfiguration?
   * Move mailman (and force all mailman mail to mccarthy)
   

 * Possible donations for new server
 * Backups (blocked by lack of hardware)
 * Spec out new server
= Tasks =

== Underway ==

 * (./) buster upgrade on shell and webserver this weekend
 * (./) buster upgrade mail/xmpp server
 * upgrade mysql to 5.7 on gibran
 * change default `PhpVersion` to 7.4
 * upgrade openafs servers to 1.8
   * Procedure for [[https://docs.openafs.org/QuickStartUnix/HDRWQ50.html|creating KeyFileExt]] to replace rxkad
 * upgrade outpost to buster
   * test bind first on ServerBusted
 * upgrade lovelace to buster after making sure things are stable with outpost upgrades
 * start dealing with increasing the krbtgt strength (just adding stronger ciphers and removing the weakest ciphers initially + re-generating high value service principals)
 * upgrade gibran to buster
 * reimplement multiple postgres instance support, upgrade to latest (on port 5432 again finally...)

 * Enter old bills into the portal and reconcile everything so we can implement FinancialReservePolicy
 * Time permitting, squeeze in setting up [[/JitsiMeet]] if tests prove it's feasible for us to host

== Medium term ==

As of this writing, this means "hopefully before the end of 2020"

 * Grant members `all on $db.*` in mysql now that we have backups so if they manage to drop their database it's not fatal anymore. Will make default mysql experience much saner.
 * Opt all members and vmail accounts into spam checking, automatically move spam to ~Maildir/.Junk, (pending discussion) auto delete mail in Junk (see TODO DaemonAdmin/SpamAssassin)
 * Letsencrypt integration into domtool

== Longer Term ==

Probably during 2021

 * Managing data when members depart better
 * Useful, member accessible backups

=== Centralize apache logging ===

domtool-tail is broken, and supporting it is challenging in a multi-server environment. Instead, we could like just use syslog to write logs to either nfs or afs in real time.

 * DomTool would generate a syslog-ng config snippet to monitor each user log file and write to the final destination

AFS:

 * Write directly to `~/.logs/apache/SERVER/DOMAIN/...`
 * Downside: Unless we want to run a bunch of syslog instances, there would have to be a shared syslog principal that could write to all member log directories

NFS:

 * Syslog would just write logs to another directory on the server, and set permissions to the user the log really belongs to (can't do this directly with the apache logs as anyone who can write to the log can do bad things, in theory, so directly mounting `/var/log/apache2` is impossible).
 * Exported directories from each host would be mounted on the member login server
 * Regular UNIX permissions would be used to control access, since we control the user database for both machines and know UIDs will match
 * Logs would be regularly rsynced to afs storage (using `USER.daemon` credentials) as is done now for longer term storage

In either case, might be a good idea to move logs to a dedicated volume? And maybe introduce `USER.log` principles ... is there any reason we would want to allow `USER.daemon` access to the log directory by default? Only case I could see would be members using daemon roles to automatically process logs, in which case presumably they would be able to handle granting the required permissions manually.
Line 86: Line 86:
== fastcgi setup notes ==

Notes to self about configuring fastcgi for at least php

http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html

=== Packaging ===

 * Depend on `libapache2-mod-fcgid`
 * `a2enmod fcgid` in postinst (the package seems to enable itself, but en/dis anyway)
 * Need to install our own `mods-available/fcgid.conf`
   * Do NOT enable `fcgid-script` script hander by default, require use of `fastCgiExtension` in DomTool
   * ?: `FcgidFixPathinfo 1` to let php work properly (seems to not affect anything else)
   * Need to configure process limits (expire fairly quickly to prevent issues with tokens/memory exhaustion)

=== Problems with openafs ===

The mod_fcgid spawner runs in its own process pool and therefore without tokens or the ability to acquire tokens for processes it launches. Thus, all fcgi processes must be wrapped to avoid surprising behavior. The `FcgidWrapper` directive is not very expressive: the "wrapper" is the fastcgi application that is launched and then passed any files matching the extension using `SCRIPT_NAME`. A wrapper wrapping script is needed to grab tokens before launching the actual wrapper, and just using `{Add,Set}Handler fcgid-script` won't work as expected (users could of course arrange for programs run that way to grab tokens manually).

mod_wsgid and mod_cgid have identical problems. Inspecting the apache source code makes it appear that it would be possible to fix the situation generally by adding a pre/post suexec hook. The process managers for mod_cgid/mod_fcgid/mod_wsgid are forked from the primordial apache process which I understand has all modules loaded. Modules like mod_auth_kerb and mod_waklog could then inject tickets/tokens/etc. into the environment from which external processes were spawned using the suexec hooks.

=== Setup Plans ===

 * Suexec requires the `FcgidWrapper` to be owned by the user, generate a wrapper wrapping script for each user in afs space (`/afs/hcoop.net/common/httpd/fastcgi/u/us/username/fcgi-wrapper`?). Must be readable by `system:anyuser`
 * Create a `php5-fcgi-wrapper` that sets `FCGI_MAX_REQUESTS` appropriately and install to `/afs/hcoop.net/common/bin`
 * Set `fcgid-script` as the global handler for `.php .phtml .php5`. Also set default `FcgidWrapper` for the case of a non-suexec host that intentionally runs without tokens.
 * Have DomTool generate `FcgidWrapper "$user-wrapper-wrapper /afs/hcoop.net/common/bin/php5-fcgi-wrapper"` for php extensions in all SuExec enabled hosts
 * Add DomTool directives to set `FcgidWrapper`
   * `Config.Apache.fastcgi_wrapper_wrapper : user -> wrapper_path`, wrap all generated `FcgidWrapper` directives with the wrapper wrapper
     * If we turned off suexec uid/gid checking, there could be a single wrapper-wrapper (perhaps implement `SuExecForceUserGroup {on|off}` as a global server config option?)
   * No analog for `cgiExtension` because we want to avoid extremely surprising behavior
   * `fastScriptAlias location -> your_path` = `scriptAlias location your_path` + set handler to `fcgid-script` and make `your_path` the fcgi wrapper for the directory
   * Allow `setFastCgiHandler program` in `[Location]` generally? Would `SetHandler` and also set the `FcgidWrapper` simultaneously
Line 123: Line 89:
   * e.g. we don't check and warn members when their certs are going to expire
   * You have to use bugzilla/irc/email to get an intermediate certificate installed
 * DomTool's SSL support leaves much to be desired
   * No abstraction for setting up an ssl-only vhost that redirects http -> https
   * No abstraction for creating an SSL+non-SSL host that serve the same content
   * SSL type is problematic
     * e.g. `sslCertificateChainFile` is forced to soft-fail at run-time
   * `use_cert` takes a full pathname, it would be better to use a short name. There may be a case where a user may need to use a cert not in the primary store (is there?). If there can be no case made for domtool allowing certs to be stored outside of a single location, we can make `use_cert` just pass its argument through for transition. `sslCertificateChainFile` has no reason to take a full pathname and could directly take short names instead.
     * {{{
   * we don't check and warn members when their certs are going to expire
   * requesting certs via the filesystem is clunky for most users, we should support upload via the portal instead
 * Need to add letsencrypt support


=== Domtool ===

Replace `use_cert` `cert` function that just takes the final name instead of the full pathname, providing the full pathname is kind of clunky.

{{{
Line 133: Line 100:
extern val cert : your_cert -> ssl_cert_path;
(* SSL = use_cert (cert "mydomain.pem"); *)

or:

extern type your_cert;
extern val cert : your_cert -> ssl;
extern val use_cert : ssl -> ssl

extern type ssl_cacert;
extern val sslCertificateChainFile : ssl_cacert -> [Vhost];
(* sslCertificateChainFile "GandiStandadSSLCA.pem"; *)
}}}
    * Abstractions like `moinMoin' and `wordPress` do not allow you to set an intermediate certificate
      * New environment variable `var SSLCertificateChain : [ssl_cacert] = []`?
extern val cert : your_cert -> ssl
(* SSL = cert "mydomain.pem"; *)
}}}

We also need to support letencrypt, perhaps like so:

{{{
extern type your_letsencrypt_cert;
extern val letsencrypt : your_letsencrypt_cert -> ssl
(* SSL = letsencrypt "my.domain"; *)
}}}

Which would find the certs in the standard location used by certbot... or we could use symlinks there from our location.
Line 151: Line 116:
The portal page should allow Intermediate CA Cert permission and installation requests. Installed CA certs should be presented as a drop-down showing the subject CN and file name.

The portal request page should display all certs and cacerts a user is permitted to use already, their common name, and their expiration date.
We should allow submission of certs / keys through the web interface. Can use an insert-only afs dir to securely allow hcoop.daemon to write certs without being able to access them, which would then be installed manually by an admin using `ca-install`.

The portal request page should display all certs a user is permitted to use already, their common name, and their expiration date.
Line 157: Line 122:
Upon installation, certificates should be recorded in a database that stores `(filename, type, subject, expiration)` (where `type = user | intermediate`). Removal, naturally, should remove the database entry and files from all web servers.

This information could then be used to present information on a member's certs on the portal page, and make requesting CA certs easier.
''Needs Updating'' Not very fleshed out, also does not consider how we're going to manage letsencrypt certs

Since we no longer need to support explicit intermediate certs (everything nowadays accepts the chain in the main certificate file), we can just use domtool's cert permission to track things.

A cron should check for things like:

 * Certs that are not owned by any member in domtool
 * Permissions to certs that don't exist
 * Expired or soon to expire certs (should exponentially back off notice until expiration date changes, like quotacheck does for low space), emailing the member as well as admins@

Certificate CN and validity dates should be shown on the portal ssl page; ssl check cron should cache this somewhere the portal can read it.

destroy-user needs to nuke certs for leaving members ... we need to overhaul this generally and stash all member data in one location when destroying for later removal / restoration (if they return in the 30 day deletion window).

=== Managing Member Data Better ===

Trying to hash out how we can better manage user data when they depart; right now we reply on someone (me...) to go in 30 days after removing a member to remove all of their non-afs data manually, which is prone to error and needs to be managed better if we want to comply with things like GDPR (although I don't think it applies to us directly as a US corporation, it would be ''nice'' to do so since we really don't want to be retaining data longer than needed anyway, and the US will likely have stringent privacy laws aimed at facebook/google that will punish all the same for failing to remove data immediately too...).

 * Databases (/srv/databases/u/us/user/*)
 * SSL certs /etc/apache2/ssl/user/$domain
   * Maybe add username to the path; we are managing these with domtool perms so it's easy enough to find which certs belonged to a member... until we remove all of their permissions when removing their account.
 * Portal database data
   * Payment records / archived member app: cannot be deleted, we are legally required to have open books for our members
   * URL and location database entries: these should definitely be purged after the retention period
 * Firewall rules
   * Currently rules are stored in one file, would be easier to manage if we split each member's rules into a separate file
 * GNU Mailman lists
   * We don't track who owns which list outside of the list control request in the portal
 * ejabberd data
   * mnesia: rosters, PEP, ... we might want to convert our mnesia database to postgres to make removing data associated with departing members simpler
   * mod_mam -- actual messages, but in SQL (`delete * from archive where username = 'XXX';`): we haven't started storing these as of 2019-05-10, so it would be a good candidate to add to an initial `purge-user` script when setting up.
   * mod_http_upload files in local storage on the ejabberd server (we expire these every 30 days already)
 * Incidental data in the roundcube webmail database (address book, preferences)
   * Including data for any vmail users associated with the member (additional wrinkle there: should we be removing this data as soon as a member deletes a vmail user?)

Not personal data, but things we should clear for housekeeping in general:

 * Apache davlockdb directory

We already manage mail and $HOME data fine, both are easy to clear (delete the volumes, done).

One aspect of the solution I've been thinking about that would also make it easier for members to export their data in general: we could set up a `backups.$user` volume, and store dumps of data we hold on behalf of the data there: database backups, mailing list archives, an exported dump of their ejabberd roster, maybe even copies of their ssl certs.

We also need a registry of data we keep on behalf of members that can't immediately be identified based on their username: at least ssl certs and mailing lists (those might be it though... and certs are already tracked with domtool perms so maybe just adding a "list" perm would be adequate, which could then be used to maybe allow members to perform some list management through a domtool program). And we need to store things like domtool permissions when destroying (might be able to achieve some of this by first freezing members before removing them), so that we can use them later when performing the final purge of data after the retention period ends (important to keep a brief retention period for those "oh right, I have to actually pay dues" moments...).
Line 163: Line 169:
 * default DirectoryIndex does not include index.shtml, should this be changed?  * default Directory``Index does not include index.shtml, should this be changed?
Line 165: Line 171:
 * Basic `git-pbuilder` setup on DebianPackaging, noting that that makes it easy for ''anyone'' to create and test trivial backports and request they be pulled/built/installed to the official archive.
Line 174: Line 180:
   * Replace `000default` with something other than the hcoop web page?
 *
Replace facebook links with other "get to know the members" text
 * (./) Replace facebook links with other "get to know the members" text
Line 178: Line 183:
 * Give BtTempleton and LaurenMcNees write access as needed  * Give RobinTempleton access as needed
Line 187: Line 192:
 * Networked domtool-tail
 * (./) Enhance easy_domain `DefaultAlias` to default to `user -> user`, but also allow dropping all or catch-all.
 * (./) `default CSymbol = Value` for binding default environment variable settings
    * Expands the scope of extensions that can be written in pure DomTool, slight flexibility gains
 * (./) vmail helper for the portal `VMailPasswdChange $user $oldpass $newpass` (restricted to users with `vmailadmin` permissions)
Line 205: Line 205:


What I'll probably do: just copy the vhost from deleuze onto navajos and call it a day for the next year or two.
Line 245: Line 242:
== Major Sysadmin Tasks for 2014 ==

 * Acquire and install new server, cleaning up the back of the rack in the process
 * Get everything off of deleuze
 * Clean up keytabs (at least `$user.mail` for mail delivery, sync to as few nodes as needed, etc.)
 * Overhaul mail delivery
   * Revisit routing/transport; the current routing config has too many entry and exit points making changes extremely complicated (i.e. likely to break mail delivery)
   * Take advantage of new exim features like DKIM.
   * Secondary MX
=== ip / ipv4 / ipv6 ===

There are a few places (mostly apache) where it would be great to be able to interchange ip and ipv6 addresses. But there's no way to subtype in domtool (except for refining base int and string).

It's been shoehorned in for now (always requiring a node to have an ipv6 address), but this can be a bit awkward (e.g. `webAtIp` requires that an ipv4 and ipv6 address be provided).

--(Also might make sense to be able to pass an array of IPs in a few spots instead of just fixing it at one ipv4 and one ipv6 address per `WebPlace`.)-- you can just pass more than one WebPlace already.

= Things to Look At =

== Easier App Install ==

We should probably figure out how we can support docker containers because they are popular and there is no stopping them (plus maybe they are a ''bit'' convenient). [[https://podman.io/|Podman]] in theory works on Debian Buster and might even run in afs (I did one test and it seemed to)... but mainly it lets users run containers as themselves without need a daemon running as root and zero security.

 * https://docs.sandstorm.io/en/latest/administering/hosting-provider/ "Sandstorm is an open source platform for self-hosting web apps" that claims to support multiple users on the same server
 * https://cockpit-project.org/blog/cockpit-206.html web gui for podman that integrates with kerberos and claims to allow unprivileged users to manage containers under their own account
Line 259: Line 264:
   * Board did approve this, but was never implemented :-\
Line 269: Line 275:
== kallithea ==

Following https://pythonhosted.org/Kallithea/installation.html

 * Grant write permissions to daemon principle for write directories:
{{{
cache_dir = %(here)s/data
index_dir = %(here)s/data/index
archive_cache_dir = %(here)s/tarballcache
}}}
== Fastcgi Problems with openafs ==

''Old Content'': This section is from before fastcgi was implemented. Leaving so I might remember one day to try the idea of adding an suexec hook to apache.

The mod_fcgid spawner runs in its own process pool and therefore without tokens or the ability to acquire tokens for processes it launches. Thus, all fcgi processes must be wrapped to avoid surprising behavior. The `FcgidWrapper` directive is not very expressive: the "wrapper" is the fastcgi application that is launched and then passed any files matching the extension using `SCRIPT_NAME`. A wrapper wrapping script is needed to grab tokens before launching the actual wrapper, and just using `{Add,Set}Handler fcgid-script` won't work as expected (users could of course arrange for programs run that way to grab tokens manually).

mod_wsgid and mod_cgid have identical problems. Inspecting the apache source code makes it appear that it would be possible to fix the situation generally by adding a pre/post suexec hook. The process managers for mod_cgid/mod_fcgid/mod_wsgid are forked from the primordial apache process which I understand has all modules loaded. Modules like mod_auth_kerb and mod_waklog could then inject tickets/tokens/etc. into the environment from which external processes were spawned using the suexec hooks.

I am Clinton Ebadi. I am the Treasurer of the coop (someone has to do it), and the current lead sysadmin / DomTool maintainer / lo-fi AdamChlipala replacement.

1. Board Statements

See /BoardStatements

2. General Coop Goals

Unstructured musing on when/what I think the coop ought to be.

3. Contact

  • <clinton at unknownlamer dot org> (Email)

  • <clinton at hcoop dot net> (XMPP)

  • unknown_lamer on freenode (IRC) in #hcoop

  • +1 443 538 8058 (Phone, SMS preffered as I will ignore your call if I don't have your number already and check voicemail approximately once per century)

4. Websites

5. Tasks

5.1. Underway

  • (./) buster upgrade on shell and webserver this weekend

  • (./) buster upgrade mail/xmpp server

  • upgrade mysql to 5.7 on gibran
  • change default PhpVersion to 7.4

  • upgrade openafs servers to 1.8
  • upgrade outpost to buster
  • upgrade lovelace to buster after making sure things are stable with outpost upgrades
  • start dealing with increasing the krbtgt strength (just adding stronger ciphers and removing the weakest ciphers initially + re-generating high value service principals)
  • upgrade gibran to buster
  • reimplement multiple postgres instance support, upgrade to latest (on port 5432 again finally...)
  • Enter old bills into the portal and reconcile everything so we can implement FinancialReservePolicy

  • Time permitting, squeeze in setting up /JitsiMeet if tests prove it's feasible for us to host

5.2. Medium term

As of this writing, this means "hopefully before the end of 2020"

  • Grant members all on $db.* in mysql now that we have backups so if they manage to drop their database it's not fatal anymore. Will make default mysql experience much saner.

  • Opt all members and vmail accounts into spam checking, automatically move spam to ~Maildir/.Junk, (pending discussion) auto delete mail in Junk (see TODO DaemonAdmin/SpamAssassin)

  • Letsencrypt integration into domtool

5.3. Longer Term

Probably during 2021

  • Managing data when members depart better
  • Useful, member accessible backups

5.3.1. Centralize apache logging

domtool-tail is broken, and supporting it is challenging in a multi-server environment. Instead, we could like just use syslog to write logs to either nfs or afs in real time.

  • DomTool would generate a syslog-ng config snippet to monitor each user log file and write to the final destination

AFS:

  • Write directly to ~/.logs/apache/SERVER/DOMAIN/...

  • Downside: Unless we want to run a bunch of syslog instances, there would have to be a shared syslog principal that could write to all member log directories

NFS:

  • Syslog would just write logs to another directory on the server, and set permissions to the user the log really belongs to (can't do this directly with the apache logs as anyone who can write to the log can do bad things, in theory, so directly mounting /var/log/apache2 is impossible).

  • Exported directories from each host would be mounted on the member login server
  • Regular UNIX permissions would be used to control access, since we control the user database for both machines and know UIDs will match
  • Logs would be regularly rsynced to afs storage (using USER.daemon credentials) as is done now for longer term storage

In either case, might be a good idea to move logs to a dedicated volume? And maybe introduce USER.log principles ... is there any reason we would want to allow USER.daemon access to the log directory by default? Only case I could see would be members using daemon roles to automatically process logs, in which case presumably they would be able to handle granting the required permissions manually.

6. Admin Stuff

6.1. Improve SSL Experience

  • We are not managing certificates well
    • we don't check and warn members when their certs are going to expire
    • requesting certs via the filesystem is clunky for most users, we should support upload via the portal instead
  • Need to add letsencrypt support

6.1.1. Domtool

Replace use_cert cert function that just takes the final name instead of the full pathname, providing the full pathname is kind of clunky.

extern type your_cert;
extern val cert : your_cert -> ssl
(* SSL = cert "mydomain.pem"; *)

We also need to support letencrypt, perhaps like so:

extern type your_letsencrypt_cert;
extern val letsencrypt : your_letsencrypt_cert -> ssl
(* SSL = letsencrypt "my.domain"; *)

Which would find the certs in the standard location used by certbot... or we could use symlinks there from our location.

6.1.2. Portal

We should allow submission of certs / keys through the web interface. Can use an insert-only afs dir to securely allow hcoop.daemon to write certs without being able to access them, which would then be installed manually by an admin using ca-install.

The portal request page should display all certs a user is permitted to use already, their common name, and their expiration date.

6.1.3. Managing Certificates

Needs Updating Not very fleshed out, also does not consider how we're going to manage letsencrypt certs

Since we no longer need to support explicit intermediate certs (everything nowadays accepts the chain in the main certificate file), we can just use domtool's cert permission to track things.

A cron should check for things like:

  • Certs that are not owned by any member in domtool
  • Permissions to certs that don't exist
  • Expired or soon to expire certs (should exponentially back off notice until expiration date changes, like quotacheck does for low space), emailing the member as well as admins@

Certificate CN and validity dates should be shown on the portal ssl page; ssl check cron should cache this somewhere the portal can read it.

destroy-user needs to nuke certs for leaving members ... we need to overhaul this generally and stash all member data in one location when destroying for later removal / restoration (if they return in the 30 day deletion window).

6.1.4. Managing Member Data Better

Trying to hash out how we can better manage user data when they depart; right now we reply on someone (me...) to go in 30 days after removing a member to remove all of their non-afs data manually, which is prone to error and needs to be managed better if we want to comply with things like GDPR (although I don't think it applies to us directly as a US corporation, it would be nice to do so since we really don't want to be retaining data longer than needed anyway, and the US will likely have stringent privacy laws aimed at facebook/google that will punish all the same for failing to remove data immediately too...).

  • Databases (/srv/databases/u/us/user/*)
  • SSL certs /etc/apache2/ssl/user/$domain
    • Maybe add username to the path; we are managing these with domtool perms so it's easy enough to find which certs belonged to a member... until we remove all of their permissions when removing their account.
  • Portal database data
    • Payment records / archived member app: cannot be deleted, we are legally required to have open books for our members
    • URL and location database entries: these should definitely be purged after the retention period
  • Firewall rules
    • Currently rules are stored in one file, would be easier to manage if we split each member's rules into a separate file
  • GNU Mailman lists
    • We don't track who owns which list outside of the list control request in the portal
  • ejabberd data
    • mnesia: rosters, PEP, ... we might want to convert our mnesia database to postgres to make removing data associated with departing members simpler
    • mod_mam -- actual messages, but in SQL (delete * from archive where username = 'XXX';): we haven't started storing these as of 2019-05-10, so it would be a good candidate to add to an initial purge-user script when setting up.

    • mod_http_upload files in local storage on the ejabberd server (we expire these every 30 days already)
  • Incidental data in the roundcube webmail database (address book, preferences)
    • Including data for any vmail users associated with the member (additional wrinkle there: should we be removing this data as soon as a member deletes a vmail user?)

Not personal data, but things we should clear for housekeeping in general:

  • Apache davlockdb directory

We already manage mail and $HOME data fine, both are easy to clear (delete the volumes, done).

One aspect of the solution I've been thinking about that would also make it easier for members to export their data in general: we could set up a backups.$user volume, and store dumps of data we hold on behalf of the data there: database backups, mailing list archives, an exported dump of their ejabberd roster, maybe even copies of their ssl certs.

We also need a registry of data we keep on behalf of members that can't immediately be identified based on their username: at least ssl certs and mailing lists (those might be it though... and certs are already tracked with domtool perms so maybe just adding a "list" perm would be adequate, which could then be used to maybe allow members to perform some list management through a domtool program). And we need to store things like domtool permissions when destroying (might be able to achieve some of this by first freezing members before removing them), so that we can use them later when performing the final purge of data after the retention period ends (important to keep a brief retention period for those "oh right, I have to actually pay dues" moments...).

6.2. etc.

  • default DirectoryIndex does not include index.shtml, should this be changed?

  • vhostDefault makes configuring the default vhost slightly unpleasant. Extend host with host_default token and eliminate?

6.3. Website

(create Website bugzilla product and move these there)

  • Convert hcoop.net into domtool config (looks trivial, a few rewrites... except for userdir support?)
    • On the topic of user dirs: allow members to register a redirect for hcoop.net/~foo?)
    • Perhaps: Move userdirs to http://users.hcoop.net/~foo (302ing from hcoop.net/~foo)

  • (./) Replace facebook links with other "get to know the members" text

    • Inspire members to join the planet
    • Make the locations tool usable again (something we can use with Openstreetmap).
  • Give RobinTempleton access as needed

6.3.1. Wiki

6.4. domtool plans

  • Feature backlog
  • Improve fwtool as needs become clearer (FirewallTool)

6.4.1. restricted modules for apache

Inspiration: hcoop.net's vhost is non generated by domtool, and only because it enabled mod_userdir

Idea: have a set of restricted modules that can only be used by superusers. Easiest to just have another ad-hoc list setting in config for domtool. ACL example: hcoop priv www, www priv overloaded to also allow use of restricted module.

Problems: no way currently to restrict access to actions or lib files.

Deficiences: priv www is a blunt instrument. priv system in general is mediocre. It might be nice to be able to do something like hcoop priv www apache-module/userdir mail/hopper.hcoop.net (i.e. access to all www nodes, access to the userdir module only, access to hopper). Keys gain some hierarchy polluting the purity of the triples db, but it is already a bit polluted... is there any difference between adding hierarchy to priv keys and the existing implicit hierarchies in domains and paths?

Solution: might be overkill just for mod_userdir, if it looks like minimal additional code is required perhaps implement hierarchical privs (extending www and mail privs to support limiting to particular admin hosts) and restricted apache modules.

6.4.2. Pattern Matching and New Types

A vague idea that may prove to be unworkable. I think at least implementing list matching in domtool would be quite useful. Abstraction syntax would be need to be improved to support multiple clauses. case would also be needed to make it useful. Syntax would be easy enough to add except for having to deal with runtime non-exhaustive match exceptions (perhaps requiring exhaustive matches and living with the limitation). Ambitious, probably time consuming, might require adding tail call optimization to the interpreter. Example:

(* A map operator *)
val map = \action -> \list -> 
  case list of head::tail =>
    begin
      action head;
      map action tail;
    end
     | [] => Skip;
(* Alias a list of email addresses to *)
val multiAlias = \sources -> \target -> map (\source -> emailAlias source target) sources;

I probably lack the skill/willpower in the short term... alternative idea, just implement a loop primitive in SML and magic the types away by making it a primitive construct (defining its type on DomTool/LanguageReference). Maybe implement polymorphic actions if adding then is secretly easy:

extern val map : (('a -> 'b) -> ['a]) -> [^Root];

(* Alias a list of email addresses to *)
val multiAlias = \sources -> \target -> map (\source -> emailAlias source target) sources;

Most of the gain, none of the pain.

New types: even more ambitious. Supporting at least tuples or named records, and perhaps a construct for querying the domtool acl database. Idea would be to use it for something like the firewall, where only primitive "generate one firewall rule" constructs would be needed, and then user firewalls could be constructed by querying the ports available to each user and matching/looping.

One pattern that has recurred in domtool is that of a special purpose client + server commands that operates on a simple database. E.g. spamassassin prefs, vmail users, firewall rules, and the domtool acl database. It would be useful to have a generalized serialize/unserialize sets of sml records library, perhaps with a generalized/queryable tuples database built on top of the primitive raw-records database. Even better would be to allow databases to be exposed to domtool, and simple queries performed on them. Maybe.

val writeRecord' : [('record -> 

6.4.3. ip / ipv4 / ipv6

There are a few places (mostly apache) where it would be great to be able to interchange ip and ipv6 addresses. But there's no way to subtype in domtool (except for refining base int and string).

It's been shoehorned in for now (always requiring a node to have an ipv6 address), but this can be a bit awkward (e.g. webAtIp requires that an ipv4 and ipv6 address be provided).

Also might make sense to be able to pass an array of IPs in a few spots instead of just fixing it at one ipv4 and one ipv6 address per WebPlace. you can just pass more than one WebPlace already.

7. Things to Look At

7.1. Easier App Install

We should probably figure out how we can support docker containers because they are popular and there is no stopping them (plus maybe they are a bit convenient). Podman in theory works on Debian Buster and might even run in afs (I did one test and it seemed to)... but mainly it lets users run containers as themselves without need a daemon running as root and zero security.

8. Board Stuff

  • What address should we be using for the "owner" of hcoop services? It's a mix of the current treasurer, registered agent, and data center right now... is the registered agent correct? Do we need to get a PO Box or something?
  • Possible policy change: pro-rate the first month of dues for new members. It seems a bit unfair to make folks pay an entire month of dues, especially if they join near the end of the month.
    • Board did approve this, but was never implemented :-\
  • Do we need a private wiki of some sort for filing sensitive information like welcome emails for services and various credentials... is it safe enough to do that using the public moin and acls? I'm not sure sure...

9. etc

Barely formed sentences.

9.1. tt-rss at hcoop

Our postgresql does not use passwords. The installer needs a single tweak to remove the required attribute of the database password to install.

9.2. Fastcgi Problems with openafs

Old Content: This section is from before fastcgi was implemented. Leaving so I might remember one day to try the idea of adding an suexec hook to apache.

The mod_fcgid spawner runs in its own process pool and therefore without tokens or the ability to acquire tokens for processes it launches. Thus, all fcgi processes must be wrapped to avoid surprising behavior. The FcgidWrapper directive is not very expressive: the "wrapper" is the fastcgi application that is launched and then passed any files matching the extension using SCRIPT_NAME. A wrapper wrapping script is needed to grab tokens before launching the actual wrapper, and just using {Add,Set}Handler fcgid-script won't work as expected (users could of course arrange for programs run that way to grab tokens manually).

mod_wsgid and mod_cgid have identical problems. Inspecting the apache source code makes it appear that it would be possible to fix the situation generally by adding a pre/post suexec hook. The process managers for mod_cgid/mod_fcgid/mod_wsgid are forked from the primordial apache process which I understand has all modules loaded. Modules like mod_auth_kerb and mod_waklog could then inject tickets/tokens/etc. into the environment from which external processes were spawned using the suexec hooks.


CategoryHomepage

ClintonEbadi (last edited 2021-11-06 17:50:12 by ClintonEbadi)