ServerMigration2018

1. Open Problems/Questions After Migration

/etc/keytabs/user.daemon/* is still manually synced and not managed by Puppet
/etc/keytabs/{domtool,hcoop} should be moved to /etc/keytabs/service/ for consistency
Systems have no swap (but a bit more ram than our existing KVM images), digital ocean discourages using swap due to wear issues on the underlying drives, looks like it would be best to respect that wish for now.

Mail server set up
- exim config rebased on latest, test keep_environment
- exim, courier, mailman, ejabberd
- migrate mailman (no reason to delay for other services, everything is local to the server)
New web server set up
- DomTool ported to apache 2.4 (reuse 2.2 config if needed, can deal with new Require syntax later)
- Defer on PHP 7.x until after dropping colo expense, unless it proves easy to support in parallel with 5.6
- Only support fastcgi php, suphp is 100% dead
Databases setup
- Keep mysql 5.6, upgrade to postgres 10
- dbs are so small, probably just do a flag day and suffer a few minutes downtime

At this point, we can start moving members. Volunteers first, followed by folks using daemons on bog, then everyone.

Volunteers manually (moving vhosts and primary mx to new servers, then migrating volumes)
HCoop services and volumes
Mail for all members and webmail interfaces
Anyone needing to move daemons + other services at once from bog
Final move of all remaining volumes and changes defaults for DomTool to new servers

Debian packages updated for stretch
- create proper signed repository on ServerNavajos, rebuild packages
- config-packages may be dropped here
Board votes on new provider
Account at provider set up
New primary and secondary afs and kerberos servers set up
- Small 10G block storage attached to primary, expanded once we begin moving volumes. No block storage needed for secondary.
- Added to existing cluster as secondaries
- Create test volumes, check cross-site performance, make sure moving volumes works reliably
Set up puppet [might be able to preload before new servers set up]
- simple puppetdb + manual git pull + puppet apply from cron on each server
- convert config-packaged config to puppet classes as needed
Set up new members vm
- Easy setup, allow members to test on stretch as early as possible

Enable IPv6 for all services that can support it
- Assuming there's no/minimal risk of people with broken ipv6 being unable to access hcoop

What to do with hardware?
- Need to scrub hard drives (mail to board member for manual destruction?), datacenter has on-site server recycling so we could just let it be recycled (essentially worth $0 at this point)
Check bandwidth overage rates... may need to limit transfer rates to avoid, but weigh against another month of hosting
Terminate contract
- have members migrated two weeks ahead of time ideally...