gibran.hcoop.net is virtual machine at digital ocean that will become our primary afs server

It is named after the author Kahlil Gibran

1. Setup Notes

Or: things that need to go into Puppet

• set domain name to hcoop.net manually

  • need to review setup... hostname = gibran, using domain hcoop.net in resolv.conf, and 159.203.101.102 gibran.hcoop.net gibran in hosts (similar setup to other hcoop servers), but ... maybe we should just be leaving hosts alone and putting the fqdn into hostname ?

  • original setup had "gibran.localdomain gibran 127.0.1.1"

• removed joe (or at least update-alternatives editor to either vim or emacs...)

• root has basic emacs config for puppet-mode and melpa (probably no need to formalize that...)
• manually installed libnss-afs

1.1. todo

• default "cloud-config" system may be active, check license and remove if it is non-free

  • looks like it might just be https://help.ubuntu.com/community/CloudInit which would make it acceptable to keep in place

2. Puppet

2.1. puppetserver

• Installed https://apt.puppetlabs.com/puppet5-release-stretch.deb manually

• Packages: puppetserver, puppet-agent
• added /opt/puppetlabs/bin/ to root $PATH in .bashrc

Puppet git structure (different repos for each): /etc/puppetlabs/puppet, /etc/puppetlabs/code/environments/production (excludes modules), /etc/puppetlabs/code/environments/production/modules/hcoop, /etc/puppetlabs/code/environments/production/modules/hcoop_private. Subject to change.

Git repos structure and tracking of installed modules will be revisited once we need to set up multiple environments. For now, /etc/puppetlabs/code/environments/production/modules/hcoop is where all of our code aside from node definitions lives. /etc/puppetlabs/code/environments/production/modules/hcoop_private is for private data (krb5 host keys, ssl keys, etc.) that needs to be managed by Puppet. Ideally we would use something like wallet for this instead. hcoop_private contains only virtual references to files tagged appropriately so they can be realized on individual servers.

Puppet module structure:

• hcoop
  • server
    • $server (e.g. gibran)
  • service
    • openafs-client

2.2. puppetdb

install guide is weird

• puppet resource package puppetdb ensure=latest puppet resource package puppetdb-termini ensure=latest puppet module install puppetlabs-puppetdb

2.3. installed modules

• puppetlabs-firewall
• puppetlabs-puppetdb
• alexharvey-firewall_multi (says incompatible, but works... enough).
• stm-resolv_conf
• ccin2p3-mit_krb5
• stm-debconf
• saz-sudo
• herculesteam-augeasproviders_pam
• herculesteam-augeasproviders_core
• saz-timezone

2.4. style guide

Ideas for keeping consistency among admins

• Use firewall_multi for all rules unless it really is ipv4 or ipv6 only, provider is set in defaults and will keep ipv4 and ipv6 firewall in sync

• Should pass puppet-lint (enforce using git hook) / rspect https://puppet.com/docs/puppet/5.5/style_guide.html

• Inheritance is discouraged? Avoiding it for now
• Files controlled by puppet have comment "This file is managed by Puppet. DO NOT EDIT." somewhere near the top
• Some structure to firewall rule numbers
  • Under 100 for core system things that need to go near the beginning
  • Over 900 for core system things that need to go near the end (e.g. jumping to fwtool output chains)

3. Manual openafs bits

Should go into thoroughly revised SetupNewAfsServer

• set up packages, bos, userlist, server cellservdb with puppet

• after cellservdb is updated on all server, have bos restart $server ptserver vlserver on existing servers before new server will join

  • Seems scary, but afs works it out...

• verify using udebug

• Need to manually sync read only volumes

Other things

• instead of using bos to add users and daemons, puppet will be used. hosts still manually managed.

• AlwaysAttach is only needed if /vicepa isn't a partition

• Not sure we want to link /etc/openafs/CellServDB to /etc/openafs/server/CellServDB or not
  • downside: client ignores dns, upside: client works if dns is down

Questions

• will openafs be smart enough find fileservers on private interfaces?
  • if not, local aliases in hosts? any way to achieve this? (private networking is unbilling, so ideally we will take advantage of it)