Server `busted.hcoop.net` is a virtual machine at DigitalOcean that was created to work on the Debian Stretch to Buster upgrade.

It's name is just an allusion to it being broken by design.

== Setup Notes ==

=== resolv.conf / initial puppet cert request ===

We can't really get around manually opening the firewall for the agent on the puppetmaster... at our scale this isn't a big deal anyway.

Like others, had to set `domain hcoop.net` manually in the config. It looks like the only reason we need this is for the initial cert request. So I tried setting the agent config at `/etc/puppetlabs/puppet/puppet.conf` to:

{{{
[main]
server = puppet.hcoop.net
}}}

But the cert for the master only has the fqdn of its concrete hostname, and the alias `puppet` with no domain

{{{
Error: Server hostname 'puppet.hcoop.net' did not match server certificate; expected one of gibran.hcoop.net, DNS:puppet, DNS:gibran.hcoop.net
Error: Could not run: Server hostname 'puppet.hcoop.net' did not match server certificate; expected one of gibran.hcoop.net, DNS:puppet, DNS:gibran.hcoop.net
}}}

If we could regenerate this to also include `CN:puppet.hcoop.net`, the manual edit that needed to be done would at least be more related to the limitation in our infrastructure that mandates it...

== Puppet porting notes ==

=== ntp tcp rule failure ===

We were setting our ntp out rule using tcp and udp, but `/etc/services` only had the udp alias now (which is correct). Pushed out a fix, but for some reason runs still failed with the same error afterward. Hacked around it by adding the `ntp/tcp` alias to `/etc/services`. Need to look into further (reinstall...).

{{{
Error: Failed to apply catalog: Parameter dport failed on Firewall[010 ntp output protocol tcp using provider iptables]: Munging failed for value "ntp" in class dport: no such service ntp/tcp (file: /etc/puppetlabs/code/environments/production/modules/firewall_multi/manifests/init.pp, line: 126)
}}}

=== afs client not immediately available ===

{{{
Loading new openafs-1.8.2 DKMS files...
Building for 4.19.0-6-cloud-amd64
Module build for kernel 4.19.0-6-cloud-amd64 was skipped since the
kernel headers for this kernel does not seem to be installed.
}}}

Issue seems to be that digitalocean is now using a the amd64-cloud variant of the kernel, so we're pulling in the wrong headers. Need to check into this more, it looks like a standard part of Debian.


=== libnss_afs installs to non-multiarch location ===

We're still installing to just /usr/lib instead of /usr/lib/x86_64-linux-gnu/ (need to update package to comply with multiarch)

=== HCoop Debian Package Repo ===

We don't trigger an apt update after we install our key and repo, so package installs fail until we manually apt-get update

== TODO ==
----
CategorySystemAdministration