welcome: please sign in

The following 313 words could not be found in the dictionary of 7 words (including 7 LocalSpellingWords) and are highlighted below:
able   add   Added   adding   address   addresses   Administration   afs   After   after   agent   alias   allusion   also   alternatives   an   and   any   anyway   appending   Apt   apt   are   arose   around   as   at   available   back   backporting   base   bat   be   because   before   Before   behaves   being   big   bin   bit   broken   bsd   build   busted   buster   Buster   But   but   by   can   Category   cert   certificate   change   Class   class   com   comply   concrete   conf   config   connection   Coop   Could   could   create   created   custom   cycle   deal   deb   default   dependency   design   did   didn   differently   difficult   Digital   does   domain   done   dpkg   easier   echo   edit   ended   Error   etc   even   exim   expected   far   find   firewall   first   fix   foo   for   force   forever   forward   fqdn   from   frozen   game   general   get   gets   gibran   gnu   gnupg   goes   going   gone   had   has   have   hcoop   headers   hostname   https   If   implementation   impossible   in   include   infrastructure   initial   install   installation   installing   installs   instead   into   ip6tables   iptables   is   isn   issue   it   It   its   just   keep   kernel   Key   key   keys   least   legacy   lib   libnss   Like   like   limitation   linux   location   logic   logs   looks   lowuid   machine   mail   mails   mailutils   mailx   main   makes   mandates   manual   manually   master   match   message   might   minor   mode   more   moved   msmtp   multiarch   name   need   needed   needs   net   new   Newer   nft   no   non   noop   not   Not   notes   Notes   now   Ocean   of   on   one   onetime   only   opening   opt   order   other   others   our   out   over   package   Package   packages   porting   Prepare   problem   proved   provider   provides   puppet   Puppet   puppet6   puppetlabs   puppetmaster   purged   re   really   reason   reboot   regenerate   related   release   removed   Repo   repos   repository   request   resolv   rewriting   rewritten   right   root   rule   run   satellite   sbin   scale   send   server   Server   service   set   setting   Setup   setup   since   So   so   some   Source   sources   ssmtp   still   Stretch   stuck   suitable   support   sure   switch   Switching   System   system   tell   test   that   The   the   then   think   this   though   to   treats   tried   type   unmaintained   until   up   update   upgrade   use   using   usr   verify   virtual   waitforcert   want   was   We   we   wget   whatnot   when   which   with   work   Worked   working   would   x86  

Clear message
Edit

ServerBusted

Server busted.hcoop.net is a virtual machine at DigitalOcean that was created to work on the Debian Stretch to Buster upgrade.

It's name is just an allusion to it being broken by design.

1. Setup Notes

1.1. Prepare system for puppet

Newer kernel and some other base packages are available right out of the bat, need to upgrade so we can have working kernel headers for the afs build and whatnot. Added to general setup notes.

Puppet does not support NFT, even using iptables-nft which is default iptables implementation in Debian 10. Before the first reboot, switch the implementation to iptables-legacy.

We have to manually install gnupg; the rule we use to force an apt-get update when adding new sources makes it impossible for Apt::Key to install gnupg and verify keys, game over. 

echo "domain hcoop.net" >> /etc/resolv.conf
apt update && apt upgrade -y && apt install gnupg -y
update-alternatives  --set iptables /usr/sbin/iptables-legacy
update-alternatives  --set ip6tables /usr/sbin/ip6tables-legacy
wget https://apt.puppetlabs.com/puppet6-release-buster.deb && dpkg -i puppet6-release-buster.deb && apt update && apt install puppet-agent
reboot
# after reboot:
/opt/puppetlabs/bin/puppet agent --test --onetime --noop --waitforcert 15

1.2. resolv.conf / initial puppet cert request

We can't really get around manually opening the firewall for the agent on the puppetmaster... at our scale this isn't a big deal anyway.

Like others, had to set domain hcoop.net manually in /etc/resolv.conf. It looks like the only reason we need this is for the initial puppet connection. So I tried setting the agent config at /etc/puppetlabs/puppet/puppet.conf to: 

[main]
server = puppet.hcoop.net

But the cert for the master only has the fqdn of its concrete hostname, and the alias puppet with no domain 

Error: Server hostname 'puppet.hcoop.net' did not match server certificate; expected one of gibran.hcoop.net, DNS:puppet, DNS:gibran.hcoop.net
Error: Could not run: Server hostname 'puppet.hcoop.net' did not match server certificate; expected one of gibran.hcoop.net, DNS:puppet, DNS:gibran.hcoop.net

If we could regenerate this to also include CN:puppet.hcoop.net, the manual edit that needed to be done would at least be more related to the limitation in our infrastructure that mandates it...

1.3. /usr/bin/mail behaves differently

GNU mailutils now provides /usr/bin/mail instead of bsd-mailx. It treats addresses a bit differently, appending the hostname. So mail -s "foo" root goes to root@busted.hcoop.net instead of just root which is then rewritten to logs@hcoop.net. The message then gets stuck in exim forever until it gets frozen and purged.

Not sure we want to switch back to bsd-mailx over this though, for now keep mailutils as the default provider.

2. Puppet porting notes

2.1. HCoop Debian Package Repo

After adding logic to hcoop::service::apt to force an `apt-get update' after any sources change before installing packages: 

Apt::Source <| |> ~> Class['apt::update'] -> Package <| |>

A new problem arose: Error: Could not find a suitable provider for apt_key, because the apt::key type needs Package['gnupg'] to verify keys in order to add the repository, and it can't install it because it can't verify the keys of the custom repos...

Worked around by installing, not sure of general fix ... might be able to force installation of gnupg before Apt::Key in the hcoop::service::apt class ... but I think that might create a dependency cycle.

2.2. libnss-afs installs to non-multiarch location

minor issue, but might want to address. We're still installing to just /usr/lib instead of /usr/lib/x86_64-linux-gnu/ (need to update package to comply with multiarch)

3. Puppet WONTFIX

3.1. ssmtp is gone

We need to switch to msmtp

Switching to msmtp proved to be difficult, lowuid rewriting to send mails to logs@ alias is not working, and can't work as far as I can tell. I ended up just backporting ssmtp since it's not removed from Debian, but just didn't get moved into buster (it's also just a bit unmaintained). It might be easier to just set up exim in satellite mode going forward instead.

4. TODO

CategorySystemAdministration

ServerBusted (last edited 2020-07-19 22:32:49 by ClintonEbadi)