<?xml version="1.0" encoding="utf-8"?><!DOCTYPE article  PUBLIC '-//OASIS//DTD DocBook XML V4.4//EN'  'http://www.docbook.org/xml/4.4/docbookx.dtd'><article><articleinfo><title>ServerBusted</title><revhistory><revision><revnumber>19</revnumber><date>2020-07-19 22:32:49</date><authorinitials>ClintonEbadi</authorinitials><revremark>condensed a bit</revremark></revision><revision><revnumber>18</revnumber><date>2020-07-19 22:01:36</date><authorinitials>ClintonEbadi</authorinitials><revremark>fixed triggering apt update before installing packages, but this breaks new things!</revremark></revision><revision><revnumber>17</revnumber><date>2020-07-19 21:56:47</date><authorinitials>ClintonEbadi</authorinitials><revremark>&quot;fixed&quot; ntp port failure by using raw port number</revremark></revision><revision><revnumber>16</revnumber><date>2020-07-19 08:05:27</date><authorinitials>ClintonEbadi</authorinitials><revremark>have to switch to iptables-legacy before doing anything, fixes fail2ban chain problems</revremark></revision><revision><revnumber>15</revnumber><date>2020-07-19 07:10:44</date><authorinitials>ClintonEbadi</authorinitials><revremark>afs builds, still have other issues</revremark></revision><revision><revnumber>14</revnumber><date>2020-03-10 02:47:17</date><authorinitials>ClintonEbadi</authorinitials><revremark>php 5.6 is not actually gone, packages just weren't available in first run</revremark></revision><revision><revnumber>13</revnumber><date>2020-03-10 02:41:36</date><authorinitials>ClintonEbadi</authorinitials><revremark>php 5.6 is not in buster, even in sury</revremark></revision><revision><revnumber>12</revnumber><date>2020-03-10 02:35:52</date><authorinitials>ClintonEbadi</authorinitials><revremark>fail2ban breaks puppet-firewall on buster, cool</revremark></revision><revision><revnumber>11</revnumber><date>2020-03-08 05:11:57</date><authorinitials>ClintonEbadi</authorinitials><revremark>punting on ssmtp for this release cycle</revremark></revision><revision><revnumber>10</revnumber><date>2020-03-08 05:11:09</date><authorinitials>ClintonEbadi</authorinitials><revremark>I think I fixed the openafs initial build failure</revremark></revision><revision><revnumber>9</revnumber><date>2020-03-08 02:48:13</date><authorinitials>ClintonEbadi</authorinitials><revremark>mail is VERY FUN</revremark></revision><revision><revnumber>8</revnumber><date>2020-03-07 23:20:44</date><authorinitials>ClintonEbadi</authorinitials><revremark>remove firewall b0rked section: it was just because we needed to update puppetlabs-firewall to a version with buster support, forces iptables-legacy instead of iptables-nft</revremark></revision><revision><revnumber>7</revnumber><date>2020-03-07 04:46:50</date><authorinitials>ClintonEbadi</authorinitials><revremark>more broken bits</revremark></revision><revision><revnumber>6</revnumber><date>2020-03-07 04:37:26</date><authorinitials>ClintonEbadi</authorinitials></revision><revision><revnumber>5</revnumber><date>2020-03-07 04:34:12</date><authorinitials>ClintonEbadi</authorinitials></revision><revision><revnumber>4</revnumber><date>2020-03-07 04:33:31</date><authorinitials>ClintonEbadi</authorinitials><revremark>kernel was outdated after booting, note so I remember to add this to common steps</revremark></revision><revision><revnumber>3</revnumber><date>2020-03-07 04:23:32</date><authorinitials>ClintonEbadi</authorinitials><revremark>a few hiccups, as expected</revremark></revision><revision><revnumber>2</revnumber><date>2020-03-07 03:52:54</date><authorinitials>ClintonEbadi</authorinitials><revremark>finally realized why we have to add domain hcoop.net resolv.conf before installing</revremark></revision><revision><revnumber>1</revnumber><date>2020-03-07 02:56:17</date><authorinitials>ClintonEbadi</authorinitials><revremark>new buster test server</revremark></revision></revhistory></articleinfo><para>Server <code>busted.hcoop.net</code> is a virtual machine at <ulink url="https://wiki.hcoop.net/ServerBusted/DigitalOcean#">DigitalOcean</ulink> that was created to work on the Debian Stretch to Buster upgrade. </para><para>It's name is just an allusion to it being broken by design. </para><section><title>Setup Notes</title><section><title>Prepare system for puppet</title><para>Newer kernel and some other base packages are available right out of the bat, need to upgrade so we can have working kernel headers for the afs build and whatnot. Added to general setup notes. </para><para>Puppet does not support NFT, even using <code>iptables-nft</code> which is default <code>iptables</code> implementation in Debian 10. Before the first reboot, switch the implementation to <code>iptables-legacy</code>. </para><para>We have to manually install <code>gnupg</code>; the rule we use to force an <code>apt-get update</code> when adding new sources makes it impossible for <code>Apt::Key</code> to install <code>gnupg</code> and verify keys, game over.  </para><screen><![CDATA[echo "domain hcoop.net" >> /etc/resolv.conf
apt update && apt upgrade -y && apt install gnupg -y
update-alternatives  --set iptables /usr/sbin/iptables-legacy
update-alternatives  --set ip6tables /usr/sbin/ip6tables-legacy
wget https://apt.puppetlabs.com/puppet6-release-buster.deb && dpkg -i puppet6-release-buster.deb && apt update && apt install puppet-agent
reboot
# after reboot:
/opt/puppetlabs/bin/puppet agent --test --onetime --noop --waitforcert 15]]></screen></section><section><title>resolv.conf / initial puppet cert request</title><para>We can't really get around manually opening the firewall for the agent on the puppetmaster... at our scale this isn't a big deal anyway. </para><para>Like others, had to set <code>domain hcoop.net</code> manually in <code>/etc/resolv.conf</code>. It looks like the only reason we need this is for the initial puppet connection. So I tried setting the agent config at <code>/etc/puppetlabs/puppet/puppet.conf</code> to: </para><screen><![CDATA[[main]
server = puppet.hcoop.net]]></screen><para>But the cert for the master only has the fqdn of its concrete hostname, and the alias <code>puppet</code> with no domain </para><screen><![CDATA[Error: Server hostname 'puppet.hcoop.net' did not match server certificate; expected one of gibran.hcoop.net, DNS:puppet, DNS:gibran.hcoop.net
Error: Could not run: Server hostname 'puppet.hcoop.net' did not match server certificate; expected one of gibran.hcoop.net, DNS:puppet, DNS:gibran.hcoop.net]]></screen><para>If we could regenerate this to also include <code>CN:puppet.hcoop.net</code>, the manual edit that needed to be done would at least be more related to the limitation in our infrastructure that mandates it... </para></section><section><title>/usr/bin/mail behaves differently</title><para>GNU <code>mailutils</code> now provides <code>/usr/bin/mail</code> instead of <code>bsd-mailx</code>. It treats addresses a bit differently, appending the hostname. So <code>mail -s &quot;foo&quot; root</code> goes to <code>root@busted.hcoop.net</code> instead of just <code>root</code> which is then rewritten to <code>logs@hcoop.net</code>. The message then gets stuck in exim forever until it gets frozen and purged. </para><para>Not sure we want to switch back to <code>bsd-mailx</code> over this though, for now keep <code>mailutils</code> as the default provider. </para></section></section><section><title>Puppet porting notes</title><section><title>HCoop Debian Package Repo</title><para>After adding logic to hcoop::service::apt to force an `apt-get update' after any sources change before installing packages: </para><screen><![CDATA[Apt::Source <| |> ~> Class['apt::update'] -> Package <| |>]]></screen><para>A new problem arose: <code>Error: Could not find a suitable provider for apt_key</code>, because the <code>apt::key</code> type needs <code>Package['gnupg']</code> to verify keys in order to add the repository, and it can't install it because it can't verify the keys of the custom repos...  </para><para>Worked around by installing, not sure of general fix ... might be able to force installation of gnupg before <code>Apt::Key</code> in the <code>hcoop::service::apt</code> class ... but I think that might create a dependency cycle. </para></section><section><title>libnss-afs installs to non-multiarch location</title><para>minor issue, but might want to address. We're still installing to just /usr/lib instead of /usr/lib/x86_64-linux-gnu/ (need to update package to comply with multiarch) </para></section></section><section><title>Puppet WONTFIX</title><section><title>ssmtp is gone</title><para><emphasis role="strikethrough">We need to switch to msmtp</emphasis> </para><para>Switching to msmtp proved to be difficult, lowuid rewriting to send mails to <code>logs@</code> alias is not working, and can't work as far as I can tell. I ended up just backporting ssmtp since it's not removed from Debian, but just didn't get moved into buster (it's also just a <emphasis>bit</emphasis> unmaintained). It might be easier to just set up exim in satellite mode going forward instead. </para></section></section><section><title>TODO</title><!--rule (<hr>) is not applicable to DocBook--><para> <ulink url="https://wiki.hcoop.net/ServerBusted/CategorySystemAdministration#">CategorySystemAdministration</ulink> </para></section></article>