Before proceeding with the AutomatedSystemInstall new nodes must be added to HCoop's infrastructure.

1. Network

After deciding on the host name through a poll of the members:

1. Allocate an addresses from the free list on IpAddresses (and update the page!)

2. Using the peer1 request portal, add a reverse dns mapping to the hostname
   • You cannot install the machine until the reverse dns mapping has been created; various services rely on the rdns mapping to behave correctly.

3. Add basic node information to DomTool config

   1. Edit /afs/hcoop.net/common/etc/domtool/lib/hcoop.dtl and add definitions for HOSTNAME_ip, HOSTNAME_private_ip, and HOSTNAME_ipv6

   2. Edit /afs/hcoop.net/user/h/hc/hcoop/.domtool/hcoop.net to add a DNS entry for $HOST.hcoop.net, using HOSTNAME_ip for the A record and HOSTNAME_ipv6 for the AAAA record; and $HOST-private.hcoop.net using HOSTNAME_private_ip.

   3. Apply DomTool configuration (run DOMTOOL_USER=hcoop domtool hcoop.net)

   4. Synchronize DomTool library with source code git repository

2. Documentation

Create a ServerHOST page and add the machine to the Hardware page. KernelVirtualMachines go into a sub-section of their current physical node. Note any relevant information such as the resources available for the node, intended purpose, etc.

Make sure the machine is listed on the IpAddresses page.

After install, update the server notes with any quirks of the install (ideally: none, but reality is a work in progress).

3. Add to Infrastructure

3.1. Kerberos

Add the server key to Kerberos. At the kadmin console ($SERVER is the fully qualified domain name):

add_principal -randkey host/$SERVER@HCOOP.NET

Update create-user to synchronize keytabs to the new node if applicable.

3.2. Puppet

TODO: Create full page on Puppet

Create class hcoop::server::$SERVER and include service classes required for the server (see existing servers for examples).

Add node '$SERVER' { include ::hcoop::server::$SERVER } to manifests/site.pp on master.

After server is installed, set up puppet:

• Install https://apt.puppetlabs.com/puppet6-release-buster.deb and then package puppet-agent

• Run systemctl stop puppet ; systemctl disable puppet before proceeding so that puppet does not start itself before the system is ready

• Request certificate on new server (/opt/puppetlabs/bin/puppet agent --test --onetime --noop --waitforcert 60)

• Sign certificate request on puppet master (puppetserver ca sign --certname $server.hcoop.net)

• Run puppet agent --test --noop to review initial changes, tweak manifests as needed

• Run puppet agent --test to set up the server

• TODO: Setup is still in initial stages and it is not quite safe to automatically update servers yet Once setup is confirmed working, enable puppet agent to fetch changes automatically

3.3. Mail

Enable mail routing by adding to exim configuration on the mail server (unless Bug 939 has been fixed, in which case update this documentation with the domtool managed procedure). In the exim config directory:

• update-exim4.conf.conf: Add to dc_relay_nets

• conf.d/main/01_exim4-config_listmacrosdefs: Add to unix_domains

• Run update-exim4.conf

3.4. Portal

Create WebNode for portal according to DaemonAdmin/Portal so that users may request packages, firewall rules, etc.

3.5. Domtool

To control the node with DomTool minimally:

• Add to Config.nodeIps

• Add to Config.Firewall.firewallNodes if it will have fwtool managed rules (user and web server nodes)

If you are configuring the node for a specific purpose, you'll need to add it to more configuration. See the DomTool documentation where it exists.

Prepare DomTool for deployment: DomTool/Installation.

CategorySystemAdministration

Set up steps specific to Digital Ocean

4. Create Virtual Machine

• Enable private networking, enable IPv6, disable monitoring agent, enable backups if needed

• Set name to fully qualified domain name for server (${hostname}.hcoop.net). Digital Ocean will automatically create PTR records for us in this case (required for most services to work correctly!)

• Login ASAP after provisioning and reset password, store new password in wallet. New password must be at least nine characters long.

CategorySystemAdministration

We've yet to install a physical node with our new installation procedure. For now see SetupNewMachines.

CategorySystemAdministration

This document was written with using the GUI virt-manager to deploy a KernelVirtualMachine using our AutomatedSystemInstall. It's probably worth automating this with virsh/virtinst after solidifying things a bit more.

5. Update postinst

Create a post install script in the machine-template. A post install script should extract kerberos keys, install any additional hcoop config packages needed, bootstrap domtool, and install a reasonable set of base software depending on the intended use. This is the install step most in need of improvement.

6. Provision Resources

Create a new VM named after the host, select a network install using the current Debian stable, and add the preseed options specified on AutomatedSystemInstall to the kernel command line. Allocate a reasonable number of maximum cores, RAM, and disk (less than 40G is not recommended, although we are using LVM and can resize disks if needed). Set network using a shared device, currently br0:$HOST (which may be the Wrong Way (tm)).

It does appear that there is a better way than bridged networking: MacVTap in bridged mode works without manual configuration.

7. Debian Install

The installation is mostly automated.

• Set the keyboard/language settings (accept the default), and enter the static network information specified on IpAddresses. The preseed will then be fetched and take over.

• Create a root password (pwgen -cny with a length >= 9 is sufficient) when prompted, and store it in the password stash.

• Confirm that the partition table should be written

After the first reboot, log in as an admin user and run the post install script to integrate it into our infrastructure. You may have to reboot the machine once more to fully complete the process. Afterward, the machine should be fully functional (in the world of the ideal at least).

CategorySystemAdministration