How to install a machine that complies with HCoop policies and will function properly as a service or user node.
For a general overview of our architecture, see SystemArchitecture. We have a semi-AutomatedSystemInstall procedure with Debian installer preseeds to instantiate common node types with ease.
Before proceeding with the AutomatedSystemInstall new nodes must be added to HCoop's infrastructure.
8. Network
After deciding on the host name through a poll of the members:
Allocate an addresses from the free list on IpAddresses (and update the page!)
- Using the peer1 request portal, add a reverse dns mapping to the hostname
- You cannot install the machine until the reverse dns mapping has been created; various services rely on the rdns mapping to behave correctly.
Add basic node information to DomTool config
Edit /afs/hcoop.net/common/etc/domtool/lib/hcoop.dtl and add definitions for HOSTNAME_ip, HOSTNAME_private_ip, and HOSTNAME_ipv6
Edit /afs/hcoop.net/user/h/hc/hcoop/.domtool/hcoop.net to add a DNS entry for $HOST.hcoop.net, using HOSTNAME_ip for the A record and HOSTNAME_ipv6 for the AAAA record; and $HOST-private.hcoop.net using HOSTNAME_private_ip.
Apply DomTool configuration (run DOMTOOL_USER=hcoop domtool hcoop.net)
Synchronize DomTool library with source code git repository
9. Documentation
Create a ServerHOST page and add the machine to the Hardware page. KernelVirtualMachines go into a sub-section of their current physical node. Note any relevant information such as the resources available for the node, intended purpose, etc.
Make sure the machine is listed on the IpAddresses page.
After install, update the server notes with any quirks of the install (ideally: none, but reality is a work in progress).
10. Add to Infrastructure
10.1. Kerberos
Add the server key to Kerberos. At the kadmin console ($SERVER is the fully qualified domain name):
add_principal -randkey host/$SERVER@HCOOP.NET
Update create-user to synchronize keytabs to the new node if applicable.
10.2. Puppet
TODO: Create full page on Puppet
Create class hcoop::server::$SERVER and include service classes required for the server (see existing servers for examples).
Add node '$SERVER' { include ::hcoop::server::$SERVER } to manifests/site.pp on master.
After server is installed, set up puppet:
Install https://apt.puppetlabs.com/puppet6-release-buster.deb and then package puppet-agent
Run systemctl stop puppet ; systemctl disable puppet before proceeding so that puppet does not start itself before the system is ready
Request certificate on new server (/opt/puppetlabs/bin/puppet agent --test --onetime --noop --waitforcert 60)
Sign certificate request on puppet master (puppetserver ca sign --certname $server.hcoop.net)
Run puppet agent --test --noop to review initial changes, tweak manifests as needed
Run puppet agent --test to set up the server
TODO: Setup is still in initial stages and it is not quite safe to automatically update servers yet Once setup is confirmed working, enable puppet agent to fetch changes automatically
10.3. Mail
Enable mail routing by adding to exim configuration on the mail server (unless Bug 939 has been fixed, in which case update this documentation with the domtool managed procedure). In the exim config directory:
update-exim4.conf.conf: Add to dc_relay_nets
conf.d/main/01_exim4-config_listmacrosdefs: Add to unix_domains
Run update-exim4.conf
10.4. Portal
Create WebNode for portal according to DaemonAdmin/Portal so that users may request packages, firewall rules, etc.
10.5. Domtool
To control the node with DomTool minimally:
Add to Config.nodeIps
Add to Config.Firewall.firewallNodes if it will have fwtool managed rules (user and web server nodes)
If you are configuring the node for a specific purpose, you'll need to add it to more configuration. See the DomTool documentation where it exists.
Prepare DomTool for deployment: DomTool/Installation.
Set up steps specific to Digital Ocean
11. Create Virtual Machine
- Enable private networking, enable IPv6, disable monitoring agent, enable backups if needed
Set name to fully qualified domain name for server (${hostname}.hcoop.net). Digital Ocean will automatically create PTR records for us in this case (required for most services to work correctly!)
- Login ASAP after provisioning and reset password, store new password in wallet. New password must be at least nine characters long.
We've yet to install a physical node with our new installation procedure. For now see SetupNewMachines.
This document was written with using the GUI virt-manager to deploy a KernelVirtualMachine using our AutomatedSystemInstall. It's probably worth automating this with virsh/virtinst after solidifying things a bit more.
12. Update postinst
Create a post install script in the machine-template. A post install script should extract kerberos keys, install any additional hcoop config packages needed, bootstrap domtool, and install a reasonable set of base software depending on the intended use. This is the install step most in need of improvement.
13. Provision Resources
Create a new VM named after the host, select a network install using the current Debian stable, and add the preseed options specified on AutomatedSystemInstall to the kernel command line. Allocate a reasonable number of maximum cores, RAM, and disk (less than 40G is not recommended, although we are using LVM and can resize disks if needed). Set network using a shared device, currently br0:$HOST (which may be the Wrong Way (tm)).
It does appear that there is a better way than bridged networking: MacVTap in bridged mode works without manual configuration.
14. Debian Install
The installation is mostly automated.
Set the keyboard/language settings (accept the default), and enter the static network information specified on IpAddresses. The preseed will then be fetched and take over.
Create a root password (pwgen -cny with a length >= 9 is sufficient) when prompted, and store it in the password stash.
- Confirm that the partition table should be written
After the first reboot, log in as an admin user and run the post install script to integrate it into our infrastructure. You may have to reboot the machine once more to fully complete the process. Afterward, the machine should be fully functional (in the world of the ideal at least).