1

<<TableOfContents: execution failed [list index out of range] (see also the log)>>

3. Puppet

As of 2018, HCoop is using Puppet to manage system configuration.

3.1. puppetserver

Puppet git structure (different repos for each): /etc/puppetlabs/puppet, /etc/puppetlabs/code/environments/production (excludes modules), /etc/puppetlabs/code/environments/production/modules/hcoop, /etc/puppetlabs/code/environments/production/modules/hcoop_private. Subject to change.

Git repos structure and tracking of installed modules will be revisited once we need to set up multiple environments. For now, /etc/puppetlabs/code/environments/production/modules/hcoop is where all of our code aside from node definitions lives. /etc/puppetlabs/code/environments/production/modules/hcoop_private is for private data (krb5 host keys, ssl keys, etc.) that needs to be managed by Puppet. Ideally we would use something like wallet for this instead. hcoop_private contains only virtual references to files tagged appropriately so they can be realized on individual servers.

Puppet module structure:

3.1.1. puppetdb

install guide is weird

 puppet resource package puppetdb ensure=latest
 puppet resource package puppetdb-termini ensure=latest
 puppet module install puppetlabs-puppetdb

3.2. Installed Modules

Please Update when installing a new module on the puppetserver.

3.2.1. Specially Installed Module

These are not in puppet forge, but seemed useful enough to deal with manually.

3.3. Style Guidelines

Ideas for keeping consistency among admins. Work in progress.

3.4. Deploying Changes

A bare git repo of the hcoop puppet module is stored in /afs/hcoop.net/user/h/hc/hcoop/private/hcoop-puppet-module.git. Only the hcoop user and members of afs system:administrators have access currently.

A cron runs every few minutes to pull changes and will send an email to all admins if any changes were made.

Each admin has their own personal environment. To deploy, push to the default origin of /afs/hcoop.net/user/h/hc/hcoop/private/hcoop-puppet-module.git, and wait for the cron to run (if changes need to be applied sooner, changes can be pulled into the production environment manually).

3.5. Personal Environments

Each admin will have a puppet environment where changes should be made and tested with puppet agent --test  --noop --environment $user. Before testing with your environment, even using --noop, make sure you have pulled in changes from the central repo. Even --noop tests cause puppetdb rules to be created/deleted, and there is no separate environment support in puppetdb so running with a stale environment could trigger unexpected side effects (lasting until the agent automatically runs again and overwrites any spurious facts with the correct ones from production).

Setting up:

We store personal environments in /srv/puppet/environments so that etckeeper does not persistently alert (causing it to be ignored) if an admin has uncommitted work in their git repo.

mkdir -p /srv/puppet/environments/USER_admin/{modules,manifests}
ln -s /srv/puppet/environments/USER_admin/ /etc/puppetlabs/code/environments/USER_admin

In /etc/puppetlabs/code/environments/USER_admin/:

set up environment.conf (can be copied from another user, it's identical for all user environments)

    modulepath = ./modules:$basemodulepath:../production/modules
    manifest = ../production/manifests

check out modules/hcoop:

    cd modules && git clone /afs/hcoop.net/user/h/hc/hcoop/private/hcoop-puppet-module.git hcoop
    cd hcoop && git submodule init
    chown USER_admin: -R /etc/puppetlabs/code/environments/USER_admin

To checkout, you will need to have afs tokens for a member of system:administrators.

problems:

4. Config Packages

From 2013 until 2018 we were using config-package-dev to manage the configuration of shared services. They proved to be cumbersome to maintain and were abandoned for Puppet.

The following is for historical reference only.

4.1. Rationale

Using config-package-dev has several advantages for HCoop:

4.1.1. Post-Mortem

However, we failed to realize many of these benefits in the end:

The benefit of automated system installs was realized.

4.2. Current Config Packages

We are managing our configuration packages in git, at /afs/hcoop.net/user/h/hc/hcoop/.hcoop-git/debian, or you can view the list of configuration packages using gitweb.

4.3. Creating a Config Package

See DebianPackaging#Creating_a_Configuration_Package for the low-level details of creating a config package and our general packaging workflow with git-buildpackage.

The Debathena documentation describes the new primitives. Primarily, config packages will consist of files installed with dh_install, a few diversions, and some transformations. Use what makes sense: dh_installcron et al are your friends. The hcoop-apache2-config package is a good example.

4.4. Services Changed But Unpackaged


CategorySystemAdministration