282023-10-13 13:57:26ClintonEbadiinstalled sysctl module to manage keyring size272022-03-05 20:20:07ClintonEbadiminor fixes to procedure for creating admin env262021-04-30 02:49:50ClintonEbadimove admin puppet envs to /srv252020-12-02 03:41:09ClintonEbadi242020-12-02 03:11:22ClintonEbadi232020-12-02 03:05:35ClintonEbadinew puppet modules for jitsi meet222019-04-20 21:19:57ClintonEbadi212019-04-20 21:02:38ClintonEbadi202019-04-20 20:51:17ClintonEbadicleaned up puppet deploys192019-04-20 20:43:49ClintonEbadihaving second thoughts about firewall_multi182018-11-11 04:09:25ClintonEbadiroot cron jobs must set MAILTO172018-10-26 01:51:06ClintonEbadiinstalled puppetlabs-mailalias_core module162018-10-16 04:00:28ClintonEbadiwe're using puppet 6 now152018-10-14 00:42:14ClintonEbadiinstalled puppetlabs-tagmail module142018-10-12 21:12:22ClintonEbadiinstalled puppetlabs-mysql module132018-10-11 23:23:24ClintonEbadiinstalled puppetlabs-stunnel module122018-07-08 03:38:33ClintonEbadiinstalled puppet-posix_acl112018-07-07 02:36:30ClintonEbadiinstalled puppet-logrotate module102018-07-05 02:23:42ClintonEbadiinstalled puppetlabs-apache module92018-05-05 22:26:39ClintonEbadiscratch notes on personal puppet envs82018-04-24 02:27:17ClintonEbadipuppet module camptocamp-systemd72018-04-22 02:15:45ClintonEbadi62018-04-22 02:12:42ClintonEbadistart date for config packages52018-04-22 02:11:42ClintonEbadi42018-04-22 02:11:21ClintonEbadinotes on difficulties encountered with config packages32018-04-22 02:04:39ClintonEbadiinitial notes on new puppet managed setup22013-05-30 17:31:07ClintonEbadiI thought I documented this, try and get some kind of start now12013-01-05 06:27:37ClintonEbadibasic stub

As of 2018, HCoop is using Puppet to manage system configuration.

Hosted on ServerGibran Installed manually Packages: puppetserver, puppet-agent Puppet git structure (different repos for each): /etc/puppetlabs/puppet, /etc/puppetlabs/code/environments/production (excludes modules), /etc/puppetlabs/code/environments/production/modules/hcoop, /etc/puppetlabs/code/environments/production/modules/hcoop_private. Subject to change. Git repos structure and tracking of installed modules will be revisited once we need to set up multiple environments. For now, /etc/puppetlabs/code/environments/production/modules/hcoop is where all of our code aside from node definitions lives. /etc/puppetlabs/code/environments/production/modules/hcoop_private is for private data (krb5 host keys, ssl keys, etc.) that needs to be managed by Puppet. Ideally we would use something like wallet for this instead. hcoop_private contains only virtual references to files tagged appropriately so they can be realized on individual servers. Puppet module structure: hcoop server $server (e.g. gibran) service openafs-client

install guide is weird

Please Update when installing a new module on the puppetserver. puppetlabs-firewall puppetlabs-puppetdb alexharvey-firewall_multi (says incompatible, but works... enough). stm-resolv_conf ccin2p3-mit_krb5 stm-debconf saz-sudo herculesteam-augeasproviders_pam herculesteam-augeasproviders_core saz-timezone dalen-dnsquery camptocamp-systemd puppetlabs-apache puppet-logrotate puppetlabs-stunnel puppetlabs-mysql puppetlabs-tagmail puppetlabs-mailalias_core smash-jitsimeet puppet-prosody (for smash-jitsimeet) puppet-nginx (for smash-jitsimeet) thias-sysctl

These are not in puppet forge, but seemed useful enough to deal with manually. puppet-posix_acl from

Ideas for keeping consistency among admins. Work in progress. Use firewall_multi for all rules unless it really is ipv4 or ipv6 only, provider is set in defaults and will keep ipv4 and ipv6 firewall in sync In retrospect this was probably bad and we might want to instead create our own defined type to wrap firewall_multi Should pass puppet-lint (enforce using git hook) / respect Inheritance is discouraged? Avoiding it for now Files controlled by puppet have comment "This file is managed by Puppet. DO NOT EDIT." somewhere near the top Some structure to firewall rule numbers Under 100 for core system things that need to go near the beginning Over 900 for core system things that need to go near the end (e.g. jumping to fwtool output chains) Any root crontabs should have environment => 'MAILTO=log+SERVICE@hcoop.net' (where SERVICE is the service the cron is related to). This ensures that mail won't go to the wrong address if another job sets MAILTO earlier in the crontab.

A bare git repo of the hcoop puppet module is stored in /afs/hcoop.net/user/h/hc/hcoop/private/hcoop-puppet-module.git. Only the hcoop user and members of afs system:administrators have access currently. A cron runs every few minutes to pull changes and will send an email to all admins if any changes were made. Each admin has their own personal environment. To deploy, push to the default origin of /afs/hcoop.net/user/h/hc/hcoop/private/hcoop-puppet-module.git, and wait for the cron to run (if changes need to be applied sooner, changes can be pulled into the production environment manually).

Each admin will have a puppet environment where changes should be made and tested with puppet agent --test --noop --environment $user. Before testing with your environment, even using --noop, make sure you have pulled in changes from the central repo. Even --noop tests cause puppetdb rules to be created/deleted, and there is no separate environment support in puppetdb so running with a stale environment could trigger unexpected side effects (lasting until the agent automatically runs again and overwrites any spurious facts with the correct ones from production). Setting up: We store personal environments in /srv/puppet/environments so that etckeeper does not persistently alert (causing it to be ignored) if an admin has uncommitted work in their git repo. In /etc/puppetlabs/code/environments/USER_admin/: set up environment.conf (can be copied from another user, it's identical for all user environments) check out modules/hcoop: To checkout, you will need to have afs tokens for a member of system:administrators. problems: no tracking of site.pp or top level hiera files

From 2013 until 2018 we were using config-package-dev to manage the configuration of shared services. They proved to be cumbersome to maintain and were abandoned for Puppet. The following is for historical reference only.

Using config-package-dev has several advantages for HCoop: They can be used to automate installation Our changes to config are kept in an executable format Basic Debian packaging is straightforward to pick up Distribution of configuration occurs through the same channel as our general software updates Everyone is forced to formalize their changes rather than leaving a trail of undocumented changes

However, we failed to realize many of these benefits in the end: Basic Debian packaging was not quite as straightforward to pick up or remember as it seemed initially An extra entry barrier for new admins by using a config management system not widely used by others Packages were cumbersome to update, and resulted in a trail of underdocumented and unpackaged changes Packages were cumbersome to port between Debian releases Distributing configuration with apt was time consuming Packages must be released, built, uploaded, and installed which could take upward of an hour, compared to Puppet that runs an agent and fetches new updates within minutes The benefit of automated system installs was realized.

We are managing our configuration packages in git, at /afs/hcoop.net/user/h/hc/hcoop/.hcoop-git/debian, or you can view the list of configuration packages using gitweb.

See DebianPackaging#Creating_a_Configuration_Package for the low-level details of creating a config package and our general packaging workflow with git-buildpackage. The Debathena documentation describes the new primitives. Primarily, config packages will consist of files installed with dh_install, a few diversions, and some transformations. Use what makes sense: dh_installcron et al are your friends. The hcoop-apache2-config package is a good example.

php5 config (changed a few variables, disabled some suhosin stuff at least). CategorySystemAdministration