welcome: please sign in

Diff for "ConfigurationManagement"

Differences between revisions 2 and 19 (spanning 17 versions)
Revision 2 as of 2013-05-30 17:31:07
Size: 1722
Editor: ClintonEbadi
Comment: I thought I documented this, try and get some kind of start now
Revision 19 as of 2019-04-20 20:43:49
Size: 6852
Editor: ClintonEbadi
Comment: having second thoughts about firewall_multi
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
We are using [[http://debathena.mit.edu/config-packages/|config-package-dev]] to manage the configuration of shared services. <<TableOfContents>>
Line 3: Line 3:
== Rationale == == Puppet ==

As of 2018, HCoop is using Puppet to manage system configuration.

=== puppetserver ===

 * Hosted on ServerGibran
 * Installed https://apt.puppetlabs.com/puppet6-release-stretch.deb manually
 * Packages: puppetserver, puppet-agent

Puppet git structure (different repos for each): /etc/puppetlabs/puppet, /etc/puppetlabs/code/environments/production (excludes modules), /etc/puppetlabs/code/environments/production/modules/hcoop, /etc/puppetlabs/code/environments/production/modules/hcoop_private. Subject to change.

Git repos structure and tracking of installed modules will be revisited once we need to set up multiple environments. For now, `/etc/puppetlabs/code/environments/production/modules/hcoop` is where all of our code aside from node definitions lives. `/etc/puppetlabs/code/environments/production/modules/hcoop_private` is for private data (krb5 host keys, ssl keys, etc.) that needs to be managed by Puppet. Ideally we would use something like [[https://www.eyrie.org/~eagle/software/wallet/|wallet]] for this instead. hcoop_private contains only virtual references to files tagged appropriately so they can be realized on individual servers.

Puppet module structure:

 * hcoop
   * server
     * $server (e.g. gibran)
   * service
     * openafs-client

==== puppetdb ====

install guide is weird

{{{
 puppet resource package puppetdb ensure=latest
 puppet resource package puppetdb-termini ensure=latest
 puppet module install puppetlabs-puppetdb
}}}

=== Installed Modules ===

Please Update when installing a new module on the puppetserver.

 * puppetlabs-firewall
 * puppetlabs-puppetdb
 * alexharvey-firewall_multi (says incompatible, but works... enough).
 * stm-resolv_conf
 * ccin2p3-mit_krb5
 * stm-debconf
 * saz-sudo
 * herculesteam-augeasproviders_pam
 * herculesteam-augeasproviders_core
 * saz-timezone
 * dalen-dnsquery
 * camptocamp-systemd
 * puppetlabs-apache
 * puppet-logrotate
 * puppetlabs-stunnel
 * puppetlabs-mysql
 * puppetlabs-tagmail
 * puppetlabs-mailalias_core

==== Specially Installed Module ====

These are not in puppet forge, but seemed useful enough to deal with manually.

 * puppet-posix_acl from https://github.com/voxpupuli/puppet-posix_acl

=== Style Guidelines ===

Ideas for keeping consistency among admins. Work in progress.

 * Use firewall_multi for all rules unless it really is ipv4 or ipv6 only, provider is set in defaults and will keep ipv4 and ipv6 firewall in sync
   * In retrospect this was probably bad and we might want to instead create our own defined type to wrap firewall_multi
 * Should pass puppet-lint (enforce using git hook) / respect https://puppet.com/docs/puppet/5.5/style_guide.html
 * Inheritance is discouraged? Avoiding it for now
 * Files controlled by puppet have comment "This file is managed by Puppet. DO NOT EDIT." somewhere near the top
 * Some structure to firewall rule numbers
  * Under 100 for core system things that need to go near the beginning
  * Over 900 for core system things that need to go near the end (e.g. jumping to fwtool output chains)
 * Any root crontabs should have `environment => 'MAILTO=log+SERVICE@hcoop.net'` (where `SERVICE` is the service the cron is related to). This ensures that mail won't go to the wrong address if another job sets `MAILTO` earlier in the crontab.

=== Personal Environments ===

Each admin will have a [[https://puppet.com/docs/puppet/5.5/environments_about.html|puppet environment]] where changes should be made and tested with `puppet agent --test --noop --environment $user`.

''incomplete.''

`mkdir -p /etc/puppetlabs/code/environments/clinton_admin/{modules,manifests}`

set up `environment.conf`
{{{
    modulepath = ./modules:$basemodulepath:../production/modules
    manifest = ../production/manifests
}}}

check out modules/hcoop:
{{{
    cd $user/modules && git clone /etc/puppetlabs/code/environments/production/modules/hcoop
    cd hcoop && git submodule init
    chown $user: -R /etc/puppetlabs/code/environments/$user
}}}

problems:

 * no tracking of site.pp or top level hiera files
 * main git repo is still in production env

== Config Packages ==

From 2013 until 2018 we were using [[http://debathena.mit.edu/config-packages/|config-package-dev]] to manage the configuration of shared services. They proved to be cumbersome to maintain and were abandoned for Puppet.

The following is for historical reference only.

=== Rationale ===
Line 13: Line 120:
A system like Puppet might make sense in the future. ==== Post-Mortem ====
Line 15: Line 122:
== Current Config Packages == However, we failed to realize many of these benefits in the end:

 * Basic Debian packaging was not quite as straightforward to pick up or remember as it seemed initially
   * An extra entry barrier for new admins by using a config management system not widely used by others
 * Packages were cumbersome to update, and resulted in a trail of underdocumented and unpackaged changes
 * Packages were cumbersome to port between Debian releases
 * Distributing configuration with apt was time consuming
   * Packages must be released, built, uploaded, and installed which could take upward of an hour, compared to Puppet that runs an agent and fetches new updates within minutes

The benefit of automated system installs was realized.

=== Current Config Packages ===
Line 19: Line 137:
== Creating a Config Package == === Creating a Config Package ===
Line 26: Line 144:
== Services Changed But Unpackaged == === Services Changed But Unpackaged ===

1. Puppet

As of 2018, HCoop is using Puppet to manage system configuration.

1.1. puppetserver

Puppet git structure (different repos for each): /etc/puppetlabs/puppet, /etc/puppetlabs/code/environments/production (excludes modules), /etc/puppetlabs/code/environments/production/modules/hcoop, /etc/puppetlabs/code/environments/production/modules/hcoop_private. Subject to change.

Git repos structure and tracking of installed modules will be revisited once we need to set up multiple environments. For now, /etc/puppetlabs/code/environments/production/modules/hcoop is where all of our code aside from node definitions lives. /etc/puppetlabs/code/environments/production/modules/hcoop_private is for private data (krb5 host keys, ssl keys, etc.) that needs to be managed by Puppet. Ideally we would use something like wallet for this instead. hcoop_private contains only virtual references to files tagged appropriately so they can be realized on individual servers.

Puppet module structure:

  • hcoop
    • server
      • $server (e.g. gibran)
    • service
      • openafs-client

1.1.1. puppetdb

install guide is weird

 puppet resource package puppetdb ensure=latest
 puppet resource package puppetdb-termini ensure=latest
 puppet module install puppetlabs-puppetdb

1.2. Installed Modules

Please Update when installing a new module on the puppetserver.

  • puppetlabs-firewall
  • puppetlabs-puppetdb
  • alexharvey-firewall_multi (says incompatible, but works... enough).
  • stm-resolv_conf
  • ccin2p3-mit_krb5
  • stm-debconf
  • saz-sudo
  • herculesteam-augeasproviders_pam
  • herculesteam-augeasproviders_core
  • saz-timezone
  • dalen-dnsquery
  • camptocamp-systemd
  • puppetlabs-apache
  • puppet-logrotate
  • puppetlabs-stunnel
  • puppetlabs-mysql
  • puppetlabs-tagmail
  • puppetlabs-mailalias_core

1.2.1. Specially Installed Module

These are not in puppet forge, but seemed useful enough to deal with manually.

1.3. Style Guidelines

Ideas for keeping consistency among admins. Work in progress.

  • Use firewall_multi for all rules unless it really is ipv4 or ipv6 only, provider is set in defaults and will keep ipv4 and ipv6 firewall in sync
    • In retrospect this was probably bad and we might want to instead create our own defined type to wrap firewall_multi
  • Should pass puppet-lint (enforce using git hook) / respect https://puppet.com/docs/puppet/5.5/style_guide.html

  • Inheritance is discouraged? Avoiding it for now
  • Files controlled by puppet have comment "This file is managed by Puppet. DO NOT EDIT." somewhere near the top
  • Some structure to firewall rule numbers
    • Under 100 for core system things that need to go near the beginning
    • Over 900 for core system things that need to go near the end (e.g. jumping to fwtool output chains)
  • Any root crontabs should have environment => 'MAILTO=log+SERVICE@hcoop.net' (where SERVICE is the service the cron is related to). This ensures that mail won't go to the wrong address if another job sets MAILTO earlier in the crontab.

1.4. Personal Environments

Each admin will have a puppet environment where changes should be made and tested with puppet agent --test  --noop --environment $user.

incomplete.

mkdir -p /etc/puppetlabs/code/environments/clinton_admin/{modules,manifests}

set up environment.conf

    modulepath = ./modules:$basemodulepath:../production/modules
    manifest = ../production/manifests

check out modules/hcoop:

    cd $user/modules && git clone /etc/puppetlabs/code/environments/production/modules/hcoop
    cd hcoop && git submodule init
    chown $user: -R /etc/puppetlabs/code/environments/$user

problems:

  • no tracking of site.pp or top level hiera files
  • main git repo is still in production env

2. Config Packages

From 2013 until 2018 we were using config-package-dev to manage the configuration of shared services. They proved to be cumbersome to maintain and were abandoned for Puppet.

The following is for historical reference only.

2.1. Rationale

Using config-package-dev has several advantages for HCoop:

  • They can be used to automate installation

  • Our changes to config are kept in an executable format
  • Basic Debian packaging is straightforward to pick up
  • Distribution of configuration occurs through the same channel as our general software updates
  • Everyone is forced to formalize their changes rather than leaving a trail of undocumented changes

2.1.1. Post-Mortem

However, we failed to realize many of these benefits in the end:

  • Basic Debian packaging was not quite as straightforward to pick up or remember as it seemed initially
    • An extra entry barrier for new admins by using a config management system not widely used by others
  • Packages were cumbersome to update, and resulted in a trail of underdocumented and unpackaged changes
  • Packages were cumbersome to port between Debian releases
  • Distributing configuration with apt was time consuming
    • Packages must be released, built, uploaded, and installed which could take upward of an hour, compared to Puppet that runs an agent and fetches new updates within minutes

The benefit of automated system installs was realized.

2.2. Current Config Packages

We are managing our configuration packages in git, at /afs/hcoop.net/user/h/hc/hcoop/.hcoop-git/debian, or you can view the list of configuration packages using gitweb.

2.3. Creating a Config Package

See DebianPackaging#Creating_a_Configuration_Package for the low-level details of creating a config package and our general packaging workflow with git-buildpackage.

The Debathena documentation describes the new primitives. Primarily, config packages will consist of files installed with dh_install, a few diversions, and some transformations. Use what makes sense: dh_installcron et al are your friends. The hcoop-apache2-config package is a good example.

2.4. Services Changed But Unpackaged

  • php5 config (changed a few variables, disabled some suhosin stuff at least).


CategorySystemAdministration

ConfigurationManagement (last edited 2023-10-13 13:57:26 by ClintonEbadi)