#pragma section-numbers off

These steps are listed in approximately the order in which they should be performed; please try to maintain that as you add to it.

{{{#!wiki caution
See InstallationProcedure for an up to date guide to installing a new HCoop node. This document is (in December 2012) a semi-accurate representation of what the preseeded installer is doing, but keep the ''semi-'' part in mind. The page is being kept for historical reference.
}}}

<<TableOfContents>>


= List the Machine on the Wiki =

The hostname of the machine should be decided through a Members Poll (accessible from members portal) such as [[https://members.hcoop.net/portal/poll?id=31]].

Add the machine to the [[Hardware]] page.

It is a very good idea to photograph the front and back panels of the machine and put those images on the wiki page; that way remote admins and people in the data center can be sure they're talking about the same ports.

Add the machine to the [[IpAddresses]] page.

= Set Up Out Of Band Access =

All machines owned by hcoop should, if possible, have some [[http://en.wikipedia.org/wiki/Out-of-band_infrastructure|out-of-band mechanism]] for:

 1. Keyboard access
 2. Screen access
 3. Power-cycling

Functions 1+2 are typically provided by {{{kvm.hcoop.net}}} (see KvmAccess); assuming you plan on going with that, you should connect the server's keyboard and video to the kvm switch.

Each server has its own solution for 3, usually in the form of a "service processor".  You should investigate and document the appropriate service processor settings.  If the service processor requires its own IP address, you should name it {{{foo-sp.hcoop.net}}} where {{{foo.hcoop.net}}} is the name of the server.

If there's _anything_ server-specific, please add an entry under "Specific Machines" on page AdminArea and document what it is. Rebooting procedures are an ideal candidate for this.

= Add a DNS entry for the server =

This is done as follows:

 1. Edit `/afs/hcoop.net/common/etc/domtool/lib/hcoop.dtl` and add definition for "HOSTNAME_ip" (search for "deleuze_ip" and just copy the line to new name)
 1. Edit `/afs/hcoop.net/user/h/hc/hcoop/.domtool/hcoop.net` to add the new DNS entry, using HOSTNAME_ip (again, can use deleuze_ip as example)
 1. To apply DomTool configuration, run `DOMTOOL_USER=hcoop domtool hcoop.net` in the ~hcoop/.domtool/ directory
 1. Using the peer1 request portal, add a reverse dns mapping to the hostname

= Install Debian =

{{{#!wiki caution
This section is obsolete.
}}}

We use Debian GNU.

Here are the installation notes to help you:

 1. Find Debian stable image (whichever is 'stable' at time of installation -- this documentation is written for Squeeze)
 1. Prepare a USB stick to boot from (can do it manually or with convenient tool called "unetbootin")
 1. In system BIOS, choose 'Auto-power on on power restore' (if there is such option), and see if you can make USB stick to not be the first disk (when it's the first disk, it gets assigned device name /dev/sda and makes the installation a tiny bit harder)
 1. See which network card is in the server, if it requires non-free firmware, the package needs to be manually copied from Debian's non-free repository onto the install media (example is package "firmware-bnx2" for Broadcom NetXtremeII cards (http://packages.debian.org/sid/all/firmware-bnx2/download)). Once package is on the media, the install procedure will, if it is needed, automatically find and install it
 1. For timezone, use timezone where the server is physically located, and answer Yes to "Is the hardware clock set to GMT?"
 1. Choose manual network configuration, specifying the choosen hostname, IP and network details as listed on the IpAddresses page
 1. Partition disks. Most often, this comes town to creating identical partitions on all disks that are part of RAID1, and creating RAID arrays as in the example that follows. Currently, we usually configure: all remaining free space on `/`, `/boot` (300M), `/var/cache/openafs` (set to 5 or more GB), `/tmp` (1G), and installer suggested amount of swap.  {{{

Example: 2x 160 GB system disks

System disk 1:

sda1: primary, beginning, 1 GB, ext3, /boot
sda2: primary, beginning, 8 GB, use as phys. volume for RAID (swap space: 1 GB x number of proc. cores)
sda3: primary, beginning, all available space, use as phys. volume for RAID

System disk 2:

sdb1: primary, beginning, 1 GB, ext3, unmounted
sdb2: primary, beginning, 8 GB, use as phys. volume for RAID (swap, same size as above)
sdb3: primary, beginning, all available space, use as phys. volume for RAID

Then, after RAID partitions have been assigned, new option "Configure RAID"
will appear at the top of the partitioning menu. We add the two devices in
RAID 1 mode:

md0: sda2 and sdb2
md1: sda3 and sdb3

Then, they appear in the partitions list and are configured as follows:

md0: swap
md1: ext3, /
}}}

 * Users & password setup: set root password, and create "root0" with a unique password (the installer forces at least one user account, make sure to delete the account after installation)

 * If /dev/sda is the USB stick and not the first disk, do not install GRUB to the Master Boot Record of /dev/sda. Instead, answer No at the prompt and choose /dev/sdb as the device. Then, take USB stick out, edit /boot/grub/menu.lst to replace references to hd(1,0) with hd(0,0), run '''update-grub.conf''' and '''grub-install /dev/sdb'''. No other tunings (to /etc/fstab or mdadm.conf) are needed as, if you used the partitioning example, no direct partitions occur in fstab, and for mdadm -- it uses UUIDs instead of partition names anyway.
 * In tasksel, at the end of installation, select "Standard system utilities" and "OpenSSH server"

= Booting into the new machine =

{{{#!wiki caution
This section is obsolete.
}}}

When the machine boots for the first time, run: {{{

dpkg-reconfigure debconf    # (choose interface: Dialog, priority: Low).

apt-get install less sudo vim emacs23-nox etckeeper changetrack lm-sensors openssh-server debsums logcheck bzip2 denyhosts rkhunter
}}}

Verify that disks performance is as expected using '''sync; sync; hdparm -tT /dev/sdX'''.

Activate etckeeper as documented on EtcKeeper.

Edit ''/etc/default/changetrack'' and set '''AUTO_TRACK_ALL_CONFFILES=yes'''.

Edit ''/etc/tripwire/twcfg.txt'' and set '''MAILNOVIOLATIONS =false'''. Initialize the database with '''tripwire --init'''. (If tripwire is installed)

Edit ''/etc/aliases'' and set "root" alias to "logs@hcoop.net", and possibly other addresses, separated by commas. (logs@ is an aliasMulti, defined in ~hcoop/.domtool/hcoop.net and lists people who want to receive verbose system logs).

Run '''sensors-detect''' to see if the kernel has appropriate thermal modules for the server, and add any drivers detected to ''/etc/modules''.

For all ext partitions, run '''tune2fs -j -c0 -i0 /dev/sdXX''' (and /dev/mdX for RAID arrays).

== Tune the /etc/apt/sources.list ==

{{{#!wiki caution
This section is obsolete.
}}}

{{{
cat > /etc/apt/sources.list <<\EOF
deb http://mirror.peer1.net/debian/ squeeze main
deb-src http://mirror.peer1.net/debian/ squeeze main

deb http://security.debian.org/ squeeze/updates main
deb-src http://security.debian.org/ squeeze/updates main

# backports are fairly useful and not installed by default
deb http://backports.debian.org/debian-backports squeeze-backports main
deb-src http://backports.debian.org/debian-backports squeeze-backports main
EOF

apt-get update
apt-get dist-upgrade
}}}

= Install the AFS Client =

{{{#!wiki caution
This section is obsolete.
}}}

The AFS client gets very unhappy if the partition holding {{{/var/cache/openafs}}} fills up.  To ensure that this can't happen, we'll create a 2GB file and mount it there using the loopback device.  This gives the openafs client a partition-in-a-file all to itself that no other process can interfere with.

First, create the file:
{{{
dd if=/dev/zero of=/var/cache/openafs.ext3 bs=1M count=2K
chmod go-rwx /var/cache/openafs.ext3
mke2fs -F /var/cache/openafs.ext3
tune2fs -j -i0 -c0 /var/cache/openafs.ext3
}}}

Then mount it.

Then, give our preferences to {{{debconf}}}:

{{{
debconf-set-selections <<\EOF
openafs-client openafs-client/thiscell string hcoop.net
openafs-client openafs-client/thiscell seen true
openafs-client openafs-client/dynroot boolean true
openafs-client openafs-client/dynroot seen true
openafs-client openafs-client/cachesize string 1500000
openafs-client openafs-client/cachesize seen true
openafs-client openafs-client/cell-info string
openafs-client openafs-client/cell-info seen true
openafs-client openafs-client/run-client boolean true
openafs-client openafs-client/run-client seen true
EOF
}}}

Once this is figured out (if all else fails, reboot) you should be able to

{{{
  /etc/init.d/openafs-client start
}}}

Do this and check that {{{/afs}}} shows up.

= Install Packages =

{{{#!wiki caution
This section is obsolete.
}}}

Install libnss-afs.

= Install Network Time Protocol Daemon =

{{{#!wiki caution
This section is obsolete.
}}}

Kerberos and AFS will not work correctly unless the clocks of the client and server are synchronized to within a certain tolerance.  Therefore, it is important for us to have a daemon running that keeps the clock set properly.  '''This step is not optional'''.

{{{
  apt-get install ntp
}}}

= Configure Kerberos =

{{{#!wiki caution
This section is obsolete.
}}}

'''''VERY IMPORTANT''''': put exactly the following in {{{/etc/krb5.conf}}} -- no more, no less (or actually, look up how it's done on Fritz or Hopper).

{{{
[libdefaults]
	default_realm = HCOOP.NET
	kdc_timesync = 1
	forwardable = true
	proxiable = true
	rdns = no          # undocumented option to disable reverse DNS lookups
[logging]
        default = FILE:/proc/self/fd/2
}}}

We distribute our Kerberos configuration via DNS, so it is very important that we do not "hardwire" the settings on any of the servers (except the KDCs themselves).  If we did, we wouldn't notice at first, but strange problems would crop up as soon as the DNS settings were changed.  So, it is important that we put only the bare minimum amount of information in {{{krb5.conf}}}.

= Configure Name Service =

{{{#!wiki caution
This section is obsolete.
}}}

A "name service" is Linux's mechanism for answering these queries:

 1. the userid for a given username and vice versa
 2. the groupid for a given groupname and vice versa
 3. the home directory for a user
 4. the shell for a user
 5. what groups a user is in

The {{{libnss-afs}}} package lets linux use the AFS user database (the {{{ptserver}}} or protection server) as a name service and makes PAGs show up as a special group.  To enable these changes, edit {{{/etc/nsswitch.conf}}} and change the {{{passwd}}} and {{{group}}} lines to look like this:

{{{
passwd:  afs files
group:   afs files
shadow:  files
}}}

`afs` is checked before `files` to keep UIDs consistent.

= Install Name Service Caching Daemon =

{{{#!wiki caution
This section is obsolete.
}}}

It is highly recommended to install {{{nscd}}} in order to get good performance out of {{{libnss-afs}}}.

{{{
  apt-get install nscd
}}}

We prefer to run nscd as a runit service so that it does not go down (except on deleuze, where it must be started strictly after AFS in the boot sequence).

{{{
  apt-get install runit
  mkdir /etc/service/nscd
  cat <<EOF > /etc/service/nscd/run
#!/bin/sh                                                                       
exec nscd -d
EOF
  mkdir /etc/service/nscd/log
  cat <<EOF > /etc/service/nscd/log/run
#!/bin/bash                                                                     
svlogd -tt /var/log/nscd/
EOF
  mkdir /var/log/nscd
  chmod +x /etc/service/nscd/log/run
  chmod +x /etc/service/nscd/run

  dpkg-divert --rename /etc/init.d/nscd
  ln -s /usr/bin/sv /etc/init.d/nscd
}}}

= Configure PAM =

{{{#!wiki caution
This section is obsolete.
}}}

PAM is Linux's mechanism to do the following:

 1. decide if somebody is who they say they are (authentication; in our case via kerberos)
 2. set up ''sessions'' (in the case of AFS, this means creating PAGs)
 3. change passwords (in our case, changing the password in the KDC)

Here's the usual PAM setup:

/etc/pam.d/common-account:

{{{
account sufficient      pam_unix.so
account required        pam_ldap.so
account required        pam_krb5.so debug

# temporary line for emergencies
#account required       pam_unix.so

account required pam_access.so
}}}

/etc/pam.d/common-auth:

{{{
auth    sufficient        pam_krb5.so debug forwardable ignore_root
auth    optional          pam_afs_session.so program=/usr/bin/aklog debug
auth    required          pam_unix.so nullok_secure try_first_pass

# temporary line for emergencies
#auth   required          pam_unix.so nullok_secure

auth    required          pam_env.so
}}}

/etc/pam.d/common-password:

{{{
password sufficient pam_krb5.so 
password required   pam_unix.so nullok obscure min=4 max=8 md5 shadow try_first_pass
}}}

/etc/pam.d/common-session:

{{{
session requisite pam_limits.so
session required  pam_unix_session.so      # Unix module just logs access
session optional  pam_krb5.so
session optional  pam_afs_session.so program=/usr/bin/aklog debug
}}}

/etc/pam.d/login (Add to beginning of file):

{{{
auth       required pam_listfile.so item=user sense=allow file=/etc/login.restrict  onerr=succeed
}}}

/etc/pam.d/ssh (Add just before {{{@include common-auth}}} line):

{{{
# sshd does not consult the "auth" section of pam when
# GssapiAuthentication=yes, even if UsePAM=yes.  Therefore, we add the
# check to the "account" section as well.
account    requisite    pam_listfile.so item=user sense=allow file=/etc/login.restrict onerr=succeed
auth       requisite    pam_listfile.so item=user sense=allow file=/etc/login.restrict onerr=succeed
}}}

If the machine is intended for user logins, DO NOT create /etc/login.restrict. If the machine is only
intended for admin logins, then create the file /etc/login.restrict with the following contents:

{{{
adamc_admin
clinton_admin
docelic_admin
rkd_admin
}}}

= Configure SSH =

{{{#!wiki caution
This section is obsolete.
}}}

== Configure SSH Client ==

Insert these lines in {{{/etc/ssh/ssh_config}}} so that ''outbound'' ssh connections will always try to use Kerberos if available:

{{{
  Host *
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no
}}}

== Configure SSH Server ==

You will need to create a "host principal" for the new server; if you are setting up {{{server.hcoop.net}}}, then it must have the name

{{{
   host/server.hcoop.net@HCOOP.NET
}}}

Add this principal to the KDC like this (execute these commands on the new server, as root, while holding admin tickets):

{{{
   REALM=HCOOP.NET
   ADMIN=myself_admin       # your admin username
   SERVER=server.hcoop.net
   rm -f /etc/krb5.keytab   # important -- if it already exists the new key will merely be appended
   kadmin -p $ADMIN@$REALM -r $REALM -q "add_principal -randkey host/$SERVER@$REALM" # unless it already exists (reinstall of VM for example)
   kadmin -p $ADMIN@$REALM -r $REALM -q "ktadd -k /etc/krb5.keytab host/$SERVER@$REALM"
   chown root:root /etc/krb5.keytab
   chmod go-rwx /etc/krb5.keytab
}}}

Then add these lines to the bottom of {{{/etc/ssh/sshd_config}}}:

{{{
  GssapiKeyExchange yes
  GssapiAuthentication yes
  GSSAPICleanupCredentials yes
}}}

Finally, restart the ssh server:

{{{
  /etc/init.d/ssh restart
}}}

= Populate sudoers =

{{{#!wiki caution
This section is obsolete.
}}}

Don't forget to give all of the admins lines in {{{/etc/sudoers}}}.  Each line should look like:

{{{
  user_admin  ALL=(ALL) NOPASSWD: ALL
}}}

= Set Up Some Cron Scripts =

{{{#!wiki caution
This section is obsolete.
}}}

/etc/cron.daily/hcoop-clean-tmp:

{{{
#!/bin/sh
#
# Clean /tmp periodically.
#
# Edit $TMPTIME in /etc/default/rcS to change the maximal age of /tmp entries
# before they are removed.

exec /afs/hcoop.net/common/etc/scripts/hcoop-clean-tmp
}}}

= Optional Steps =

{{{#!wiki caution
This section is obsolete.
}}}

== Install commonly-used packages ==

{{{
apt-get install \
  xbase-clients       # provides xauth, without which "ssh -Y" will not work
  dpkg-dev-el         # provide debian-changelog-mode
}}}

== Performance-Tune the OpenAFS Client ==

FIXME: AdamM needs to fill this in

== runit ==
The runit package is a mechanism for starting, stopping, and monitoring daemons.  It is an alternative to the traditional {{{/etc/init.d}}} and {{{start-stop-daemon}}} scheme.  Its chief advantages are:

  1. It launches daemons with '''clean process state'''; the daemon inherits nothing from the administrator invoking the start/stop command because the daemon is not forked as a child of the administrator's shell (rather, a request is sent {{{runit}}} daemon asking it to fork the daemon).  This is very important when dealing with tokens and pags.
  2. Runit monitors the processes that it forks, and restarts them if they die.
  3. Runit eliminates the need for pidfiles and the associated risk of starting multiple copies of a daemon.
  4. Runit captures the daemon's {{{stdout}}} and either sends it to a logger (if specified) or else displays it in the process name (output of {{{ps}}})

{{{
   apt-get install runit
}}}
When you move a process from {{{/etc/init.d/}}} control to {{{runit}}} supervision, you should inform debian that you have done so:
{{{
  # assuming /var/service/$SERVICE/run is the runit script
  dpkg-divert --rename /etc/init.d/$SERVICE
  ln -s /usr/bin/sv /etc/init.d/$SERVICE
}}}
This will cause invocations of {{{/etc/init.d/script {start|stop} }}} to do "the right thing".

== dnscache ==

You can install the dnscache package to make the server self-sufficient for dns resolution purposes (it acts as a tiny dns server just for localhost).  This improves the reliability of the overall infrastructure.

Starting dnscache via runit is often a good idea; this ensures that it starts early in the boot process and that it is restarted if it dies for any reason.

Here are the instructions for configuring it.  Make sure that bind9 (if running) is only listening to {{{127.0.0.1}}} and the public IP address of the machine.  We tell dnscache to listen on {{{127.0.0.2}}} so as to avoid conflicts with bind.

{{{
  apt-get install djbdns

  # If needed:
  addgroup --system Gdnscache
  adduser --system Gdnscache --ingroup Gdnscache

  # Create /etc/service/dnscache
  dnscache-conf Gdnscache Gdnscache /etc/service/dnscache 127.0.0.2

  # Change default listen address 127.0.0.1 to .2
  perl -pi -e 's/\.1/.2/' /etc/service/dnscache/env/IP

  # Let dnscache answer queries only from 127.0.0.2
  mv /var/dnscache/root/ip/127.0.0.1 /var/dnscache/root/ip/127.0.0.2

  sv restart dnscache
}}}

Then modify {{{/etc/resolv.conf}}}, replacing the {{{nameserver}}} lines with:

{{{
nameserver 127.0.0.2
}}}

== /etc/hosts ==

If not present already:

{{{
echo '127.0.0.1 localhost' > /etc/hosts
}}}

== ssmtp ==

Life is simpler when you run {{{ssmtp}}}.  You can direct the mail stream either to {{{deleuze}}} (preferred) or to a copy of {{{exim}}} running locally (but why bother running it?). 

Be sure to enable {{{FromLineOverride}}}, which ships defaulted to "off" in Debian.

{{{
apt-get install ssmtp
sed -i 's_FromLineOverride.*_FromLineOverride=YES_' /etc/ssmtp/ssmtp.conf
}}}

== noatime ==

By default, Linux will write to the disk in order to update the atime ("access time") every time a file is ''read from''; this substantially degrades performance.  You can disable this behavior by editing {{{/etc/fstab}}}

{{{
# <file system> <mount point>   <type>  <options>                          <dump>  <pass>
/dev/hda1       /               ext3    defaults,noatime,errors=remount-ro 0       1
}}}

This is especially important on filesystems which are used to store AFS volumes.

== etckeeper ==

{{{
apt-get install etckeeper
cd /etc
etckeeper init
etckeeper commit "Initial checkin"
git gc
}}}

== nitpicks ==

 1. Debian's installer seems to want to put an entry for the machine's own hostname in /etc/hosts, resolving to 127.0.0.1.  You'll probably want to remove it.
----
CategorySystemAdministration CategoryHistorical