welcome: please sign in

The following 953 words could not be found in the dictionary of 7 words (including 7 LocalSpellingWords) and are highlighted below:
2x   able   about   above   access   Access   accessible   account   accurate   Activate   acts   actually   Adam   adamc   Add   add   addgroup   address   addresses   Addresses   adduser   Admin   admin   Administration   administrator   admins   advantages   afs   after   again   aklog   alias   aliases   all   All   allow   already   alternative   always   amount   an   and   answer   answering   any   anything   anyway   appear   appended   apply   appropriate   approximately   apt   are   Area   arrays   as   asking   assigned   associated   assuming   at   atime   auth   Authentication   authentication   Auto   automatically   available   avoid   back   backports   band   Band   bare   Be   be   because   been   before   beginning   behavior   being   bin   bind   bind9   bit   bnx2   boolean   Boot   boot   Booting   boots   bother   bottom   Broadcom   bs   but   By   by   bzip2   c0   cache   cachesize   Caching   called   can   candidate   captures   card   cards   case   cat   Category   cause   caution   cd   cell   center   certain   change   Change   changed   changelog   changes   changetrack   changing   check   checked   checkin   chief   child   chmod   choose   Choose   choosen   chown   clean   Cleanup   Client   client   clients   clinton   clock   clocks   comes   command   commands   commas   commit   common   commonly   conf   config   configuration   Configure   configure   configured   configuring   conflicts   connect   connections   consistent   contents   Contents   control   convenient   Coop   copied   copies   copy   cores   correctly   count   create   Create   creating   Credentials   cron   Cron   crop   Cs   Currently   cycling   D1   Daemon   daemon   daemons   daily   data   database   date   dd   dealing   deb   debconf   debian   debsums   debug   December   decide   decided   default   defaulted   defaults   defined   definition   degrades   Delegate   delete   deleuze   denyhosts   details   detect   detected   dev   device   devices   Dialog   did   die   dies   direct   directory   disable   disk   disks   displays   dist   distribute   divert   djbdns   dns   dnscache   do   Do   docelic   document   documentation   documented   does   doing   Dom   domtool   Don   done   down   download   dpkg   drivers   Ds   dtl   dynroot   Each   early   echo   edit   Edit   editing   either   el   eliminates   else   emacs23   en   enable   end   ensure   ensures   entry   env   errors   especially   etc   Etc   etckeeper   every   exactly   example   Example   except   Exchange   exec   execute   exim   exists   expected   ext   ext3   fails   false   fd   figured   file   files   filesystems   fill   fills   Finally   find   Find   firmware   first   First   following   follows   foo   For   for   forces   forget   fork   forked   forks   form   forwardable   free   Fritz   from   From   front   fstab   Functions   gc   Gdnscache   get   gets   git   give   given   gives   go   going   good   group   groupid   groupname   groups   grub   Gs   Gssapi   guide   happen   harder   hardware   Hardware   hardwire   has   have   hc   hcoop   hd   hda1   hdparm   help   Here   highly   historical   Historical   holding   home   Hopper   host   Host   hostname   hosts   how   http   https   i0   id   idea   ideal   identical   if   If   ignore   image   images   important   improves   in   In   include   info   inform   information   infrastructure   ingroup   inherits   init   Initial   Initialize   Insert   install   Install   Installation   installation   installed   installer   installing   instead   Instead   instructions   intended   interface   interfere   into   investigate   invocations   invoking   ip   Ip   is   Is   it   It   item   its   Its   itself   just   kadmin   kdc   keep   Keeper   keeps   kept   Kerberos   kerberos   kernel   Key   key   Keyboard   keyboard   keytab   krb5   ktadd   Kvm   kvm   launches   ldap   least   less   Let   lets   lib   libdefaults   libnss   Life   like   limits   Line   line   lines   Linux   linux   list   List   listed   listen   listening   listfile   lists   ll   lm   ln   localhost   locally   located   log   logcheck   logger   logging   login   logins   logs   look   lookups   loopback   Low   lst   machine   Machine   machines   Machines   mail   main   maintain   make   Make   makes   manual   manually   mapping   Master   max   md   md0   md1   md5   mdadm   means   mechanism   media   members   Members   menu   merely   min   mind   minimum   mirror   mkdir   mke2fs   mode   modify   module   modules   monitoring   monitors   more   Most   mount   move   Multi   multiple   must   mv   myself   name   Name   names   nameserver   need   needed   needs   net   Net   network   Network   new   nitpicks   No   no   noatime   node   non   not   notes   nothing   notice   nox   nscd   nsswitch   ntp   nullok   number   obscure   obsolete   occur   of   Of   off   often   on   Once   one   onerr   only   onto   Open   openafs   openssh   option   Optional   optional   or   order   org   other   our   Out   out   outbound   output   overall   Override   own   owned   package   Packages   packages   page   pags   pam   panels   part   partition   Partition   partitioning   partitions   pass   passwd   password   passwords   peer1   people   Performance   performance   performed   perl   photograph   phys   physically   pi   pidfiles   plan   please   Poll   poll   Populate   portal   ports   possible   possibly   power   Power   prefer   preferences   preferred   Prepare   preseeded   present   primary   principal   priority   probably   problems   proc   Procedure   procedure   procedures   process   processes   processor   program   prompt   properly   protection   Protocol   provide   provided   provides   proxiable   ps   ptserver   public   purposes   put   queries   randkey   rather   rdns   re   read   realm   reason   reboot   Rebooting   receive   recommended   reconfigure   Record   reference   references   reinstall   reliability   remaining   remote   remount   remove   rename   replace   replacing   repository   representation   request   required   requires   requisite   resolution   resolv   resolving   restart   restarted   restarts   restore   restrict   reverse   right   risk   rkd   rkhunter   rm   ro   root   root0   Run   run   Runit   runit   running   rwx   same   say   scheme   Screen   script   scripts   Scripts   sd   sda   sda1   sda2   sda3   sdb   sdb1   sdb2   sdb3   search   section   secure   security   sed   See   see   seems   seen   select   selections   self   semi   sends   sense   sensors   sent   separated   sequence   Server   server   servers   service   Service   session   sessions   Set   set   setting   settings   setup   shadow   shell   ships   should   show   shows   sid   simpler   size   So   so   solution   Some   some   somebody   soon   sources   sp   space   special   Specific   specific   specified   specifying   Squeeze   squeeze   src   ssh   sshd   ssmtp   stable   Standard   start   started   Starting   starting   starts   state   stdout   step   Steps   steps   stick   stop   stopping   store   strange   stream   strictly   string   substantially   succeed   such   sudo   sudoers   sufficient   suggested   supervision   sure   sv   svlogd   swap   switch   sync   synchronized   system   System   Table   take   talking   tasksel   tell   that   The   the   them   themselves   then   Then   there   Therefore   thermal   These   these   they   thing   this   This   thiscell   those   through   tickets   Time   time   timesync   timezone   tiny   tmp   to   To   tokens   tolerance   tool   Tool   top   town   traditional   tripwire   true   try   tt   Tune   tune2fs   tunings   twcfg   two   txt   typically   under   undocumented   unetbootin   unhappy   unique   unix   Unix   unless   unmounted   Up   up   update   updates   upgrade   us   use   used   user   userid   username   Users   uses   Using   using   usr   usual   usually   utilities   var   verbose   Verify   versa   very   via   vice   video   vim   volume   volumes   want   way   We   we   were   what   When   when   where   which   whichever   while   who   why   Wiki   wiki   wikipedia   will   with   within   without   work   would   wouldn   write   written   xauth   xbase   Xtreme   Yes   yes   You   you   your   zero  

Clear message


These steps are listed in approximately the order in which they should be performed; please try to maintain that as you add to it.

See InstallationProcedure for an up to date guide to installing a new HCoop node. This document is (in December 2012) a semi-accurate representation of what the preseeded installer is doing, but keep the semi- part in mind. The page is being kept for historical reference.

List the Machine on the Wiki

The hostname of the machine should be decided through a Members Poll (accessible from members portal) such as https://members.hcoop.net/portal/poll?id=31.

Add the machine to the Hardware page.

It is a very good idea to photograph the front and back panels of the machine and put those images on the wiki page; that way remote admins and people in the data center can be sure they're talking about the same ports.

Add the machine to the IpAddresses page.

Set Up Out Of Band Access

All machines owned by hcoop should, if possible, have some out-of-band mechanism for:

  1. Keyboard access
  2. Screen access
  3. Power-cycling

Functions 1+2 are typically provided by kvm.hcoop.net (see KvmAccess); assuming you plan on going with that, you should connect the server's keyboard and video to the kvm switch.

Each server has its own solution for 3, usually in the form of a "service processor". You should investigate and document the appropriate service processor settings. If the service processor requires its own IP address, you should name it foo-sp.hcoop.net where foo.hcoop.net is the name of the server.

If there's _anything_ server-specific, please add an entry under "Specific Machines" on page AdminArea and document what it is. Rebooting procedures are an ideal candidate for this.

Add a DNS entry for the server

This is done as follows:

  1. Edit /afs/hcoop.net/common/etc/domtool/lib/hcoop.dtl and add definition for "HOSTNAME_ip" (search for "deleuze_ip" and just copy the line to new name)

  2. Edit /afs/hcoop.net/user/h/hc/hcoop/.domtool/hcoop.net to add the new DNS entry, using HOSTNAME_ip (again, can use deleuze_ip as example)

  3. To apply DomTool configuration, run DOMTOOL_USER=hcoop domtool hcoop.net in the ~hcoop/.domtool/ directory

  4. Using the peer1 request portal, add a reverse dns mapping to the hostname

Install Debian

This section is obsolete.

We use Debian GNU.

Here are the installation notes to help you:

  1. Find Debian stable image (whichever is 'stable' at time of installation -- this documentation is written for Squeeze)
  2. Prepare a USB stick to boot from (can do it manually or with convenient tool called "unetbootin")
  3. In system BIOS, choose 'Auto-power on on power restore' (if there is such option), and see if you can make USB stick to not be the first disk (when it's the first disk, it gets assigned device name /dev/sda and makes the installation a tiny bit harder)
  4. See which network card is in the server, if it requires non-free firmware, the package needs to be manually copied from Debian's non-free repository onto the install media (example is package "firmware-bnx2" for Broadcom NetXtremeII cards (http://packages.debian.org/sid/all/firmware-bnx2/download)). Once package is on the media, the install procedure will, if it is needed, automatically find and install it

  5. For timezone, use timezone where the server is physically located, and answer Yes to "Is the hardware clock set to GMT?"
  6. Choose manual network configuration, specifying the choosen hostname, IP and network details as listed on the IpAddresses page

  7. Partition disks. Most often, this comes town to creating identical partitions on all disks that are part of RAID1, and creating RAID arrays as in the example that follows. Currently, we usually configure: all remaining free space on /, /boot (300M), /var/cache/openafs (set to 5 or more GB), /tmp (1G), and installer suggested amount of swap.

    Example: 2x 160 GB system disks
    System disk 1:
    sda1: primary, beginning, 1 GB, ext3, /boot
    sda2: primary, beginning, 8 GB, use as phys. volume for RAID (swap space: 1 GB x number of proc. cores)
    sda3: primary, beginning, all available space, use as phys. volume for RAID
    System disk 2:
    sdb1: primary, beginning, 1 GB, ext3, unmounted
    sdb2: primary, beginning, 8 GB, use as phys. volume for RAID (swap, same size as above)
    sdb3: primary, beginning, all available space, use as phys. volume for RAID
    Then, after RAID partitions have been assigned, new option "Configure RAID"
    will appear at the top of the partitioning menu. We add the two devices in
    RAID 1 mode:
    md0: sda2 and sdb2
    md1: sda3 and sdb3
    Then, they appear in the partitions list and are configured as follows:
    md0: swap
    md1: ext3, /
  8. Users & password setup: set root password, and create "root0" with a unique password (the installer forces at least one user account, make sure to delete the account after installation)

  9. If /dev/sda is the USB stick and not the first disk, do not install GRUB to the Master Boot Record of /dev/sda. Instead, answer No at the prompt and choose /dev/sdb as the device. Then, take USB stick out, edit /boot/grub/menu.lst to replace references to hd(1,0) with hd(0,0), run update-grub.conf and grub-install /dev/sdb. No other tunings (to /etc/fstab or mdadm.conf) are needed as, if you used the partitioning example, no direct partitions occur in fstab, and for mdadm -- it uses UUIDs instead of partition names anyway.

  10. In tasksel, at the end of installation, select "Standard system utilities" and "OpenSSH server"

Booting into the new machine

This section is obsolete.

When the machine boots for the first time, run:

dpkg-reconfigure debconf    # (choose interface: Dialog, priority: Low).

apt-get install less sudo vim emacs23-nox etckeeper changetrack lm-sensors openssh-server debsums logcheck bzip2 denyhosts rkhunter

Verify that disks performance is as expected using sync; sync; hdparm -tT /dev/sdX.

Activate etckeeper as documented on EtcKeeper.

Edit /etc/default/changetrack and set AUTO_TRACK_ALL_CONFFILES=yes.

Edit /etc/tripwire/twcfg.txt and set MAILNOVIOLATIONS =false. Initialize the database with tripwire --init. (If tripwire is installed)

Edit /etc/aliases and set "root" alias to "logs@hcoop.net", and possibly other addresses, separated by commas. (logs@ is an aliasMulti, defined in ~hcoop/.domtool/hcoop.net and lists people who want to receive verbose system logs).

Run sensors-detect to see if the kernel has appropriate thermal modules for the server, and add any drivers detected to /etc/modules.

For all ext partitions, run tune2fs -j -c0 -i0 /dev/sdXX (and /dev/mdX for RAID arrays).

Tune the /etc/apt/sources.list

This section is obsolete.

cat > /etc/apt/sources.list <<\EOF
deb http://mirror.peer1.net/debian/ squeeze main
deb-src http://mirror.peer1.net/debian/ squeeze main

deb http://security.debian.org/ squeeze/updates main
deb-src http://security.debian.org/ squeeze/updates main

# backports are fairly useful and not installed by default
deb http://backports.debian.org/debian-backports squeeze-backports main
deb-src http://backports.debian.org/debian-backports squeeze-backports main

apt-get update
apt-get dist-upgrade

Install the AFS Client

This section is obsolete.

The AFS client gets very unhappy if the partition holding /var/cache/openafs fills up. To ensure that this can't happen, we'll create a 2GB file and mount it there using the loopback device. This gives the openafs client a partition-in-a-file all to itself that no other process can interfere with.

First, create the file:

dd if=/dev/zero of=/var/cache/openafs.ext3 bs=1M count=2K
chmod go-rwx /var/cache/openafs.ext3
mke2fs -F /var/cache/openafs.ext3
tune2fs -j -i0 -c0 /var/cache/openafs.ext3

Then mount it.

Then, give our preferences to debconf:

debconf-set-selections <<\EOF
openafs-client openafs-client/thiscell string hcoop.net
openafs-client openafs-client/thiscell seen true
openafs-client openafs-client/dynroot boolean true
openafs-client openafs-client/dynroot seen true
openafs-client openafs-client/cachesize string 1500000
openafs-client openafs-client/cachesize seen true
openafs-client openafs-client/cell-info string
openafs-client openafs-client/cell-info seen true
openafs-client openafs-client/run-client boolean true
openafs-client openafs-client/run-client seen true

Once this is figured out (if all else fails, reboot) you should be able to

  /etc/init.d/openafs-client start

Do this and check that /afs shows up.

Install Packages

This section is obsolete.

Install libnss-afs.

Install Network Time Protocol Daemon

This section is obsolete.

Kerberos and AFS will not work correctly unless the clocks of the client and server are synchronized to within a certain tolerance. Therefore, it is important for us to have a daemon running that keeps the clock set properly. This step is not optional.

  apt-get install ntp

Configure Kerberos

This section is obsolete.

VERY IMPORTANT: put exactly the following in /etc/krb5.conf -- no more, no less (or actually, look up how it's done on Fritz or Hopper).

        default_realm = HCOOP.NET
        kdc_timesync = 1
        forwardable = true
        proxiable = true
        rdns = no          # undocumented option to disable reverse DNS lookups
        default = FILE:/proc/self/fd/2

We distribute our Kerberos configuration via DNS, so it is very important that we do not "hardwire" the settings on any of the servers (except the KDCs themselves). If we did, we wouldn't notice at first, but strange problems would crop up as soon as the DNS settings were changed. So, it is important that we put only the bare minimum amount of information in krb5.conf.

Configure Name Service

This section is obsolete.

A "name service" is Linux's mechanism for answering these queries:

  1. the userid for a given username and vice versa
  2. the groupid for a given groupname and vice versa
  3. the home directory for a user
  4. the shell for a user
  5. what groups a user is in

The libnss-afs package lets linux use the AFS user database (the ptserver or protection server) as a name service and makes PAGs show up as a special group. To enable these changes, edit /etc/nsswitch.conf and change the passwd and group lines to look like this:

passwd:  afs files
group:   afs files
shadow:  files

afs is checked before files to keep UIDs consistent.

Install Name Service Caching Daemon

This section is obsolete.

It is highly recommended to install nscd in order to get good performance out of libnss-afs.

  apt-get install nscd

We prefer to run nscd as a runit service so that it does not go down (except on deleuze, where it must be started strictly after AFS in the boot sequence).

  apt-get install runit
  mkdir /etc/service/nscd
  cat <<EOF > /etc/service/nscd/run
exec nscd -d
  mkdir /etc/service/nscd/log
  cat <<EOF > /etc/service/nscd/log/run
svlogd -tt /var/log/nscd/
  mkdir /var/log/nscd
  chmod +x /etc/service/nscd/log/run
  chmod +x /etc/service/nscd/run

  dpkg-divert --rename /etc/init.d/nscd
  ln -s /usr/bin/sv /etc/init.d/nscd

Configure PAM

This section is obsolete.

PAM is Linux's mechanism to do the following:

  1. decide if somebody is who they say they are (authentication; in our case via kerberos)
  2. set up sessions (in the case of AFS, this means creating PAGs)

  3. change passwords (in our case, changing the password in the KDC)

Here's the usual PAM setup:


account sufficient      pam_unix.so
account required        pam_ldap.so
account required        pam_krb5.so debug

# temporary line for emergencies
#account required       pam_unix.so

account required pam_access.so


auth    sufficient        pam_krb5.so debug forwardable ignore_root
auth    optional          pam_afs_session.so program=/usr/bin/aklog debug
auth    required          pam_unix.so nullok_secure try_first_pass

# temporary line for emergencies
#auth   required          pam_unix.so nullok_secure

auth    required          pam_env.so


password sufficient pam_krb5.so 
password required   pam_unix.so nullok obscure min=4 max=8 md5 shadow try_first_pass


session requisite pam_limits.so
session required  pam_unix_session.so      # Unix module just logs access
session optional  pam_krb5.so
session optional  pam_afs_session.so program=/usr/bin/aklog debug

/etc/pam.d/login (Add to beginning of file):

auth       required pam_listfile.so item=user sense=allow file=/etc/login.restrict  onerr=succeed

/etc/pam.d/ssh (Add just before @include common-auth line):

# sshd does not consult the "auth" section of pam when
# GssapiAuthentication=yes, even if UsePAM=yes.  Therefore, we add the
# check to the "account" section as well.
account    requisite    pam_listfile.so item=user sense=allow file=/etc/login.restrict onerr=succeed
auth       requisite    pam_listfile.so item=user sense=allow file=/etc/login.restrict onerr=succeed

If the machine is intended for user logins, DO NOT create /etc/login.restrict. If the machine is only intended for admin logins, then create the file /etc/login.restrict with the following contents:


Configure SSH

This section is obsolete.

Configure SSH Client

Insert these lines in /etc/ssh/ssh_config so that outbound ssh connections will always try to use Kerberos if available:

  Host *
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no

Configure SSH Server

You will need to create a "host principal" for the new server; if you are setting up server.hcoop.net, then it must have the name


Add this principal to the KDC like this (execute these commands on the new server, as root, while holding admin tickets):

   ADMIN=myself_admin       # your admin username
   rm -f /etc/krb5.keytab   # important -- if it already exists the new key will merely be appended
   kadmin -p $ADMIN@$REALM -r $REALM -q "add_principal -randkey host/$SERVER@$REALM" # unless it already exists (reinstall of VM for example)
   kadmin -p $ADMIN@$REALM -r $REALM -q "ktadd -k /etc/krb5.keytab host/$SERVER@$REALM"
   chown root:root /etc/krb5.keytab
   chmod go-rwx /etc/krb5.keytab

Then add these lines to the bottom of /etc/ssh/sshd_config:

  GssapiKeyExchange yes
  GssapiAuthentication yes
  GSSAPICleanupCredentials yes

Finally, restart the ssh server:

  /etc/init.d/ssh restart

Populate sudoers

This section is obsolete.

Don't forget to give all of the admins lines in /etc/sudoers. Each line should look like:

  user_admin  ALL=(ALL) NOPASSWD: ALL

Set Up Some Cron Scripts

This section is obsolete.


# Clean /tmp periodically.
# Edit $TMPTIME in /etc/default/rcS to change the maximal age of /tmp entries
# before they are removed.

exec /afs/hcoop.net/common/etc/scripts/hcoop-clean-tmp

Optional Steps

This section is obsolete.

Install commonly-used packages

apt-get install \
  xbase-clients       # provides xauth, without which "ssh -Y" will not work
  dpkg-dev-el         # provide debian-changelog-mode

Performance-Tune the OpenAFS Client

FIXME: AdamM needs to fill this in


The runit package is a mechanism for starting, stopping, and monitoring daemons. It is an alternative to the traditional /etc/init.d and start-stop-daemon scheme. Its chief advantages are:

  1. It launches daemons with clean process state; the daemon inherits nothing from the administrator invoking the start/stop command because the daemon is not forked as a child of the administrator's shell (rather, a request is sent runit daemon asking it to fork the daemon). This is very important when dealing with tokens and pags.

  2. Runit monitors the processes that it forks, and restarts them if they die.
  3. Runit eliminates the need for pidfiles and the associated risk of starting multiple copies of a daemon.
  4. Runit captures the daemon's stdout and either sends it to a logger (if specified) or else displays it in the process name (output of ps)

   apt-get install runit

When you move a process from /etc/init.d/ control to runit supervision, you should inform debian that you have done so:

  # assuming /var/service/$SERVICE/run is the runit script
  dpkg-divert --rename /etc/init.d/$SERVICE
  ln -s /usr/bin/sv /etc/init.d/$SERVICE

This will cause invocations of /etc/init.d/script {start|stop to do "the right thing".


You can install the dnscache package to make the server self-sufficient for dns resolution purposes (it acts as a tiny dns server just for localhost). This improves the reliability of the overall infrastructure.

Starting dnscache via runit is often a good idea; this ensures that it starts early in the boot process and that it is restarted if it dies for any reason.

Here are the instructions for configuring it. Make sure that bind9 (if running) is only listening to and the public IP address of the machine. We tell dnscache to listen on so as to avoid conflicts with bind.

  apt-get install djbdns

  # If needed:
  addgroup --system Gdnscache
  adduser --system Gdnscache --ingroup Gdnscache

  # Create /etc/service/dnscache
  dnscache-conf Gdnscache Gdnscache /etc/service/dnscache

  # Change default listen address to .2
  perl -pi -e 's/\.1/.2/' /etc/service/dnscache/env/IP

  # Let dnscache answer queries only from
  mv /var/dnscache/root/ip/ /var/dnscache/root/ip/

  sv restart dnscache

Then modify /etc/resolv.conf, replacing the nameserver lines with:



If not present already:

echo ' localhost' > /etc/hosts


Life is simpler when you run ssmtp. You can direct the mail stream either to deleuze (preferred) or to a copy of exim running locally (but why bother running it?).

Be sure to enable FromLineOverride, which ships defaulted to "off" in Debian.

apt-get install ssmtp
sed -i 's_FromLineOverride.*_FromLineOverride=YES_' /etc/ssmtp/ssmtp.conf


By default, Linux will write to the disk in order to update the atime ("access time") every time a file is read from; this substantially degrades performance. You can disable this behavior by editing /etc/fstab

# <file system> <mount point>   <type>  <options>                          <dump>  <pass>
/dev/hda1       /               ext3    defaults,noatime,errors=remount-ro 0       1

This is especially important on filesystems which are used to store AFS volumes.


apt-get install etckeeper
cd /etc
etckeeper init
etckeeper commit "Initial checkin"
git gc


  1. Debian's installer seems to want to put an entry for the machine's own hostname in /etc/hosts, resolving to You'll probably want to remove it.

CategorySystemAdministration CategoryHistorical

SetupNewMachines (last edited 2012-12-20 21:13:00 by ClintonEbadi)