1562012-12-20 21:13:00ClintonEbadideprecate page1552012-09-03 08:53:25ClintonEbadiprepare for overhauling document with new automagic installs1542012-06-05 08:49:52ClintonEbadimentioning that you might want to get a reverse dns mapping from peer1 could be helpful1532012-04-28 05:05:31ClintonEbadidemystify kadmin invocations (ank is way deprecated!)1522012-03-17 05:28:12ClintonEbadiremove obsolete info and update a bit for squeeze1512011-03-02 14:52:44ClintonEbadiactually, check afs before files (didn't realize why this was important...)1502011-03-02 08:18:39ClintonEbadi1492011-03-02 08:09:18ClintonEbadi1482010-12-04 02:33:12ClintonEbadidebian runit services are in /etc/ now1472010-12-01 03:25:53ClintonEbadiactual procedure for adding a new dns entry1462010-11-29 08:33:27DavorOcelic1452010-11-29 08:28:45DavorOcelic1442010-11-29 08:25:43DavorOcelic1432010-05-21 07:48:02DavorOcelicOverride some cretin's edit.1422010-05-21 07:46:13polom.alumontplast.czVery nice site!1412010-05-21 07:45:16200.249.245.134 Hello! dfbedff interesting dfbedff site!1402010-01-08 15:42:23DavorOcelic1392010-01-07 22:49:03DavorOcelic1382010-01-07 22:46:57DavorOcelic1372010-01-07 01:40:45DavorOcelic1362010-01-07 01:13:18DavorOcelic1352010-01-06 23:49:48DavorOcelic1342010-01-06 13:00:15DavorOcelic1332010-01-06 12:22:25DavorOcelic1322010-01-06 11:45:43DavorOcelic1312010-01-06 11:41:05DavorOcelic1302010-01-06 11:35:01DavorOcelic1292010-01-06 11:22:13DavorOcelic1282009-10-10 19:52:36212.15.179.2251272009-09-28 09:04:35DavorOcelic1262009-09-23 07:12:3078-2-84-198.adsl.net.t-com.hr1252009-09-22 18:35:3278-2-84-198.adsl.net.t-com.hr1242009-08-30 21:22:34169.229.3.1781232009-07-11 23:59:32dhcp-37-70.EECS.Berkeley.EDU1222009-07-11 23:28:51dhcp-37-70.EECS.Berkeley.EDU1212009-07-09 23:30:23192.18.41.1961202009-06-21 20:56:25dhcp-37-70.EECS.Berkeley.EDU1192009-06-21 20:53:55dhcp-37-70.EECS.Berkeley.EDU1182009-06-21 20:53:22dhcp-37-70.EECS.Berkeley.EDU1172009-06-14 06:49:10dhcp-37-70.EECS.Berkeley.EDU1162009-06-14 06:19:05dhcp-37-70.EECS.Berkeley.EDU1152009-06-14 01:24:07dhcp-37-70.EECS.Berkeley.EDUadd etckeeper1142009-06-07 03:57:25sca-ea-fw-1.Sun.COM1132008-12-30 22:43:34dhcp-37-70.EECS.Berkeley.EDU1122008-12-30 20:37:35dhcp-37-70.EECS.Berkeley.EDU1112008-07-07 04:28:02localhostconverted to 1.6 markup1102008-06-08 19:17:38AdamMegacz1092008-06-08 19:11:19AdamMegacz1082008-06-08 18:49:23AdamMegacz1072008-06-08 18:15:01AdamMegacz1062008-06-08 18:11:07AdamMegacz1052008-05-01 18:44:28AdamMegacz1042008-04-21 04:07:15AdamMegacz1032008-04-16 02:08:45AdamMegacz1022008-04-16 02:08:02AdamMegaczadd a note about 127.0.0.1 in /etc/hosts1012008-04-06 17:31:22AdamMegacz1002008-03-31 21:24:30AdamMegacz992008-03-31 17:53:21office4.tmcs.netRemove stray "/"982008-03-31 05:01:54MichaelOlsonRevert to revision 96.972008-03-31 04:59:46MichaelOlsonRemove the mv .../10.0.0.1 step962008-03-31 04:38:49MichaelOlsonFinish dnscache instructions952008-03-31 04:36:08MichaelOlsonFix location to dnscache package942008-03-31 04:34:42MichaelOlsonConfigure dnscache to listen on 127.0.0.2932008-03-31 01:59:47AdamMegaczadd dpkg-divert lines so debian {pre,post}{inst,rm} scripts know about runit-ized nscd922008-03-31 01:57:52AdamMegaczremove unnecessary NIS-specific stuff from nscd runit script912008-03-31 01:43:20MichaelOlson902008-03-31 01:42:06MichaelOlson892008-03-31 01:41:24MichaelOlsonAdd cron scripts section882008-03-31 01:40:03MichaelOlsonDemote headings872008-03-30 23:19:40MichaelOlsonOnly consult login.restrict for ssh and login862008-03-30 23:06:33MichaelOlsonmake nscd log dir852008-03-30 23:03:33MichaelOlsonStart nscd as a runit service842008-03-30 02:06:49AdamMegacz832008-03-30 02:04:48AdamMegacz822008-03-30 02:03:27AdamMegacz812008-03-29 17:08:56AdamMegacz802008-03-28 17:40:20AdamMegacz792008-03-28 17:38:46AdamMegacz782008-03-28 17:37:47AdamMegacz772008-03-28 17:37:03AdamMegacz762008-03-28 17:32:58AdamMegacz752008-03-28 17:31:23AdamMegacz742008-03-28 17:23:48AdamMegacz732008-03-28 17:22:58AdamMegacz722008-03-28 17:22:07AdamMegacz712008-03-28 17:11:00AdamMegacz702008-03-28 16:55:39AdamMegacz692008-03-28 16:55:18AdamMegacz682008-03-28 16:54:48AdamMegacz672008-03-10 17:44:12netblock-68-183-198-235.dslextreme.com662008-03-10 17:40:53netblock-68-183-198-235.dslextreme.com652008-03-10 17:40:13netblock-68-183-198-235.dslextreme.com642008-03-10 01:12:06DavorOcelic632008-03-10 01:01:4577.237.104.49622008-03-09 21:34:57netblock-68-183-198-235.dslextreme.com612008-03-09 21:33:55netblock-68-183-198-235.dslextreme.com602008-03-03 22:06:05dhcp-37-70.EECS.Berkeley.EDU592008-03-03 22:03:22dhcp-37-70.EECS.Berkeley.EDU582008-03-03 22:02:52dhcp-37-70.EECS.Berkeley.EDU572008-03-03 21:58:30dhcp-37-70.EECS.Berkeley.EDU562008-03-03 21:57:36dhcp-37-70.EECS.Berkeley.EDU552008-03-02 23:32:15dhcp-37-70.EECS.Berkeley.EDU542008-03-02 23:26:24dhcp-37-70.EECS.Berkeley.EDU532008-03-02 23:25:21dhcp-37-70.EECS.Berkeley.EDU522008-03-02 23:21:34dhcp-37-70.EECS.Berkeley.EDU512008-03-02 23:19:26dhcp-37-70.EECS.Berkeley.EDU502008-03-02 22:02:22dhcp-37-70.EECS.Berkeley.EDU492008-03-02 21:10:09dhcp-37-70.EECS.Berkeley.EDU482008-03-02 21:09:31dhcp-37-70.EECS.Berkeley.EDU472008-03-02 21:08:34dhcp-37-70.EECS.Berkeley.EDU462008-03-02 21:08:08dhcp-37-70.EECS.Berkeley.EDU452008-03-02 21:06:15dhcp-37-70.EECS.Berkeley.EDU442008-03-02 21:05:38dhcp-37-70.EECS.Berkeley.EDU432008-03-02 21:05:28dhcp-37-70.EECS.Berkeley.EDU422008-03-02 21:02:55dhcp-37-70.EECS.Berkeley.EDU412008-03-02 21:01:21dhcp-37-70.EECS.Berkeley.EDU402008-03-02 20:59:05dhcp-37-70.EECS.Berkeley.EDU392008-03-02 20:57:37dhcp-37-70.EECS.Berkeley.EDU382008-03-02 20:56:29dhcp-37-70.EECS.Berkeley.EDU372008-03-02 20:55:44dhcp-37-70.EECS.Berkeley.EDU362008-03-02 20:54:47dhcp-37-70.EECS.Berkeley.EDU352008-03-02 20:53:29dhcp-37-70.EECS.Berkeley.EDU342008-03-02 20:51:28dhcp-37-70.EECS.Berkeley.EDU332008-03-02 20:48:32dhcp-37-70.EECS.Berkeley.EDU322008-03-02 20:45:36dhcp-37-70.EECS.Berkeley.EDU312008-03-02 20:43:33dhcp-37-70.EECS.Berkeley.EDU302008-03-02 20:33:09dhcp-37-70.EECS.Berkeley.EDU292008-03-02 20:32:24dhcp-37-70.EECS.Berkeley.EDU282008-03-02 20:29:11dhcp-37-70.EECS.Berkeley.EDU272008-03-02 20:28:45dhcp-37-70.EECS.Berkeley.EDU262008-03-02 20:26:03dhcp-37-70.EECS.Berkeley.EDU252008-03-02 20:24:44dhcp-37-70.EECS.Berkeley.EDU242008-03-02 20:24:08dhcp-37-70.EECS.Berkeley.EDU232008-03-02 20:22:26dhcp-37-70.EECS.Berkeley.EDU222008-03-02 20:21:19dhcp-37-70.EECS.Berkeley.EDU212008-03-02 20:20:57dhcp-37-70.EECS.Berkeley.EDU202008-03-02 20:20:24dhcp-37-70.EECS.Berkeley.EDU192008-03-02 20:20:04dhcp-37-70.EECS.Berkeley.EDU182008-03-02 20:19:12dhcp-37-70.EECS.Berkeley.EDU172008-03-02 20:18:26dhcp-37-70.EECS.Berkeley.EDU162008-03-02 20:17:41dhcp-37-70.EECS.Berkeley.EDU152008-03-02 20:13:39dhcp-37-70.EECS.Berkeley.EDU142008-03-02 20:10:17dhcp-37-70.EECS.Berkeley.EDU132008-03-02 20:09:42dhcp-37-70.EECS.Berkeley.EDU122008-03-02 20:09:27dhcp-37-70.EECS.Berkeley.EDU112008-03-02 20:08:17dhcp-37-70.EECS.Berkeley.EDU102008-03-02 20:06:07dhcp-37-70.EECS.Berkeley.EDU92008-03-02 20:03:11dhcp-37-70.EECS.Berkeley.EDU82008-03-02 20:01:11dhcp-37-70.EECS.Berkeley.EDU72008-03-02 19:52:12dhcp-37-70.EECS.Berkeley.EDU62008-03-02 19:49:24dhcp-37-70.EECS.Berkeley.EDU52008-03-02 19:47:21dhcp-37-70.EECS.Berkeley.EDU42008-03-02 19:44:00dhcp-37-70.EECS.Berkeley.EDU32008-03-02 19:30:42dhcp-37-70.EECS.Berkeley.EDU22008-03-02 19:30:29dhcp-37-70.EECS.Berkeley.EDU12008-03-02 19:29:45dhcp-37-70.EECS.Berkeley.EDUThese steps are listed in approximately the order in which they should be performed; please try to maintain that as you add to it. See InstallationProcedure for an up to date guide to installing a new HCoop node. This document is (in December 2012) a semi-accurate representation of what the preseeded installer is doing, but keep the semi- part in mind. The page is being kept for historical reference.

The hostname of the machine should be decided through a Members Poll (accessible from members portal) such as . Add the machine to the Hardware page. It is a very good idea to photograph the front and back panels of the machine and put those images on the wiki page; that way remote admins and people in the data center can be sure they're talking about the same ports. Add the machine to the IpAddresses page.

All machines owned by hcoop should, if possible, have some out-of-band mechanism for: Keyboard access Screen access Power-cycling Functions 1+2 are typically provided by kvm.hcoop.net (see KvmAccess); assuming you plan on going with that, you should connect the server's keyboard and video to the kvm switch. Each server has its own solution for 3, usually in the form of a "service processor". You should investigate and document the appropriate service processor settings. If the service processor requires its own IP address, you should name it foo-sp.hcoop.net where foo.hcoop.net is the name of the server. If there's _anything_ server-specific, please add an entry under "Specific Machines" on page AdminArea and document what it is. Rebooting procedures are an ideal candidate for this.

This is done as follows: Edit /afs/hcoop.net/common/etc/domtool/lib/hcoop.dtl and add definition for "HOSTNAME_ip" (search for "deleuze_ip" and just copy the line to new name) Edit /afs/hcoop.net/user/h/hc/hcoop/.domtool/hcoop.net to add the new DNS entry, using HOSTNAME_ip (again, can use deleuze_ip as example) To apply DomTool configuration, run DOMTOOL_USER=hcoop domtool hcoop.net in the ~hcoop/.domtool/ directory Using the peer1 request portal, add a reverse dns mapping to the hostname

This section is obsolete. We use Debian GNU. Here are the installation notes to help you: Find Debian stable image (whichever is 'stable' at time of installation -- this documentation is written for Squeeze) Prepare a USB stick to boot from (can do it manually or with convenient tool called "unetbootin") In system BIOS, choose 'Auto-power on on power restore' (if there is such option), and see if you can make USB stick to not be the first disk (when it's the first disk, it gets assigned device name /dev/sda and makes the installation a tiny bit harder) See which network card is in the server, if it requires non-free firmware, the package needs to be manually copied from Debian's non-free repository onto the install media (example is package "firmware-bnx2" for Broadcom NetXtremeII cards ()). Once package is on the media, the install procedure will, if it is needed, automatically find and install it For timezone, use timezone where the server is physically located, and answer Yes to "Is the hardware clock set to GMT?" Choose manual network configuration, specifying the choosen hostname, IP and network details as listed on the IpAddresses page Partition disks. Most often, this comes town to creating identical partitions on all disks that are part of RAID1, and creating RAID arrays as in the example that follows. Currently, we usually configure: all remaining free space on /, /boot (300M), /var/cache/openafs (set to 5 or more GB), /tmp (1G), and installer suggested amount of swap. Users & password setup: set root password, and create "root0" with a unique password (the installer forces at least one user account, make sure to delete the account after installation) If /dev/sda is the USB stick and not the first disk, do not install GRUB to the Master Boot Record of /dev/sda. Instead, answer No at the prompt and choose /dev/sdb as the device. Then, take USB stick out, edit /boot/grub/menu.lst to replace references to hd(1,0) with hd(0,0), run update-grub.conf and grub-install /dev/sdb. No other tunings (to /etc/fstab or mdadm.conf) are needed as, if you used the partitioning example, no direct partitions occur in fstab, and for mdadm -- it uses UUIDs instead of partition names anyway. In tasksel, at the end of installation, select "Standard system utilities" and "OpenSSH server"

This section is obsolete. When the machine boots for the first time, run: Verify that disks performance is as expected using sync; sync; hdparm -tT /dev/sdX. Activate etckeeper as documented on EtcKeeper. Edit /etc/default/changetrack and set AUTO_TRACK_ALL_CONFFILES=yes. Edit /etc/tripwire/twcfg.txt and set MAILNOVIOLATIONS =false. Initialize the database with tripwire --init. (If tripwire is installed) Edit /etc/aliases and set "root" alias to "logs@hcoop.net", and possibly other addresses, separated by commas. (logs@ is an aliasMulti, defined in ~hcoop/.domtool/hcoop.net and lists people who want to receive verbose system logs). Run sensors-detect to see if the kernel has appropriate thermal modules for the server, and add any drivers detected to /etc/modules. For all ext partitions, run tune2fs -j -c0 -i0 /dev/sdXX (and /dev/mdX for RAID arrays).

This section is obsolete. /etc/apt/sources.list <<\EOF deb http://mirror.peer1.net/debian/ squeeze main deb-src http://mirror.peer1.net/debian/ squeeze main ]]>

This section is obsolete. The AFS client gets very unhappy if the partition holding /var/cache/openafs fills up. To ensure that this can't happen, we'll create a 2GB file and mount it there using the loopback device. This gives the openafs client a partition-in-a-file all to itself that no other process can interfere with. First, create the file: Then mount it. Then, give our preferences to debconf: Once this is figured out (if all else fails, reboot) you should be able to Do this and check that /afs shows up.

This section is obsolete. Install libnss-afs.

This section is obsolete. Kerberos and AFS will not work correctly unless the clocks of the client and server are synchronized to within a certain tolerance. Therefore, it is important for us to have a daemon running that keeps the clock set properly. This step is not optional.

This section is obsolete. VERY IMPORTANT: put exactly the following in /etc/krb5.conf -- no more, no less (or actually, look up how it's done on Fritz or Hopper). We distribute our Kerberos configuration via DNS, so it is very important that we do not "hardwire" the settings on any of the servers (except the KDCs themselves). If we did, we wouldn't notice at first, but strange problems would crop up as soon as the DNS settings were changed. So, it is important that we put only the bare minimum amount of information in krb5.conf.

This section is obsolete. A "name service" is Linux's mechanism for answering these queries: the userid for a given username and vice versa the groupid for a given groupname and vice versa the home directory for a user the shell for a user what groups a user is in The libnss-afs package lets linux use the AFS user database (the ptserver or protection server) as a name service and makes PAGs show up as a special group. To enable these changes, edit /etc/nsswitch.conf and change the passwd and group lines to look like this: afs is checked before files to keep UIDs consistent.

This section is obsolete. It is highly recommended to install nscd in order to get good performance out of libnss-afs. We prefer to run nscd as a runit service so that it does not go down (except on deleuze, where it must be started strictly after AFS in the boot sequence). /etc/service/nscd/run #!/bin/sh exec nscd -d EOF mkdir /etc/service/nscd/log cat < /etc/service/nscd/log/run #!/bin/bash svlogd -tt /var/log/nscd/ EOF mkdir /var/log/nscd chmod +x /etc/service/nscd/log/run chmod +x /etc/service/nscd/run ]]>

This section is obsolete. PAM is Linux's mechanism to do the following: decide if somebody is who they say they are (authentication; in our case via kerberos) set up sessions (in the case of AFS, this means creating PAGs) change passwords (in our case, changing the password in the KDC) Here's the usual PAM setup: /etc/pam.d/common-account: /etc/pam.d/common-auth: /etc/pam.d/common-password: /etc/pam.d/common-session: /etc/pam.d/login (Add to beginning of file): /etc/pam.d/ssh (Add just before @include common-auth line): If the machine is intended for user logins, DO NOT create /etc/login.restrict. If the machine is only intended for admin logins, then create the file /etc/login.restrict with the following contents:

This section is obsolete.

Insert these lines in /etc/ssh/ssh_config so that outbound ssh connections will always try to use Kerberos if available:

You will need to create a "host principal" for the new server; if you are setting up server.hcoop.net, then it must have the name Add this principal to the KDC like this (execute these commands on the new server, as root, while holding admin tickets): Then add these lines to the bottom of /etc/ssh/sshd_config: Finally, restart the ssh server:

This section is obsolete. Don't forget to give all of the admins lines in /etc/sudoers. Each line should look like:

This section is obsolete. /etc/cron.daily/hcoop-clean-tmp:

This section is obsolete.

FIXME: AdamM needs to fill this in

The runit package is a mechanism for starting, stopping, and monitoring daemons. It is an alternative to the traditional /etc/init.d and start-stop-daemon scheme. Its chief advantages are: It launches daemons with clean process state; the daemon inherits nothing from the administrator invoking the start/stop command because the daemon is not forked as a child of the administrator's shell (rather, a request is sent runit daemon asking it to fork the daemon). This is very important when dealing with tokens and pags. Runit monitors the processes that it forks, and restarts them if they die. Runit eliminates the need for pidfiles and the associated risk of starting multiple copies of a daemon. Runit captures the daemon's stdout and either sends it to a logger (if specified) or else displays it in the process name (output of ps) When you move a process from /etc/init.d/ control to runit supervision, you should inform debian that you have done so: This will cause invocations of /etc/init.d/script {start|stop} to do "the right thing".

You can install the dnscache package to make the server self-sufficient for dns resolution purposes (it acts as a tiny dns server just for localhost). This improves the reliability of the overall infrastructure. Starting dnscache via runit is often a good idea; this ensures that it starts early in the boot process and that it is restarted if it dies for any reason. Here are the instructions for configuring it. Make sure that bind9 (if running) is only listening to 127.0.0.1 and the public IP address of the machine. We tell dnscache to listen on 127.0.0.2 so as to avoid conflicts with bind. Then modify /etc/resolv.conf, replacing the nameserver lines with:

If not present already: /etc/hosts]]>

Life is simpler when you run ssmtp. You can direct the mail stream either to deleuze (preferred) or to a copy of exim running locally (but why bother running it?). Be sure to enable FromLineOverride, which ships defaulted to "off" in Debian.

By default, Linux will write to the disk in order to update the atime ("access time") every time a file is read from; this substantially degrades performance. You can disable this behavior by editing /etc/fstab /dev/hda1 / ext3 defaults,noatime,errors=remount-ro 0 1]]>This is especially important on filesystems which are used to store AFS volumes.

Debian's installer seems to want to put an entry for the machine's own hostname in /etc/hosts, resolving to 127.0.0.1. You'll probably want to remove it. CategorySystemAdministration CategoryHistorical