This page explains some nuances of the Andrew File System (AFS), which we use to serve home directories.
Contents
Basic Architecture
Using the shared filesystem involves a combination of Kerberos and OpenAFS.
File conventions
The /afs tree contains shared filesystems. /afs/hcoop.net (symlinked from /afs/hcoop as well) is our piece of the AFS-o-sphere. Subdirectories include:
/afs/hcoop.net/user, the home of home directories
/afs/hcoop.net/user/U/US/$USERNAME, $USERNAME's home directory
/afs/hcoop.net/common/etc, the home of non-platform-specific fun stuff like DomTool
Connecting to AFS from an HCoop server
I found this handy summary of the commands that must be run:
On our servers, it seems sufficient to run:
kinit aklog
These should be run automatically if you log in normally, but admins who manually kinit to different users (for the purpose of testing access permissions most often), need to of course run both kinit; aklog to completely switch to a target user.
The kadmin shell
All Kerberos administration commands are run from a special shell, called Kadmin. There are two variants of Kadmin: kadmin is the usual, remote version of the command which can be run on any machine; kadmin.local is the "local" version which can only be ran on the AFS fileserver (deleuze).
Invoke kadmin.local as sudo kadmin.local -p YOURUSERNAME_admin. It is good to include "-p YOURUSERNAME_admin", or kadmin will "authenticate" as the first user it finds in the ticket cache, which may or may not be the username you expected. All the administrative commands would work anyway (since you ran kadmin.local), but an incorrect principal name would make various statistics incorrect (like name of principal who was adding/changing entries in the DB).
To invoke kadmin, use kadmin -p YOURUSERNAME_admin. In normal course of action, kadmin asks for a password. This is impractical for automated scripts. As usual, instead of a password, you can also pass a keytab file. Our keytabs are saved in /etc/keytabs/ on each system, and they are readable by group 'wheel'. So administrators should be able to invoke 'kadmin' (use control shell) or kinit/k5start (impersonate any user) by supplying target user's key from a keytab, such as kadmin -p domtool -k -t /etc/keytabs/domtool .
Creating a new user
We follow the convention that Kerberos users for daemons are named $DAEMON, where $DAEMON is the name of the daemon (for instance, the name of system user it runs as, or the name of its /etc/init.d file). Some daemons currently use DAEMON/HOST scheme, but this will be changed later and is not to be used for any new principals you create.
To add the Kerberos principal for a daemon, run this in kadmin:
addprinc -randkey -policy service $DAEMON
AFS users exist separately from Kerberos principals. To add the AFS user for a daemon to which you want to assign UID $UID, run:
pts createuser $DAEMON
"keytab" files smooth the way to running daemons that run with AFS privileges. An access-protected local file contains a user's credentials, and daemons read these files on starting up in order to authenticate.
To create a keytab for a daemon, run this in kadmin:
ktadd -k /etc/keytabs/$DAEMON -e "des3-hmac-sha1:normal rc4-hmac:normal" $DAEMON chown $DAEMON:wheel /etc/keytabs/$DAEMON chmod 440 /etc/keytabs/$DAEMON
In the example above, only one key (of 4 or 5 created) is exported for a user. Sometimes it might be desirable to only export a specific key into a keytab file, but we generally just omit the -e KEY_TYPE parameter and export all keys to the keytab file.
You can view keys stored in a keytab by doing sudo klist -k /etc/keytabs/KEYTAB_FILE.
To make daemons properly kinit/aklog as the user you created for them, use k5start command. Many examples of how to use it are already found in our /etc/init.d/ scripts. Important options include -U (which kinits as the first principal found in the keytab file, without the need to explicitly name a principal), -f (which specifies the keytab file to kinit from), and -K MINUTES (which re-news the ticket after MINUTES, so that daemons can run for long periods of time).
To give $DAEMON the actual permission in AFS space, for most common actions, fs setacl DIR $DAEMON read or write are good. All subdirectories that get created within the toplevel directory for which you give permissions, will inherit all the permissions.
Listing and setting quotas
To list volume quota, run
fs lq DIR
To set volume quota in 1-kilobyte blocks, run
fs sq DIR -max SIZE