This is the chapter of the MemberManual that describes how to periodically run unattended commands, either as long-running processes (daemons) or as short periodic tasks from cron or at.

Introduction

All users' home directories in HCoop setup are located on AFS partitions. The use of AFS implies the use of Kerberos. In essence, your Kerberos (and AFS) "identity" is completely unrelated to your Unix username. While you do automatically obtain Kerberos and AFS identity (so-called "tokens") when you log-in to HCoop machines over ssh, be aware that Unix and Kerberos/AFS login are two separate things. That's why the scripts you run unattended cannot write (or read) files because, without extra steps taken, they do not have any useful identity or access privileges to partitions where all the relevant data is residing.

So, in general, when you want to access AFS space (that means any file in your home directory), you first need to authenticate with Kerberos to obtain a valid TGT ("Ticket-granting ticket"). As the name implies, the TG Ticket is then used in automatically obtaining futher tickets for access to your files (in AFS) or to specific services (such as to ssh, ftp, bugzilla or members portal).

The AFS "Login" Process

Following the above, here's the complete, "expanded" series of events that take place in a typical remote shell session:

1. You log in by providing your Unix username and password

2. You authenticate to Kerberos and obtain the TGT by running kinit. (Verify with klist -5).

3. You use the TGT to obtain AFS "token" by running aklog. (Verify with tokens).

4. You access files in the AFS space. Actual access privileges are determined by the combination of the token you are holding and the access control lists (ACLs) set on a directory. (List access rules with fs la DIRECTORY).

Interactive SSH process

Our SSH service is configured in such a way that your password is, in fact, the secret Kerberos key. So when you log in over SSH, steps 1 to 3 above are performed for you automatically and you can use AFS right away.

Non-interactive (Unattended) Processes

When a script is started in your Unix name by Cron, At or any other delayed/controlled-execution facility, no Kerberos ticket (or AFS token) is obtained automatically. Part of the reason lies in the fact that Kerberos' security model makes it almost impossible - even for root users - to authenticate as yourself if the password is not provided. (Where in Unix we would use "sudo" to easily impersonate any user, here it is impossible).

So the way to obtain Kerberos ticket and AFS token from unattended processes will be explained.

Ways of Obtaining AFS Tokens from Unattended Scripts

As hinted before, a password must be present to obtain any Kerberos identity. However, that password may come either from an interactive terminal, or from a file. (A file that is residing outside of the AFS space, of course!).

Kerberos discourages exporting of actual password keys into files, so at HCoop we create two Kerberos "identities" for each user: one named USER (your Unix username) for interactive sessions, and the other named USER.daemon for unattended sessions.

1. Your USER principal has the password saved only in the protected Kerberos database and it is not possible to obtain its ticket without providing the password.

2. Your USER.daemon principal has a very long random secret assigned to it and its key exported to a file named /etc/keytabs/user.daemon/USER. Your scripts will use the file /etc/keytabs/user.daemon/USER as a password in obtaining Kerberos/AFS identity "USER.daemon". In fact, all shared HCoop daemons also use that file to obtain permissions to write into your home directory (such as to deliver mail). Of course, Unix permissions and ownership on the keytab file are such that no other user can read your keytab file.

Token "scope"

Kerberos and AFS introduce a concept called Process Authentication Group ("PAG").

• If you obtain the Kerberos ticket and AFS token within the PAG, the tokens will apply only to the current process (usually a shell) and the processes started from it (its children).

• If you obtain the Kerberos ticket and AFS token outside the PAG, the tokens will apply to all processes running under your Unix username (well, to those that are not members of some existing PAGs, of course).

To "enter" a PAG, you start shell named /usr/bin/pagsh.openafs. With SSH, even though you find yourself in the shell of preference, a PAG is created for you just beforehand. (Verify by running id and noticing one numerical, untranslated entry such as 1105575254). Once within a PAG, there is basically no way to "escape" from it, so in effect, it is not possible to affect Kerberos/AFS identity of any of your other running processes by SSH-ing into a machine and kinitting as a different principal or obtaining different AFS tokens - they only apply to your current shell and its subprocesses.

In contrast, when unattended processes are started in your name, they are free of a PAG so you have the freedom of choice - influencing all "pagless" processes running under your Unix username, or starting pagsh manually and restricting influence to the current process and its children.

This quickly leads to two possible strategies:

1. Have one pagless process running which is refreshing USER.daemon token periodically (to keep it from expiring), and also run all scripts pagless - they will automatically find themselves to have that USER.daemon token.

2. Invoke all your scripts with shell /usr/bin/pagsh.openafs (can be in the #! "shebang" line) and obtain AFS token immediately after. Then rely on all subprocesses started from that script to inherit the obtained identity.

Unattended Access Privileges

In any case, using file /etc/keytabs/user.daemon/USER to obtain your Kerberos/AFS identity will "log you in" into AFS as USER.daemon, not USER. Therefore, make sure that all directories you want to access from unattended scripts have read or write permission for USER.daemon in addition to USER.

For example, here's how the permissions look for your ~/Maildir/ which gives USER.daemon write access:

$ fs la ~/Maildir/

Access list for /afs/hcoop.net/user/U/US/USER/Maildir/ is
Normal rights:
  system:administrators rlidwka
  USER rlidwka
  USER.daemon rlidwka

To give read permission on a directory, use

fs sa DIRECTORY USER.daemon read

To give write permission on a directory, use

fs sa DIRECTORY USER.daemon write

Note that we mention directory permissions only - in AFS, there are no file permissions. Directory permissions apply to all contained files, except subdirectories. Each subdirectory defines its own permissions. When new subdirectories are created, they inherit the ACL list from the parent directory. If you want to change ACLs on an existing directory tree, just use the fsr command in place of fs, with the same arguments.

Recipes and Examples

While you can use kinit to obtain tokens, we will use k5start in all examples. K5start is equivalent or better to kinit for all purposes.

Where are the said examples?

Creating the Directories Used in Examples

Create a directory that will contain your local executables and give USER.daemon read permission to it:

$ mkdir ~/bin/
$ fs sa ~/bin USER.daemon read

Make sure your ~/.bash_profile or equivalent file has ~/bin/ at the beginning of PATH, by enabling this within ~/.bash_profile:

# set PATH so it includes user's private bin if it exists
if [ -d ~/bin ] ; then
    PATH=~/bin:"${PATH}"
fi

Create a directory that will contain PID files for all your user processes:

$ mkdir ~/.run
$ fs sa ~/.run system:anyuser rl
$ fs sa ~/.run USER.daemon write

Script to Run a Particular Service Within Pagsh

Please be sure to also check RunningUnattendedCommandsWithoutRunInPagsh.

A wrapper script that runs a user-specified daemon is provided on Mire as run-in-pagsh. When calling it, the first argument should be a unique name with no spaces that describes the daemon, and then provide the command line for calling that daemon afterwards. Be sure to follow the instructions in the previous section for creating a ~/.run directory with correct permissions.

Here is an example of calling this script:

run-in-pagsh interch ~/bin/interchange

If you want to run a command from cron then use the following instead. The --fg argument tells run-in-pagsh that the command will run in the foreground, rather than the background.

run-in-pagsh --fg clean-mail ~/scripts/clean-mail

For the curious, the run-in-pagsh script may be viewed here.