#pragma section-numbers off

In 2007, at the time of switch to Peer1 colocation provider and expanding our infrastructure, we decided to use AFS (OpenAFS) as the basis for our technical setup.

AFS is, strictly speaking, just a distributed filesystem, but it mandates usage of Kerberos and has a whole set of its own rules. Since we have decided to keep all our data files in AFS, the config and init scripts of most (if not all) services had to be modified to support AFS.

We have configured most traditional Unix services, DomTool, and mail delivery/access to use AFS, and where possible, services fork processes under corresponding user privileges and obtain users' AFS identity.

<<TableOfContents>>

= Basic Architecture =

Using the shared filesystem involves a combination of Kerberos and OpenAFS.

= File conventions =

The `/afs` tree contains shared filesystems.  `/afs/hcoop.net` is our piece of the AFS-o-sphere, but is not (yet) listed in the global CellServDB. See [[http://docs.openafs.org/AdminGuide/HDRWQ44.html|about volumes]] in the openafs docs for information on the standard naming scheme for volumes when adding new volumes

Subdirectories include:

 * `/afs/hcoop.net/user`, the home of home directories
 * `/afs/hcoop.net/user/U/US/$USERNAME`, `$USERNAME`'s home directory
 * `/afs/hcoop.net/old/user/U/US/$USERNAME`, `$USERNAME`'s daily backup of their home directory
 * `/afs/hcoop.net/common/etc`, the home of non-platform-specific fun stuff like DomTool

= Connecting to AFS =

Upon login (which goes through PAM krb5 and afs modules), Kerberos ticket and AFS token should automatically be initialized for the user, and they should find themselves in their home directory.

Users wishing to manually authenticate can run '''kinit''' and '''aklog''' (see manpages for all options) to obtain ticket and token, and '''klist -5f''' and '''tokens''' to verify.

Also, AFS is a distributed filesystem and allows access from users' workstations. Using '''kinit''' and '''aklog''' cmdline switches, one can remotely authenticate to cell HCOOP.NET and then directly SSH to HCoop without a password, or better yet, access their home  directory from their local workstation, in `/afs/hcoop.net/user/U/US/$USERNAME`.

= Users and tokens =

Every HCoop user "owns" a Kerberos principal and AFS PTS entry named after their username. This "account" is intended to be used only interactively (people using it).

For each, there's also another principal named "$USER/daemon" in Kerberos (and "$USER.daemon" in AFS). This principal's key is exported into file `/etc/keytabs/user.daemon/$USER` on all relevant machines and is chown-ed to the user's Unix account. This allows users' batch/noninteractive scripts to authenticate to Krb/AFS using password from a file.

This also allow for more fine-grained control as permissions need to be explicitly granted to $USER.daemon in order to do anything with the data. So even if the service running under certain Unix user (or root!) is compromised, the attacker's choice of action will be minimal.

Furthermore, user tickets and tokens expire periodically. One has to either invoke kinit/aklog again, or set up tools such as '''k5start''' to perform automatic renewal.

= Privileges =

AFS uses ACLs, a more elaborate permissions model than the traditional Unix rwx modes. (Although the advantage is not that great any more, with the availability of POSIX ACLs for Linux).

However, there are a few intrinsic AFS properties that must be mentioned:

 1. AFS ACLs are per directory. All contained files inherit directory's ACL. (A subdirectory can define its own ACLs, of course).
 1. When a subdirectory is created, it inherits ACL of its parent. (Much better approach than as with Unix filesystems where you need +s on the immediate parent directory to get this behavior).
 1. It's possible to make user files unreadable to an attacker, even if they break in the "root" account on the machine

== Permissions and quota ==

To give $USER.daemon the actual permission in AFS space, for most common actions, `fs setacl DIR $USER.daemon read` or `write` 
are good. All subdirectories that get created within that toplevel directory for which you give permissions, will, as said,
inherit all the permissions, and this is what you want in 99% of the cases.

== Listing and setting quotas ==

To list volume quota, run{{{
fs lq DIR
}}}

To set volume quota in 1-kilobyte blocks, run{{{
fs sq DIR -max SIZE
}}}

= Problems =

HCoop members have so far reported the following problems with AFS:

 * They can not access files (Permission denied). This happens when their ticket/token expires in a long-running SSH session, or (most notably) when they detach a SCREEN session and return later. Solution is to manually run kinit/aklog or have k5start running in the background.
 * They can not access files (Timed out). Sometimes the volume is marked as needing salvage and becomes inaccessible. It is needed to run "vos salvage" on the user volume (not the whole partition!).
 * They can not access files (Timed out). Sometimes this is due to perceived inaccessibility of the fileserver. It helps if one runs '''fs checks; fs checkv'''.
----
CategorySystemAdministration CategoryMemberManual