welcome: please sign in

The following 878 words could not be found in the dictionary of 7 words (including 7 LocalSpellingWords) and are highlighted below:
ability   able   about   abstracting   abstraction   access   accomplish   across   action   actions   Actual   actual   actually   ad   add   additional   address   addressed   adds   Adds   Admin   admin   affected   afs   Afs   after   again   aimed   Alias   alias   all   allowed   allows   almost   also   alternatives   always   an   analysis   and   another   answer   any   anymore   anything   anywhere   Apache   apologize   applied   apply   appropriate   arbitrary   architecture   are   argument   arguments   as   As   ask   asks   aspects   assigning   assume   Assuming   assuming   at   authoritative   automatically   average   avoid   back   bad   based   basics   Be   be   because   been   before   behind   being   Besides   best   beyond   birth   bit   block   blocks   both   Both   Break   breaking   bugs   build   but   by   called   calling   calls   can   capture   case   cases   Category   cd   cell   central   change   changed   changes   character   characters   check   checker   checking   checks   choices   choose   churning   Circle   clause   client   code   cogent   com   come   comfortable   coming   command   common   commonly   complicated   concept   conditions   configurable   configuration   Configuration   configurations   configure   configured   configures   configuring   conflict   connection   consistent   construct   contacts   contains   content   contents   Contents   context   Context   contexts   control   Coop   correct   could   couldn   Create   create   Creates   creating   current   custom   customizable   cutting   daemon   daemons   data   database   day   deal   Debugging   declares   Declares   Default   default   define   defined   definition   definitions   demonstrate   demonstrates   dependency   deposit   depth   describe   described   designed   detail   details   determine   differ   different   direct   directive   directives   directories   directory   Directs   directs   dispatcher   distributed   dns   do   doc   document   Document   documentation   documents   does   doesn   doing   Dom   dom   Domain   domain   domains   Domtool   domtool   don   down   dtl   each   Each   easier   easiest   easy   edit   efficient   element   elsewhere   emacs   en   enable   enables   end   enforce   enough   entities   entries   environment   Environment   equivalent   error   especially   evaluate   even   every   everyone   Everything   Evil   evolving   exactly   example   Examples   execute   Exim   expanded   experience   experimentation   experts   explain   explicit   explicitly   express   Expression   expression   expressions   extensible   facility   fact   factory   fail   false   familiar   Fast   features   feel   file   filename   files   filesystem   final   find   Fingers   finish   finished   first   following   For   for   form   formal   format   formats   found   free   from   front   full   function   Function   functional   functionality   functions   further   Further   gave   general   generality   generates   get   Getting   given   gives   Giving   global   glossed   gmail   go   goals   granted   guide   Guide   Hacker   had   handle   happen   happy   has   Haskell   hats   have   Have   haven   having   hcoop   heard   Hell   help   helpful   helps   here   Here   high   highly   his   history   hoc   home   home1   home2   hood   hopefully   host   hosting   hostname   hosts   house   how   How   However   html   http   https   humans   icons   idea   if   If   illustrate   implementations   implemented   important   in   In   include   included   including   incompatibility   individually   information   inside   installed   instance   instead   Internet   interpreter   into   intro   introduce   introduces   involve   is   isn   it   It   its   jargon   just   keep   kinds   know   language   Language   languages   latter   layers   learn   leave   less   let   Let   lets   level   lib   library   like   likely   line   lines   links   Linux   Lisp   list   lists   ll   local   locally   logged   logic   longer   look   lots   lucky   machine   machines   made   mail   Mail   mailbox   mailing   main   maintain   maintaining   make   Manual   many   Many   mapped   mapping   mappings   matter   may   Maybe   me   meanings   means   Member   member   members   mention   mentions   message   messages   might   more   mounted   much   mucking   multiple   my   mydomain   myhouse   name   named   names   nameserver   nameservers   native   neat   need   Need   Needed   nested   Nested   net   new   next   no   non   none   normal   Not   not   Note   noticeably   Now   ns1   ns2   number   Of   of   off   on   Once   once   One   one   ones   only   open   option   or   order   orderer   org   organization   Other   other   our   out   output   outside   over   overall   overriding   Overview   own   owns   package   packages   page   pages   parameters   parse   parsing   part   particular   path   paths   patterns   perform   permanent   permission   permissions   Permissions   perms   persist   Php   php56   places   pleasant   please   plug   point   policies   portal   possible   possibly   post   predicates   prefix   prefixes   presents   prevented   primary   primitive   principles   private   privileges   probably   problem   processes   programming   programs   promotes   proper   properties   provide   public   publish   published   publishing   Purely   purely   purposely   put   putting   queries   quite   rather   re   read   readers   reading   real   recommend   reconcile   redundant   Reference   reference   regardless   register   related   relates   relaying   reload   rely   Remember   Remove   Removing   repetition   replace   reply   representing   reprocess   reprocessing   request   requested   required   requires   resources   rest   result   resulting   results   reuse   rewrite   rich   right   rights   rmdom   root   Root   row   rule   rules   Run   run   running   Running   safe   same   save   saved   say   saying   scared   Scheme   scope   scratch   second   security   see   See   seeing   send   sense   sent   separate   sequence   series   serve   Server   server   servers   serves   services   serving   set   sets   setting   settings   Seventh   several   shallow   shared   shortcut   shortcuts   should   showing   shows   similar   simple   simplest   simply   since   Since   single   slightly   small   so   softlinks   software   solid   solution   Some   some   someone   something   Sometimes   sometimes   somewhere   source   space   special   specific   specification   specify   specifying   spread   standard   stands   start   Started   state   statically   Statically   stays   step   steps   store   storing   stray   string   stuff   subdirectories   subject   submitting   successfully   succinctly   such   sufficient   summary   support   sure   syntax   syntaxes   system   systems   table   Table   take   taken   takes   talk   tc   technical   techniques   temporary   terminology   test   tests   text   than   that   That   The   the   their   them   then   Then   There   there   these   they   thing   think   thinking   this   This   those   though   through   Throughout   time   times   to   To   together   too   Tool   tools   treated   tried   trouble   try   trying   tweak   two   type   typed   types   typing   under   underlying   understand   understanding   undoes   unique   up   us   usage   use   used   useful   user   User   username   users   uses   using   usr   val   valid   value   values   variable   variables   ve   verbose   verifies   Version   version   very   Vhost   vhost   via   virtual   want   wanted   was   way   ways   We   we   web   well   what   What   When   when   where   wherever   which   While   while   who   whohas   whom   why   wiki   wikipedia   will   with   without   work   working   works   would   write   Write   writing   wrong   www   www2   yo   You   you   Your   your   yourdomain   yourself   zone  

Clear message

DomTool / UserGuide

This is the DomTool User Guide. The properties of DomTool are described, as well as its use and its configuration file format.


HCoop is almost unique in the history of the Internet. We are trying to provide highly configurable Internet hosting to the general public while maintaining a high level of security, so that your services keep running and your data stays safe and private. You can probably see that these goals conflict in a number of ways! Since our organization's birth in 2002, we've been evolving tools to help us reconcile these different goals. This document introduces the primary element of our current solution.

This solution is called DomTool, and here's the one-line summary of what it's all about:

Some other cogent properties of DomTool:

Some readers may be scared off by this level of technical jargon. We apologize, but we just couldn't help showing it off. ;-) The rest of this document will be aimed at the average member, assuming only a solid understanding of the basics of Linux and the Internet. In particular, we assume no experience with particular kinds of programming.

Configuration files

In this example, we'll assume that you've requested and been granted privileges to some Internet domain name (like you.com) via our domain permission request form. If you don't want to configure a domain that you own, then you don't need to be using DomTool!

You will edit a text file with the configuration directives that you want applied your domain. Throughout this document, $USER is your HCoop username, $DOMAIN is your domain name, and you should write configuration into a file ~$USER/.domtool/$DOMAIN. If you're logged into one of our servers, cd ~/.domtool should get you to the proper directory.

Your home directory is something like /afs/hcoop.net/user/u/us/username, where you should replace username with your HCoop username and u and us with the one- and two-character prefixes of that username. Be sure that your .domtool directory has an AFS ACL set that allows the user domtool to read it, or your domains may go down when we need to reload everyone's configuration. If you end up with the wrong permissions, see MemberManual/GettingStarted/AfsExamples for the right command to run.

We leave it up to you to choose how to edit the source file. You might run something like emacs $DOMAIN over SSH, or you might open it locally if you have our AFS cell mounted in your local filesystem. In the latter case, those of you lucky enough to be using UNIX-based home systems will probably want to create softlinks from your local home directories to your HCoop home directories in AFS, to save some typing each time you want to open Domtool files.

In this file $DOMAIN, you can write:

dom "$DOMAIN" with

Remember, you should replace $DOMAIN with the actual domain name.

This example introduces the simplest useful configuration, based on the dom directive. What does this source file accomplish? It:

There are more primitive configuration directives to set up each of these features individually. dom packages all of the functionality together in an easy-to-use package.

One of the main principles behind DomTool and our use of it is to avoid configuration repetition wherever possible. The DomTool language contains several abstraction features familiar from high-level programming languages, as well as some new ones, all in support of abstracting common code patterns into programming language entities that can be called multiple times. In general, if you find yourself writing the same code over and over again, let us know, and we'll try to add a new abstraction to our standard library! You could even write such an abstraction yourself, but that's beyond the scope of this intro document.

Now, enough talk; let's execute our source file! Assuming that you're in your .domtool directory, run

domtool $DOMAIN

This contacts our central dispatcher server over an SSL connection. Once the server verifies that you are who you say you are, it will publish your configuration to the affected daemons, and you should be able to start using your domain.

You could also simply run


which calls the dispatcher with all configuration files in your .domtool directory, regardless of what your current working directory is. This can be a helpful shortcut sometimes, though it can be noticeably less efficient than calling domtool with the single domain that you know you've changed.

In fact, the names of the source files in your .domtool directory don't matter. We recommend storing the configuration for each of your domains in a separate file named after that domain, but you might want to make different choices. For instance, you might want to use an abstraction that configures multiple domains at once. However, it is important that you keep all your permanent configuration files in your .domtool directory. Sometimes we need to reprocess all configuration from scratch, and in such cases our dispatcher will only look for source files in your .domtool directory. For temporary experimentation, though, you can feel free to store source files elsewhere and run them explicitly with

domtool $FILENAME

Debugging configuration files

To check a configuration file without actually publishing its results, run:

domtool -tc $FILENAME

Here, -tc stands for "type-check." The Domtool language is designed to capture all rules of which configuration is valid and which isn't in its parsing and typing rules, so using the -tc option should enable you to find any bugs in your configuration. We might have bugs in our implementations of the configuration directives from time to time, but -tc should be sufficient to help you find your own bugs.

Note that using your ~/.domtool directory as a scratch space with lots of stray files is a bad idea, especially if you have multiple files representing different alternatives for configuring the same domain. Sometimes we make a change to Domtool that requires reprocessing all user configuration. In such cases, we run every file found in any user's ~/.domtool directory. In that case, your ~/.domtool files will be run in an arbitrary order, including any files that you have been thinking of as "test cases." It just might happen that the "test cases" end up overriding the "real" files. In summary, use another directory for storing tests, and run them by specifying explicit filename arguments to domtool, possibly with the -tc option.

Debugging for experts

Many of our members are quite familiar with the native configuration formats of standard UNIX daemons like Apache and BIND. For them, we provide a facility for seeing which "real" configuration is being sent to those daemons as a result of Domtool configuration files. To see all published configuration for $DOMAIN, run:

domtool-admin describe $DOMAIN

If you are having trouble coming up with the right configuration to accomplish something, and if you understand the configuration format of the underlying daemon, then please run domtool-admin describe before submitting a support request, to see if you'd have the same problem even if you had direct access to underlying daemon configuration.


Now let's put on our Evil Hacker from the Seventh Circle of Hell hats. If you write this to a file hcoop.net:

dom "hcoop.net" with

and run:

domtool hcoop.net

you should see an error message like:

hcoop.net:0.0-1.14:error: Function argument has wrong type.
 Expression: "hcoop.net"
Actual type: string
Needed type: your_domain

What this is saying is that you are only allowed to use dom with domains that you are allowed to configure. You tried to configure hcoop.net, which is not one of those domains, and so is treated like an arbitrary string (sequence of characters). The type checker has saved the day, and the Evil Hacker is prevented from mucking with hcoop.net configuration.

How exactly does DomTool determine which domains you're allowed to configure? It uses a general permissions system based on access control lists. You can list all of your permissions by running:

domtool-admin perms

You should see output like this:

Permissions for you:
domain: you.com you.net you.org
path: /afs/hcoop.net/user/y/yo/you
user: you

where you stands for your username. The domain list gives the Internet domains to which you've been granted configuration rights. user lists the UNIX users as whom you may run programs, and path gives the filesystem paths that you're allowed to reference in your configurations. You have rights to all subdirectories of path entries, too.

You might like to perform some other queries on the permissions database, too. For instance, you might like to know which member owns someone.com. You could run:

domtool-admin whohas domain someone.com

and hopefully get back a reply like:

whohas domain / someone.com: someone

In general, running

domtool-admin whohas $CLASS $VALUE

will list every user who has $VALUE in the $CLASS row of his permissions table.

Nested configuration directives

Let's look under the hood at what dom is doing by writing an equivalent configuration file that doesn't use it.

domain "$DOMAIN" with
  nameserver "ns1.hcoop.net";
  nameserver "ns2.hcoop.net";

  defaultAlias "$USER";

  web "www" with

Now we see what that with..end syntax was about: it relates to nested configuration directives. Both dom and the more primitive domain directive take additional configuration specific to the named domain. Everything we include here inside domain could also have been included inside a dom use, though it would be redundant.

The contents of the domain block are a series of DomTool language expressions that evaluate to actions. Each action is given using consistent programming language syntax, rather than the ad-hoc configuration syntaxes used by daemons like Apache and Exim. This means that the directives are sometimes slightly more verbose, but they are much easier for humans and machines to parse, since no directive-specific ad-hoc syntax information is required.

Let's step through the nested directives one by one to explain their meanings.

The nameserver lines register DNS mappings to be included in $DOMAIN's zone. In particular, we list the authoritative nameservers in order.

handleMail asks Exim to provide relaying for any e-mail message addressed to any address at $DOMAIN.

defaultAlias "$USER" directs Exim to deposit any mail addressed to any user at $DOMAIN in $USER's mailbox.

The www directive demonstrates several layers of nested configuration. It declares an Apache virtual host named www.$DOMAIN. web also adds the appropriate DNS mapping for this host. The nested directive serverAliasDefault asks to have this web host also answer to the name $DOMAIN, with no www in front.

Configuration contexts

Not all configuration directives make sense in all contexts. For instance, it doesn't make sense to specify a URL rewrite rule outside of a vhost directive. The DomTool language uses a concept of contexts or predicates to enforce correct usage of directives.

To illustrate, let's try breaking the rule we just gave in our example:

dom "hcoop.net" with
  alias "/doc" "/usr/local/doc"

We ask for the URI prefix doc/ to be mapped to file path /usr/local/doc. The alias directive is only allowed inside vhosts, so we get an error like this:

hcoop.net:0.0-3.4:error: Context incompatibility for nested action.
Have: Domain
Need: Vhost

This one's simple enough. We're in a Domain, but we need to be in a Vhost to use that directive. In full generality, DomTool's context specification language is a small logic that can be used to express much more complicated conditions, while maintaining the ability for the interpreter to check proper usage automatically. See the DomTool/LanguageReference for a full specification.

Environment variables

DomTool has its own concept of typed environment variables that are used to tweak parameters of configuration directives. Here's an example to demonstrate how they're used and why they're useful.

Running this configuration:

vhost "www" with

gives you an Apache virtual host that serves documents out of ~/public_html. If you wanted to serve documents out of $DIR instead, you could run:

vhost "www" where
  DocumentRoot = "$DIR"

Maybe you want to use the same document root for two different virtual hosts. In that case, you could use:

  DocumentRoot = "$DIR"
  vhost "www" with

  vhost "www2" with

Both of these alternatives involve assigning a new value to the environment variable DocumentRoot of type your_path, the type of filesystem paths that you may reference in configuration. Both where and let only introduce local values for environment variables. That is, once the let or the expression with the where clause has finished, it undoes changes to the environment made in the where clause or the first part of the let. This helps you maintain neat code that doesn't rely too much on global state.

As this example hopefully shows, environment variables are useful because they make it more pleasant to use highly-customizable configuration directives. The less-commonly-configured aspects of a directive can be taken in as environment variables rather than normal arguments. That way, you don't need to write any code to deal with them when you are happy with their default values. When you do want to set a non-standard value, you can use a construct like let to apply it to multiple directives at once.

Giving Your Fingers a Break

Domtool promotes code re-use using standard techniques from general programming. You can define shortcuts in one file in your ~/.domtool directory that you can then use from several other files. When you run domtool with no arguments, it checks which of your files mention shortcuts defined in other files, and it processes your files in the right order for this to work out right.

For instance, say that your home machine has IP address Maybe you find yourself creating many different DNS mappings to that IP address. You could put this code in ~/.domtool/lib.dtl:

val homeIP = \host -> dnsIP host "";

Then you could use your shortcut (called a "function" in Domtool and standard programming terminology) in another file in ~/.domtool:

dom "mydomain" with
  homeIP "home1";
  homeIP "home2"

You could also put the definition of homeIP in the same file as the use, at some point before that use but not inside any dom directive.

You can go further and create a factory for easy churning out of domains with similar configuration. Let's say that you have several domains that differ only in which directory you use to house the web pages for each one. Write this into ~/.domtool/lib.dtl:

val myDom = \name -> dom name where
  CreateWWW = false
  homeIP "myhouse";

  web "www" where
    ServerAdmin = "me@gmail.com"
    alias "/special" (home "shared_special_directory")

Now you can use your new directive like:

myDom "mydomain" where
  DocumentRoot = home "public_html/mydomain"

Besides all the standard stuff that dom sets up, myDom has given mydomain a DNS mapping from myhouse.mydomain to and a mapping from http://www.mydomain/special/ to ~/public_html/mydomain, a shared directory where you might be storing a standard set of icons or other content you want to reuse across domains. The DocumentRoot you set in the use of myDom is automatically used by the web directive, because it sets an environment variable.

You can go even further and write your own directives that takes blocks of configuration as arguments, and then plug them into the resulting configuration in the right places. Here's an expanded version of myDom that lets you specify custom configuration for the domain overall:

val myDom = \name -> \\for_domain : Domain -> dom name where
  CreateWWW = false
  homeIP "myhouse";

  web "www" where
    ServerAdmin = "me@gmail.com"
    alias "/special" (home "shared_special_directory")


Now you can use the new directive with a domain-specific DNS mapping:

myDom "mydomain" where
  DocumentRoot = home "public_html/mydomain"
  dnsIP "custom" ""

In fact, the dom directive is implemented in more or less the same way in ~domtool/lib/easy_domain.dtl.

You might also want to make some standard environment variable settings that will persist across your different configuration files. For instance, to make FastCGI based PHP version 5 the default for all of your domains, put this in lib.dtl:

PhpVersion = php56;

Now this will be the default environment variable setting in all of your domains. While domtool does dependency analysis to find the right order for running files that use functions defined in other files with val, it doesn't do this for environment variable settings, which is why it's important to put default environment variable settings in lib.dtl. The dependency orderer has a special case for files with this name, always putting them first.

One final request about this subject: if you come up with function definitions that you think would be useful to other members, post about them on the hcoop-help mailing list, and we may add them to our standard library with your permission.

Removing a domain

When you no longer want to host a domain $DOMAIN with HCoop, do the following:

The second step should fail if you haven't been granted permission to configure $DOMAIN. If you finish both steps successfully, then none of the shared HCoop daemons should serve anything related to $DOMAIN anymore.

Further reading

We're purposely cutting off this user guide at this shallow level of detail. The best thing to read next is the DomTool/Examples, which should introduce by example all of the language features and configuration directives that you're likely to use. If you want to understand the type error messages that the interpreter generates as more than just "there is a problem somewhere," then you'll probably want to learn a bit more about the language.

For a more in-depth understanding of DomTool, including how to build and use your own directives, see the DomTool/LanguageReference. The standard library reference presents all of the standard directives with full DomTool type information. We've glossed over details of the type system in this guide, but members comfortable with ML or Haskell programming will probably find this formal documentation the easiest to use, since types express very succinctly how directives may be used.

The main DomTool page links to all of these resources and several more.


DomTool/UserGuide (last edited 2021-02-24 02:49:10 by ClintonEbadi)