welcome: please sign in

Diff for "DomTool/UserGuide"

Differences between revisions 4 and 10 (spanning 6 versions)
⇤ ← Revision 4 as of 2006-12-17 19:47:34 →
Size: 9586
Editor: AdamChlipala
Comment: Nested configuration directives
← Revision 10 as of 2007-07-03 00:50:40 → ⇥
Size: 14950
Editor: MichaelOlson
Comment: Use new nameservers
Deletions are marked like this. Additions are marked like this.
Line 21: Line 21:
You will edit a text file with the configuration directives you want applied your domain. Where `$USER` is your HCoop username and `$DOMAIN` is your domain name, you should write configuration into a file `/afs/hcoop.net/usr/$USER/domtool/$DOMAIN`. If you're logged into one of our servers, `cd ~/domtool` should get you to the proper directory. You will edit a text file with the configuration directives you want applied your domain. Where `$USER` is your HCoop username and `$DOMAIN` is your domain name, you should write configuration into a file `/afs/hcoop.net/usr/$USER/.domtool/$DOMAIN`. If you're logged into one of our servers, `cd ~/.domtool` should get you to the proper directory.
Line 38: Line 38:
Now, enough talk; let's execute our source file! Assuming that you're in your `domtool` directory, run{{{ Now, enough talk; let's execute our source file! Assuming that you're in your `.domtool` directory, run{{{
Line 45: Line 45:
which calls the dispatcher with '''all''' configuration files in your `domtool` directory, regardless of what your current working directory is. This can be a helpful shortcut sometimes, though it can be noticeably less efficient than calling `domtool` with the single domain that you know you've changed. which calls the dispatcher with '''all''' configuration files in your `.domtool` directory, regardless of what your current working directory is. This can be a helpful shortcut sometimes, though it can be noticeably less efficient than calling `domtool` with the single domain that you know you've changed.
Line 47: Line 47:
In fact, the names of the source files in your `domtool` directory don't matter. We recommend storing the configuration for each of your domains in a separate file named after that domain, but you might want to make different choices. For instance, you might want to use an abstraction that configures multiple domains at once. However, it '''is''' important that you keep all your permanent configuration files in your `domtool` directory. Sometimes we need to reprocess all configuration from scratch, and in such cases our dispatcher will only look for source files in your `domtool` directory. For temporary experimentation, though, you can feel free to store source files elsewhere and run them explicitly with `domtool $FILENAME`. In fact, the names of the source files in your `.domtool` directory don't matter. We recommend storing the configuration for each of your domains in a separate file named after that domain, but you might want to make different choices. For instance, you might want to use an abstraction that configures multiple domains at once. However, it '''is''' important that you keep all your permanent configuration files in your `.domtool` directory. Sometimes we need to reprocess all configuration from scratch, and in such cases our dispatcher will only look for source files in your `.domtool` directory. For temporary experimentation, though, you can feel free to store source files elsewhere and run them explicitly with `domtool $FILENAME`.

= Debugging configuration files =

To check a configuration file without actually publishing its results, run:{{{
domtool -tc $FILENAME}}}

Here, `-tc` stands for "type-check." The Domtool language is designed to capture all rules of which configuration is valid and which isn't in its parsing and typing rules, so using the `-tc` option ''should'' enable you to find any bugs in your configuration. We might have bugs in our implementations of the configuration directives from time to time, but `-tc` should be sufficient to help you find your own bugs.

Note that using your `~/.domtool` directory as a scratch space with lots of stray files is a bad idea, especially if you have multiple files representing different alternatives for configuring the same domain. Sometimes we make a change to Domtool that requires reprocessing all user configuration. In such cases, we run every file found in any user's `~/.domtool` directory. In that case, your `~/.domtool` files will be run in an arbitrary order, including any files that you have been thinking of as "test cases." It just might happen that the "test cases" end up overriding the "real" files. Also, if ''any'' file in your `~/.domtool` fails to type-check, we will unpublish ''all'' of your domains automatically. In summary, use another directory for storing tests, and run them by specifying explicit filename arguments to `domtool`, possibly with the `-tc` option.
Line 85: Line 94:
       dns (dnsNS "ns.hcoop.net");
        dns (dnsNS "ns2.hcoop.net");		   dns (dnsNS "ns1.hcoop.net");
  dns (dnsNS "ns3.hcoop.net");
Line 88: Line 97:
       dns (dnsA "www" web_ip);   dns (dnsA "www" web_ip);
Line 90: Line 99:
       handleMail;
       catchAllAlias "$USER";		   handleMail;
  catchAllAlias "$USER";
Line 93: Line 102:
       vhost "www" with
        end;		   vhost "www" with
  end;
Line 110: Line 119:

= Configuration contexts =

Not all configuration directives make sense in all contexts. For instance, it doesn't make sense to specify a URL rewrite rule outside of a `vhost` directive. The DomTool language uses a concept of '''contexts''' or '''predicates''' to enforce correct usage of directives.

To illustrate, let's try breaking the rule we just gave in our example:
{{{dom "hcoop.net" with
  alias "doc/" "/usr/local/doc"
end;}}}

We ask for the URI prefix `doc/` to be mapped to file path `/usr/local/doc`. The `alias` directive is only allowed inside `vhost`s, so we get an error like this:
{{{hcoop.net:0.0-3.4:error: Context incompatibility for nested action.
Have: Domain
Need: Vhost}}}

This one's simple enough. We're in a `Domain`, but we need to be in a `Vhost` to use that directive. In full generality, DomTool's context specification language is a small logic that can be used to express much more complicated conditions, while maintaining the ability for the interpreter to check proper usage automatically. See the DomTool/LanguageReference for a full specification.

= Environment variables =

DomTool has its own concept of typed '''environment variables''' that are used to tweak parameters of configuration directives. Here's an example to demonstrate how they're used and why they're useful.

Running this configuration:
{{{vhost "www" with
end;}}}
gives you an Apache virtual host that serves documents out of `~/public_html`. If you wanted to serve documents out of `$DIR` instead, you could run:
{{{vhost "www" where
  DocumentRoot = "$DIR"
with
end;}}}
Maybe you want to use the same document root for two different virtual hosts. In that case, you could use:
{{{let
  DocumentRoot = "$DIR"
in
  vhost "www" with
  end;

  vhost "www2" with
  end;
end;}}}
Both of these alternatives involve assigning a new value to the '''environment variable''' `DocumentRoot` of type `your_path`, the type of filesystem paths that you may reference in configuration. Both `where` and `let` only introduce ''local values'' for environment variables. That is, once the `let` or the expression with the `where` clause has finished, it undoes changes to the environment made in the `where` clause or the first part of the `let`. This helps you maintain neat code that doesn't rely too much on global state.

As this example hopefully shows, environment variables are useful because they make it more pleasant to use highly-customizable configuration directives. The less-commonly-configured aspects of a directive can be taken in as environment variables rather than normal arguments. That way, you don't need to write any code to deal with them when you are happy with their default values. When you ''do'' want to set a non-standard value, you can use a construct like `let` to apply it to multiple directives at once.

= Further reading =

We're purposely cutting off this user guide at this shallow level of detail. The best thing to read next is the ["DomTool/Examples"], which should introduce by example all of the language features and configuration directives that you're likely to use. If you want to understand the type error messages that the interpreter generates as more than just "there is a problem somewhere," then you'll probably want to learn a bit more about the language.

For a more in-depth understanding of DomTool, including how to build and use your own directives, see the DomTool/LanguageReference. The [http://deleuze.hcoop.net/domtool/ standard library reference] presents all of the standard directives with full DomTool type information. We've glossed over details of the type system in this guide, but members comfortable with ML or Haskell programming will probably find this formal documentation the easiest to use, since types express very succinctly how directives may be used.

The main DomTool page links to all of these resources and several more.

TableOfContents()

1. Overview

HCoop is almost unique in the history of the Internet. We are trying to provide highly configurable Internet hosting to the general public while maintaining a high level of security, so that your services keep running and your data stays safe and private. You can probably see that these goals conflict in a number of ways! Since our organization's birth in 2002, we've been evolving tools to help us reconcile these different goals. This document introduces the primary element of our current solution.

This solution is called DomTool, and here's the one-line summary of what it's all about:

  • DomTool enables HCoop members to use a domain-specific programming language to describe how to configure daemons spread across our servers.

Some other cogent properties of DomTool:

  • The language is a [http://en.wikipedia.org/wiki/Statically_typed statically-typed], [http://en.wikipedia.org/wiki/Purely_functional purely functional] programming language. Other functional programming languages that you might have heard of include Lisp, Scheme, ML, and Haskell. Haskell and ML are statically-typed, and Haskell is purely functional.

  • The language has a rich and extensible type system that can enforce complicated security policies through type-checking.

  • Configuration works via a distributed client/server architecture, so, with the right software installed, you can configure your domains from anywhere.

Some readers may be scared off by this level of technical jargon. We apologize, but we just couldn't help showing it off. ;-) The rest of this document will be aimed at the average member, assuming only a solid understanding of the basics of Linux and the Internet. In particular, we assume no experience with particular kinds of programming.

2. Configuration files

In this example, we'll assume that you've requested and been granted privileges to some Internet domain name (like you.com) via our [https://members.hcoop.net/portal/domain web portal]. If you don't want to configure a domain that you own, then you don't need to be using DomTool!

You will edit a text file with the configuration directives you want applied your domain. Where $USER is your HCoop username and $DOMAIN is your domain name, you should write configuration into a file /afs/hcoop.net/usr/$USER/.domtool/$DOMAIN. If you're logged into one of our servers, cd ~/.domtool should get you to the proper directory.

We leave it up to you to choose how to edit the source file. You might run something like emacs $DOMAIN over SSH, or you might open it locally if you have our AFS cell mounted in your local filesystem.

In this file $DOMAIN, you can write: {{{dom "$DOMAIN" with end;}}}

You should replace $DOMAIN with the actual domain name. This example introduces the simplest useful configuration, based on the dom directive. What does this source file accomplish? It:

  • Declares our nameservers as authoritative for your domain.
  • Declares that Exim should handle e-mail for your domain.
  • Directs Exim to send all mail to your domain to your mailbox.

  • Adds a DNS mapping from www.yourdomain.com to our web server's IP address.

  • Creates an Apache virtual host at that hostname serving ~$USER/public_html/.

There are more primitive configuration directives to set up each of these features individually. dom packages all of the functionality together in an easy-to-use package. One of the main principles behind DomTool and our use of it is to avoid configuration repetition wherever possible. The DomTool language contains several abstraction features familiar from high-level programming languages, as well as some new ones, all in support of abstracting common code patterns into programming language entities that can be called multiple times. In general, if you find yourself writing the same code over and over again, let us know, and we'll try to add a new abstraction to our standard library! You could even write such an abstraction yourself, but that's beyond the scope of this intro document.

Now, enough talk; let's execute our source file! Assuming that you're in your .domtool directory, run

domtool $DOMAIN

This contacts our central dispatcher server over an SSL connection. Once the server verifies that you are who you say you are, it will publish your configuration to the affected daemons, and you should be able to start using your domain.

You could also simply run

domtool

which calls the dispatcher with all configuration files in your .domtool directory, regardless of what your current working directory is. This can be a helpful shortcut sometimes, though it can be noticeably less efficient than calling domtool with the single domain that you know you've changed.

In fact, the names of the source files in your .domtool directory don't matter. We recommend storing the configuration for each of your domains in a separate file named after that domain, but you might want to make different choices. For instance, you might want to use an abstraction that configures multiple domains at once. However, it is important that you keep all your permanent configuration files in your .domtool directory. Sometimes we need to reprocess all configuration from scratch, and in such cases our dispatcher will only look for source files in your .domtool directory. For temporary experimentation, though, you can feel free to store source files elsewhere and run them explicitly with domtool $FILENAME.

3. Debugging configuration files

To check a configuration file without actually publishing its results, run:

domtool -tc $FILENAME

Here, -tc stands for "type-check." The Domtool language is designed to capture all rules of which configuration is valid and which isn't in its parsing and typing rules, so using the -tc option should enable you to find any bugs in your configuration. We might have bugs in our implementations of the configuration directives from time to time, but -tc should be sufficient to help you find your own bugs.

Note that using your ~/.domtool directory as a scratch space with lots of stray files is a bad idea, especially if you have multiple files representing different alternatives for configuring the same domain. Sometimes we make a change to Domtool that requires reprocessing all user configuration. In such cases, we run every file found in any user's ~/.domtool directory. In that case, your ~/.domtool files will be run in an arbitrary order, including any files that you have been thinking of as "test cases." It just might happen that the "test cases" end up overriding the "real" files. Also, if any file in your ~/.domtool fails to type-check, we will unpublish all of your domains automatically. In summary, use another directory for storing tests, and run them by specifying explicit filename arguments to domtool, possibly with the -tc option.

4. Permissions

Now let's put on our Evil Hacker from the Seventh Circle of Hell hats. If you write this to a file hcoop.net: {{{dom "hcoop.net" with end;}}} and run:

domtool hcoop.net

you should see an error message like: {{{hcoop.net:0.0-1.14:error: Function argument has wrong type.

  • Expression: "hcoop.net"

Actual type: string Needed type: your_domain}}}

What this is saying is that you are only allowed to use dom with domains that you are allowed to configure. You tried to configure hcoop.net, which is not one of those domains, and so is treated like an arbitrary string (sequence of characters). The type checker has saved the day, and the Evil Hacker is prevented from mucking with hcoop.net configuration.

How exactly does DomTool determine which domains you're allowed to configure? It uses a general permissions system based on access control lists. You can list all of your permissions by running:

domtool-admin perms

You should see output like this: {{{Permissions for you: domain: you.com you.net you.org group: you path: /afs/hcoop.net/usr/you user: you}}} where you stands for your username. The domain list gives the Internet domains to which you've been granted configuration rights. user and group list the UNIX users and groups as whom you may run programs, and path gives the filesystem paths that you're allowed to reference in your configurations. You have rights to all subdirectories of path entries, too.

You might like to perform some other queries on the permissions database, too. For instance, you might like to know which member owns someone.com. You could run:

domtool-admin whohas domain someone.com

and hopefully get back a reply like:

whohas domain / someone.com: someone

In general, domtool-admin whohas $CLASS $VALUE will list every user who has $VALUE in the $CLASS row of his permissions table.

5. Nested configuration directives

Let's look under the hood at what dom is doing by writing an equivalent configuration file that doesn't use it. {{{domain "$DOMAIN" with

  • dns (dnsNS "ns1.hcoop.net"); dns (dnsNS "ns3.hcoop.net"); dns (dnsA "www" web_ip); handleMail; catchAllAlias "$USER"; vhost "www" with end;

end;}}}

Now we see what that with..end syntax was about: it relates to nested configuration directives. Both dom and the more primitive domain directive take additional configuration specific to the named domain. Everything we include here inside domain could also have been included inside a dom use, though it would be redundant.

The contents of the domain block are a series of DomTool language expressions that evaluate to actions. Each action is given using consistent programming language syntax, rather than the ad-hoc configuration syntaxes used by daemons like Apache and Exim. This means that the directives are sometimes slightly more verbose, but they are much easier for humans and machines to parse, since no directive-specific ad-hoc syntax information is required.

Let's step through the nested directives one by one to explain their meanings.

The dns lines register DNS mappings to be included in $DOMAIN's zone. dnsNS records list the authoritative nameservers in order. dnsA records provide mappings from hostnames to IP addresses. web_ip is a variable containing the IP address of our default web server for member use, defined in the standard library.

handleMail asks Exim to provide relaying for any e-mail message addressed to any address at $DOMAIN.

catchAllAlias "$USER" directs Exim to deposit any mail addressed to any user at $DOMAIN in $USER's mailbox.

The vhost directive demonstrates several layers of nested configuration. It declares an Apache virtual host named www.$DOMAIN. We could have included further Apache-specific configuration inside the vhost directive.

6. Configuration contexts

Not all configuration directives make sense in all contexts. For instance, it doesn't make sense to specify a URL rewrite rule outside of a vhost directive. The DomTool language uses a concept of contexts or predicates to enforce correct usage of directives.

To illustrate, let's try breaking the rule we just gave in our example: {{{dom "hcoop.net" with

  • alias "doc/" "/usr/local/doc"

end;}}}

We ask for the URI prefix doc/ to be mapped to file path /usr/local/doc. The alias directive is only allowed inside vhosts, so we get an error like this: {{{hcoop.net:0.0-3.4:error: Context incompatibility for nested action. Have: Domain Need: Vhost}}}

This one's simple enough. We're in a Domain, but we need to be in a Vhost to use that directive. In full generality, DomTool's context specification language is a small logic that can be used to express much more complicated conditions, while maintaining the ability for the interpreter to check proper usage automatically. See the DomTool/LanguageReference for a full specification.

7. Environment variables

DomTool has its own concept of typed environment variables that are used to tweak parameters of configuration directives. Here's an example to demonstrate how they're used and why they're useful.

Running this configuration: {{{vhost "www" with end;}}} gives you an Apache virtual host that serves documents out of ~/public_html. If you wanted to serve documents out of $DIR instead, you could run: {{{vhost "www" where

with end;}}} Maybe you want to use the same document root for two different virtual hosts. In that case, you could use:

DomTool/UserGuide (last edited 2021-02-24 02:49:10 by ClintonEbadi)