This page is out of date and was only meant as a preview. See DomTool instead.

Herein lies an overview of the new configuration management tool that AdamChlipala is developing. Many things that could be included aren't here yet, and they'll be added as they seem to be important.

New members coming to this page: This system isn't yet rolled out. You probably want DomainTool instead.

1. Why something new?

The vast majority of configuration changes made at HCoop today go through the (already second-generation) DomainTool developed by AdamChlipala. We would have collapsed under the weight of the collective needs of our growing member base if we hadn't had something like Domtool to allow members to configure their own domains in a secure way. However, today's Domtool has some serious weaknesses. Here are a few in rough increasing order of seriousness.

• Extending Domtool is painful. The source code has some nice organizing principles, but still not very many. Adding a new feature tends to involve a hearty helping of boilerplate code. This doesn't have a big effect on anyone but Adam, but there's always the issue that more complicated coding increases the risk of bugs.

• It doesn't support configuration of multiple machines. Since we will be switching to running multiple machines (see ColocationPlans), this is a problem! The current Domtool could probably be patched to support this, but the other complaints make it not worth doing.

• The syntax and semantics of each configuration directive are ad-hoc. New directives are added by writing custom code to parse lines of input, check that they satsify any constraints necessary for security, etc.. Just like in programming, it's nice to have a common abstraction for the basic building blocks of, in this case, configuration files.

• There is no support for user-programmed abstractions. Nearly every subdirectory of /etc/domains has a .dns file with a shared prefix, declaring common elements like name servers. These should instead be calling a library function or otherwise using a reified object. All sorts of other patterns reappear frequently in Domtool configuration files and ought to be replaced with function calls.

2. Overview of Domtool 2

There are two main pieces to the Domtool 2 system: a programming language and a system for distributed configuration implemented around it.

2.1. The Language

The Domtool language is a statically-typed, pure functional programming language influenced by Haskell and ML. It uses the function as the basic unit of abstraction, both for describing primitive configuration directives and higher-level subroutines built from them.

Configuration itself as modeled as a stateful process of making a series of changes to the world, and this process is modeled inside the Domtool language with a configuration monad. The monad also includes the notion of typed environment variables that are a principled alternative to default function parameters, where it's possible to change defaults locally via lexical scoping.

Not all configuration directives are valid in all contexts. For instance, it only makes sense to specify that a file is a CGI script if you're working in the context of an Apache virtual host. Further, some Apache directives are only valid in the context of a particular directory. Values in the configuration monad are labeled with predicates in a tiny logic describing valid contexts for their use, and the type checker won't let you use a directive elsewhere.

The language interpreter also allows the addition of new base types by providing membership predicates implemented in SML. For instance, the standard library has a type for the name of a domain that the current user is allowed to configure. Rich types like this make it possible to use types of library functions as machine-checked documentation that often doesn't need to be supplemented with English.

2.2. Distributed Configuration

As the language allows primitive functions to be implemented externally in SML code, it's possible to build all sorts of systems on top of it. The one that we're interested in here is a system for changing configuration of network daemons running on multiple machines.

The basic system has three tiers.

The first tier is the members, merrily configuring away, laughing, crying, enjoying life's ups and downs, and sometimes attempting to configure things that they ought not to, through honest mistakes or (God forbid) malice. They write their configuration files as programs in the Domtool language and type check them locally to make sure that they will be accepted. If type checking succeeds, the scripts are sent over our LAN to a daemon process living on our main server.

This daemon type-checks the files again (since we don't want to have to trust users to do it right!) and, assuming they check out, executes them to modify a directory tree on our shared AFS filesystem. This tree looks a lot like /etc/domains does now, except that it's populated with summaries outputted by the daemon on how other daemons should behave.

At this point, we reach the final tier. A given set of changes may affect any subset of the configurable servers. The main configuration daemon sends each affected server a note listing the files that have changed, and each of these servers then retrieves the changes from AFS and uses them to rewrite its local configuration in a service-specific way, restarting any local daemons that need restarting.

3. Documentation

You can grab the source code from HCoop CVS on SourceForge, or browse it on the web. Be forewarned that it's implemented in Standard ML and doesn't have much documentation on building and using it.

The Domtool 2 language toolset generates documentation automatically, in a way that you may be familiar with from tools like JavaDoc and Doxygen. You can read the documentation for the standard library so far.

4. Configuration Examples

4.1. The Model T of Domain Configuration

A lot of folks at least start out with very simple configuration with their domains. In fact, a lot of the domains hosted with HCoop have the exact same configuration, modulo usernames and other blanks to fill in. A standard recipe is:

• Declare our nameservers as authoritative for your domain.
• Declare that Exim should handle e-mail for your domain.
• Direct Exim to send all mail to your domain to your mailbox.

• Add a DNS mapping from www.yourdomain.com to our web server's IP address.

• Create an Apache virtual host at that hostname serving ~/public_html/.

With the old Domtool, this configuration is scattered throughout a number of files. With Domtool 2, this source code accomplishes it:

dom "yourdomain.com" with
end;

Simple, eh? dom is a library function defined elsewhere. For comparison, an equivalent configuration that doesn't use it is:

domain "yourdomain.com" with
        dns (dnsNS "ns.hcoop.net");
        dns (dnsNS "ns2.hcoop.net");

        dns (dnsA "www" web_ip);

        handleMail;
        mailbox <- Mailbox;
        catchAllAlias mailbox;

        vhost "www" with
        end;
end;

Still quite a bit better than what the old Domtool allows!

4.2. A Very Busy Example

I'm not even going to bother setting out to explain this. Here is a load of configuration sitting in a bucket.

domain "hcoop.net" with
        dns (dnsNS "ns.hcoop.net");

        dns (dnsA "a" "1.2.3.4");
        dns (dnsCNAME "b" "a.hcoop.net");
        dns (dnsMX 1 "mail.nowhere.eu");

        handleMail;

        emailAlias "someone" "someoneElse";
        aliasMulti "me" ["nowhere","smelly@yikes.com"];

        catchAllAlias "me@gmail.com";

        mailmanWebHost "lists.hcoop.net";

        dns (dnsA "www" web_ip);
        vhost "www" where
                DocumentRoot = "/home/adamc/html";
                ServerAdmin = "my@other.address"
        with
                serverAlias "hcoop.net";
                addDefaultCharset "mumbo-jumbo/incomprehensible";

                location "/theMorgue" with
                        rewriteRule "A" "B" [];
                end;
        end;
end;

domain "schizomaniac.net" where
        TTL = 1234
with
        vhost "www" with
                directory "/home/adamc/thisPlace" with
                        unset_options [includesNOEXEC];
                        indexOptions [iconsAreLinks, scanHtmlTitles, iconWidth 45];
                end
        end;

        vhost "proxy" with
                proxyPass "/proxyLand" "http://localhost:1234/otherProxyLand";
                proxyPassReverse "/proxyLand" "http://localhost:1234/otherProxyLand";
        end;
end;

5. Summary of Security-Related Architecture

The system for distributed configuration, as implemented now, works something like this:

We create our own OpenSSL Certificate Authority for signing certificates related to the operation of Domtool 2. Every check mentioned below will only accept certificates signed by this authority.

When a user wants to push out some new configuration, he opens an SSL connection to the dispatcher daemon. The user looks in the shared AFS filesystem for his personal, automatically-generated certificate and private key. The certificate is world readable, while the permissions on the key are such that only that user can read it. (This means that we are depending on the security of AFS/Kerberos/whatever else we use.) The server verifies that the certificate is made out to the user's UNIX username and then performs the usual SSL handshake to make sure the party on the other end of the connection really has read access to the private key.

If everything has gone properly, at this point the dispatcher is convinced that it's talking to a particular UNIX user. The dispatcher consults a special Domtool access control list, telling which filesystem paths, domains, etc., that user is authorized to use. These settings are used to determine the meanings of some custom base types in the Domtool language. The dispatcher type-checks the user's configuration source in that context, aborting if type-checking fails.

The dispatcher now executes the configuration to update the directory tree (on AFS) representing network-wide configuration. The dispatcher opens an SSL connection to each server affected by the changes in turn. The other server verifies that the connecting party's certificate is signed by our CA and that it is made out to the dispatcher's hostname. If it is, the other server can accept the list of configuration files that have changed. We again trust AFS security, as the other server reads the updated files from AFS and makes the appropriate local changes.

6. Open Design Decisions

• Where should configuration source files be stored? Domtool 2 does away with the association between filenames/paths and configuration semantics. However, enforcing a standard filesystem layout for configuration is nice, because it makes it easy to recompile everyone's configuration in the event that something changes in the tools. How should we handle this issue with Domtool 2 while letting members keep the flexibility of this programming language?

• How does that SSL stuff sound? A lot of it was designed to make this easiest to implement, not because it leads to the conceptually cleanest mechanism. By the way, the OpenSSL library is a bear to learn!

7. Epilogue

After we've migrated to the new architecture and everything seems stable for a few months, I want to try creating a "web control panel"-style interface to Domtool 2. I believe I've designed the language such that this should be trivial. Specifically, I think the Domtool language's type system gives all of the information needed to generate a nice control panel interface automatically. We'd want to add some extra metadata, like cute icons for particular configuration functions, but I think it should be straightforward to implement a generic control panel that (aside from icons and extra descriptive text) builds its interface automatically from the source code of our standard library.

The gist of the interface would be a tree-editor GUI for the abstract syntax trees of the Domtool language. I think we can present it in a way such that even people who aren't comfortable with programming view it as a natural way of doing things, and the abstraction enabled by the language behind the scenes should let us blow away the competition with how quickly we can add new configuration functionality.