welcome: please sign in

The following 740 words could not be found in the dictionary of 7 words (including 7 LocalSpellingWords) and are highlighted below:
aborting   abstract   abstraction   abstractions   accept   accepted   access   accomplishes   ad   Adam   adamc   add   Add   added   Adding   addition   address   Admin   affect   affected   After   again   Alias   alias   all   All   allow   allowed   allows   already   also   alternative   always   an   and   any   anyone   Apache   appropriate   Architecture   architecture   Are   are   aren   around   as   As   aside   association   assuming   at   At   attempting   authoritative   Authority   authority   authorized   automatically   away   base   basic   Be   be   bear   because   behave   behind   believe   below   better   between   big   bit   blanks   blocks   blow   boilerplate   both   bother   browse   bucket   bugs   build   building   builds   built   Busy   but   By   by   calling   calls   can   case   catch   Certificate   certificate   certificates   change   changed   changes   changing   Charset   check   checked   checker   checking   checks   Chlipala   cleanest   code   coding   collapsed   collective   Colocation   com   comfortable   coming   common   comparison   competition   complaints   complicated   conceptually   configurable   configuration   Configuration   configure   configuring   connecting   connection   constraints   consults   Contents   context   contexts   control   convinced   Coop   could   Create   create   creating   crying   current   custom   cute   cvs   daemon   daemons   date   Decisions   Declare   declaring   Default   default   defaults   defined   depending   describing   descriptive   Design   designed   determine   developed   developing   Direct   directive   directives   directory   dispatcher   Distributed   distributed   dns   do   Doc   Document   Documentation   documentation   does   doesn   doing   Dom   dom   domain   Domain   domains   Domtool   domtool2   don   downs   Doxygen   each   easiest   easy   editor   effect   eh   elements   else   Else   elsewhere   email   enabled   end   enforcing   English   enjoying   environment   Epilogue   equivalent   etc   eu   even   event   every   Every   everyone   everything   exact   Example   Examples   except   executes   Exim   explain   Extending   externally   extra   fact   fails   familiar   far   feature   few   fhp   file   filenames   files   filesystem   fill   final   first   flexibility   folks   For   for   forbid   forewarned   Forge   frequently   from   function   Functional   functional   functionality   functions   Further   generate   generated   generates   generation   generic   gist   given   gives   gmail   go   God   going   gone   grab   group   growing   had   hadn   handle   handshake   has   Haskell   have   hcoop   he   hearty   helping   here   Here   Herein   higher   his   hoc   home   honest   host   Host   hosted   hostname   how   How   However   hprog   Html   html   http   icon   icons   id   if   If   implement   implemented   important   in   In   included   includes   incomprehensible   increases   increasing   index   influenced   information   input   inside   instance   instead   interested   interface   interpreter   involve   ip   is   isn   issue   it   It   its   itself   Java   jumbo   Just   keep   Kerberos   key   labeled   Land   language   Language   laughing   layout   leads   learn   least   let   letting   level   lexical   library   lies   life   like   lines   Links   list   listing   lists   living   ll   load   local   localhost   locally   location   logic   looks   lot   machine   machines   made   mail   Mail   Mailbox   mailbox   mailman   main   majority   make   makes   making   malice   management   Many   many   mapping   may   me   meanings   means   meant   mechanism   member   members   membership   mentioned   merrily   metadata   migrated   mistakes   Ml   Model   modeled   modify   modulo   monad   months   more   Morgue   much   Multi   multiple   mumbo   my   name   nameservers   natural   Nearly   necessary   need   needed   needs   net   network   new   New   nice   no   Not   not   note   notion   now   nowhere   ns   ns2   number   object   Of   of   often   old   on   one   only   Open   opens   operation   options   Options   or   order   org   organizing   other   otherwise   ought   our   out   outputted   over   Overview   overview   own   page   painful   panel   parameters   parse   particular   party   Pass   patched   paths   patterns   Pedia   people   performs   permissions   personal   pieces   Place   Plans   point   populated   possible   predicates   prefix   present   preview   primitive   principled   principles   private   probably   problem   process   programmed   programming   programs   properly   providing   proxy   Proxy   public   pure   push   quickly   quite   re   reach   read   readable   reads   really   reappear   recipe   recompile   reified   Related   related   replaced   representing   restarting   retrieves   Reverse   rewrite   Rich   right   risk   rolled   Root   rough   Rule   running   same   satsify   scan   scattered   scenes   schizomaniac   scoping   script   scripts   second   secure   Security   security   See   see   seem   seems   semantics   send   sends   sense   sent   series   serious   seriousness   Server   server   servers   service   serving   set   setting   settings   shared   should   signed   signing   simple   Simple   since   Since   sitting   smelly   so   some   someone   something   sometimes   sorts   sound   Source   source   sourceforge   special   specific   Specifically   specify   stable   Standard   standard   start   stateful   statically   still   Still   stored   straightforward   stuff   style   subdirectory   subroutines   subset   succeeds   such   summaries   Summary   supplemented   support   sure   switching   syntax   system   systems   Table   talking   telling   tends   text   than   that   The   the   their   them   then   There   there   these   These   they   They   things   think   this   This   three   through   throughout   tier   tiers   tiny   Titles   to   today   tool   Tool   tools   toolset   top   tree   trees   trivial   trust   try   turn   two   type   typed   types   under   unit   unset   update   updated   ups   us   use   used   user   username   usernames   users   uses   using   usual   valid   Values   variables   vast   ve   verifies   very   Very   vhost   via   view   virtual   want   wants   was   way   We   we   weaknesses   Web   web   weight   what   whatever   When   Where   where   which   while   who   Why   wide   Width   Wiki   will   With   with   won   working   works   world   worth   would   write   writing   www   yet   yikes   You   you   your   yourdomain  

Clear message
Edit

DomtoolTwo

This page is out of date and was only meant as a preview. See DomTool instead.

Herein lies an overview of the new configuration management tool that AdamChlipala is developing. Many things that could be included aren't here yet, and they'll be added as they seem to be important.

New members coming to this page: This system isn't yet rolled out. You probably want DomainTool instead.

1. Why something new?

The vast majority of configuration changes made at HCoop today go through the (already second-generation) DomainTool developed by AdamChlipala. We would have collapsed under the weight of the collective needs of our growing member base if we hadn't had something like Domtool to allow members to configure their own domains in a secure way. However, today's Domtool has some serious weaknesses. Here are a few in rough increasing order of seriousness.

2. Overview of Domtool 2

There are two main pieces to the Domtool 2 system: a programming language and a system for distributed configuration implemented around it.

2.1. The Language

The Domtool language is a statically-typed, pure functional programming language influenced by Haskell and ML. It uses the function as the basic unit of abstraction, both for describing primitive configuration directives and higher-level subroutines built from them.

Configuration itself as modeled as a stateful process of making a series of changes to the world, and this process is modeled inside the Domtool language with a configuration monad. The monad also includes the notion of typed environment variables that are a principled alternative to default function parameters, where it's possible to change defaults locally via lexical scoping.

Not all configuration directives are valid in all contexts. For instance, it only makes sense to specify that a file is a CGI script if you're working in the context of an Apache virtual host. Further, some Apache directives are only valid in the context of a particular directory. Values in the configuration monad are labeled with predicates in a tiny logic describing valid contexts for their use, and the type checker won't let you use a directive elsewhere.

The language interpreter also allows the addition of new base types by providing membership predicates implemented in SML. For instance, the standard library has a type for the name of a domain that the current user is allowed to configure. Rich types like this make it possible to use types of library functions as machine-checked documentation that often doesn't need to be supplemented with English.

2.2. Distributed Configuration

As the language allows primitive functions to be implemented externally in SML code, it's possible to build all sorts of systems on top of it. The one that we're interested in here is a system for changing configuration of network daemons running on multiple machines.

The basic system has three tiers.

The first tier is the members, merrily configuring away, laughing, crying, enjoying life's ups and downs, and sometimes attempting to configure things that they ought not to, through honest mistakes or (God forbid) malice. They write their configuration files as programs in the Domtool language and type check them locally to make sure that they will be accepted. If type checking succeeds, the scripts are sent over our LAN to a daemon process living on our main server.

This daemon type-checks the files again (since we don't want to have to trust users to do it right!) and, assuming they check out, executes them to modify a directory tree on our shared AFS filesystem. This tree looks a lot like /etc/domains does now, except that it's populated with summaries outputted by the daemon on how other daemons should behave.

At this point, we reach the final tier. A given set of changes may affect any subset of the configurable servers. The main configuration daemon sends each affected server a note listing the files that have changed, and each of these servers then retrieves the changes from AFS and uses them to rewrite its local configuration in a service-specific way, restarting any local daemons that need restarting.

3. Documentation

You can grab the source code from HCoop CVS on SourceForge, or browse it on the web. Be forewarned that it's implemented in Standard ML and doesn't have much documentation on building and using it.

The Domtool 2 language toolset generates documentation automatically, in a way that you may be familiar with from tools like JavaDoc and Doxygen. You can read the documentation for the standard library so far.

4. Configuration Examples

4.1. The Model T of Domain Configuration

A lot of folks at least start out with very simple configuration with their domains. In fact, a lot of the domains hosted with HCoop have the exact same configuration, modulo usernames and other blanks to fill in. A standard recipe is:

With the old Domtool, this configuration is scattered throughout a number of files. With Domtool 2, this source code accomplishes it:

dom "yourdomain.com" with
end;

Simple, eh? dom is a library function defined elsewhere. For comparison, an equivalent configuration that doesn't use it is:

domain "yourdomain.com" with
        dns (dnsNS "ns.hcoop.net");
        dns (dnsNS "ns2.hcoop.net");

        dns (dnsA "www" web_ip);

        handleMail;
        mailbox <- Mailbox;
        catchAllAlias mailbox;

        vhost "www" with
        end;
end;

Still quite a bit better than what the old Domtool allows!

4.2. A Very Busy Example

I'm not even going to bother setting out to explain this. Here is a load of configuration sitting in a bucket.

domain "hcoop.net" with
        dns (dnsNS "ns.hcoop.net");

        dns (dnsA "a" "1.2.3.4");
        dns (dnsCNAME "b" "a.hcoop.net");
        dns (dnsMX 1 "mail.nowhere.eu");

        handleMail;

        emailAlias "someone" "someoneElse";
        aliasMulti "me" ["nowhere","smelly@yikes.com"];

        catchAllAlias "me@gmail.com";

        mailmanWebHost "lists.hcoop.net";

        dns (dnsA "www" web_ip);
        vhost "www" where
                DocumentRoot = "/home/adamc/html";
                ServerAdmin = "my@other.address"
        with
                serverAlias "hcoop.net";
                addDefaultCharset "mumbo-jumbo/incomprehensible";

                location "/theMorgue" with
                        rewriteRule "A" "B" [];
                end;
        end;
end;

domain "schizomaniac.net" where
        TTL = 1234
with
        vhost "www" with
                directory "/home/adamc/thisPlace" with
                        unset_options [includesNOEXEC];
                        indexOptions [iconsAreLinks, scanHtmlTitles, iconWidth 45];
                end
        end;

        vhost "proxy" with
                proxyPass "/proxyLand" "http://localhost:1234/otherProxyLand";
                proxyPassReverse "/proxyLand" "http://localhost:1234/otherProxyLand";
        end;
end;

5. Summary of Security-Related Architecture

The system for distributed configuration, as implemented now, works something like this:

We create our own OpenSSL Certificate Authority for signing certificates related to the operation of Domtool 2. Every check mentioned below will only accept certificates signed by this authority.

When a user wants to push out some new configuration, he opens an SSL connection to the dispatcher daemon. The user looks in the shared AFS filesystem for his personal, automatically-generated certificate and private key. The certificate is world readable, while the permissions on the key are such that only that user can read it. (This means that we are depending on the security of AFS/Kerberos/whatever else we use.) The server verifies that the certificate is made out to the user's UNIX username and then performs the usual SSL handshake to make sure the party on the other end of the connection really has read access to the private key.

If everything has gone properly, at this point the dispatcher is convinced that it's talking to a particular UNIX user. The dispatcher consults a special Domtool access control list, telling which filesystem paths, domains, etc., that user is authorized to use. These settings are used to determine the meanings of some custom base types in the Domtool language. The dispatcher type-checks the user's configuration source in that context, aborting if type-checking fails.

The dispatcher now executes the configuration to update the directory tree (on AFS) representing network-wide configuration. The dispatcher opens an SSL connection to each server affected by the changes in turn. The other server verifies that the connecting party's certificate is signed by our CA and that it is made out to the dispatcher's hostname. If it is, the other server can accept the list of configuration files that have changed. We again trust AFS security, as the other server reads the updated files from AFS and makes the appropriate local changes.

6. Open Design Decisions

7. Epilogue

After we've migrated to the new architecture and everything seems stable for a few months, I want to try creating a "web control panel"-style interface to Domtool 2. I believe I've designed the language such that this should be trivial. Specifically, I think the Domtool language's type system gives all of the information needed to generate a nice control panel interface automatically. We'd want to add some extra metadata, like cute icons for particular configuration functions, but I think it should be straightforward to implement a generic control panel that (aside from icons and extra descriptive text) builds its interface automatically from the source code of our standard library.

The gist of the interface would be a tree-editor GUI for the abstract syntax trees of the Domtool language. I think we can present it in a way such that even people who aren't comfortable with programming view it as a natural way of doing things, and the abstraction enabled by the language behind the scenes should let us blow away the competition with how quickly we can add new configuration functionality.

DomtoolTwo (last edited 2009-04-09 21:08:04 by AdamChlipala)