welcome: please sign in

Diff for "NavajosBogMigrationGuide"

Differences between revisions 1 and 22 (spanning 21 versions)
Revision 1 as of 2012-12-09 09:22:31
Size: 1865
Editor: ClintonEbadi
Comment: stub page for guide to moving your services to our new machines
Revision 22 as of 2013-01-04 23:38:55
Size: 12214
Editor: ClintonEbadi
Comment: official mail help
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
A guide to migrating your services to bog. A guide to migrating your services to navajos and bog.

Please '''read this entire document'''. Your sites and mail delivery may very well break sometime after the declared transition period if you do not.
Line 7: Line 9:
Debian squeeze, firewall, AMD64. Our current machines are hopelessly obsolete, in particular mire. Members may have noticed their sites running slowly for the last few months, things like php are so out of date modern software no longer works, and there are so many warts in the setup upgrading to a newer Debian is impossible.
Line 9: Line 11:
Because of openafs, this isn't really a "migration", but merely setting things to run on different nodes. And so, using fritz's virtualization capabilities, we've created new KernelVirtualMachine``s running Debian squeeze on 64-bit x86 and with access to roughly four times the processing power as mire.

We've also closed one of our biggest security flaws: the new servers restore the FirewallRules system we had before migrating to Peer1. This means all incoming and outgoing traffic is blocked by default, with access only granted as requested (don't worry, firewall exceptions are granted liberally; the main goal is to prevent malicious outsiders from gaining access via an exploited web service).

Because of openafs and kerberos, this isn't really a "migration": all accounts and data are available automatically on the new nodes. This time around all you have to do is flip a few domtool flags to serve your sites with the new server, and ssh into a different host for shell access. Except for the 32-bit to 64-bit architecture change and newer software, it's the same environment you're used to.

== Change your Password ==

If you have not changed your password within the last 30 days or so, take the opportunity to change it now. The version of Kerberos we were running on older machines did not support modern high-strength cryptography methods, so if you haven't changed your password recently you will be unable to forward tickets or access certain services like Postgres from the new server. Changing your password will regenerate your principle with the more secure key types, restoring functionality.

You will need to log in to `bog.hcoop.net` and use `kpasswd` to change your password. The normal `passwd` command will not work.

== Using the New Shell Server ==

Just `ssh $user@bog.hcoop.net`

As with `navajos`, the environment is pretty spartan. Feel free to liberally request new packages since this is the chaotic shell server. Note that you have little network permission by default; if you're using our shell services for e.g. IRC you'll need to request FirewallRules.

`bog` will become `ssh.hcoop.net` in mid-January. Mire will still be accessible through `mire.hcoop.net`.

== Changes to Mail ==

Due to the continued filtering of official messages by Google as spam, we're changing our mail setup. It is now officially recommended that you send mail for domains hosted with us through our mail hub, and '''required''' that you send mail from your `@hcoop.net` address through our mail hub (due to an improved SPF record). The realities of sending mail due to policies by much larger ISPs than us has changed significantly in the last few years.

More importantly: '''Catch-all addresses will be disabled by default on February 15, 2013'''. If you are actually using the catch-all and also `dom`, you can simply set `DefaultAlias = true` to continue catching all mail for your domain to your primary mail account (the DomTool manual explains how to do it when you use `domain` instead). To provide equivalent functionality, our mail server now supports sub-addressing in the form `mailbox+$subaddress@domain`, which delivers mail to `mailbox@domain`.

If you are forwarding to Gmail, please do not do use a catch-all if you can avoid it all because forwarding all of the spam you inevitably receive is causing Google to flag us as a spam source, and generating apparently unavoidable [[http://en.wikipedia.org/wiki/Backscatter_%28email%29|backscatter]] that makes us bad netizens.

If you do not actually need a catch-all, you should explicitly set `DefaultAlias = false` (or remove your `defaultAlias` action) as soon as possible to reduce the burden on admins following up with members before making the switch (ideally, all current members will explicitly set the value so that the admins are certain no one will begin losing mail).

If you are already sending '''all''' of your mail through `mail.hcoop.net`, you may want to use `addDefaultSPF` to indicate that mail for a given domain will only be sent from our mail exchangers.

{{{
dom "yourdomain" where
  ...
with
  addDefaultSPF;
  ...
end;
}}}

See [[MemberManual/Email]] for more details, in particular [[MemberManual/Email#Forwarding]].


=== Ensuring You Receive Official Mail ===

If you are using Gmail, scan your spam folder for messages from hcoop.net, and delete the spam flag from any you find. This might help Google's spam filter recognize our messages as misclassified ham. Make sure to review the email section of the MemberManual for additional tips including a filter that will whitelist official mail.

== DNS ==

If your records at your registrar are using nameservers other than `ns1.hcoop.net` and `ns2.hcoop.net` (e.g. `ns3.hcoop.net`), They have been broken for over a year, and no longer resolve. We have not documented any other nameservers for several years so most everyone should not be using them, but if you have been a long term member from the fyodor era you may still have a straggling reference.
Line 13: Line 65:
With the restrictive firewall in place, you will have to request rules if you need to access Internet resources from your cgi programs, or want to use irc or similar from bog. With the restrictive firewall in place, you will have to request rules if you need to access Internet resources from your cgi programs, or want to use irc or similar from bog. The interface for requesting firewall rules is clunky; if you need to request multiple rules or have other questions, file a bug against the Firewall component.
Line 15: Line 67:
FirewallRules See FirewallRules for full documentation.

You will most likely request rules on node `bog`. Pretty much the only case to request rules on `navajos` is for a cgi program that needs to access Internet resources.

The current policy regarding firewall rules is more or less what was done ages ago on Fyodor. Therefore it may not be ideal. If you don't like it, complain loudly! We're democratically organized after all.
Line 19: Line 75:
Static web sites will work on navajos without any special effort, but if you're using cgi you will probably have to request packages. Before proceeding, re-famliarize yourself with [[DomTool/UserGuide]]. Almost all problems you might think require admin help can be solved using various domtool utilities.
Line 21: Line 77:
(Mention domtool-tail for checking error logs, since it's hidden well in the manual) Static web sites will work on navajos without any special effort. To minimize any perceived down time, the default TTL for all domains has been dropped to an hour.

Don't be surprised if any CGI programs do not run as expected initially; the new systems have not had many packages installed, and so you will probably have to [[https://members.hcoop.net/portal/apt?node=3|request packages]] on node `navajos`.

Examining your log files using `domtool-tail` should reveal the missing software (see the [[MemberManual/ServingWebsites#Examiningyourlogs|member manual for more information]]). If you still can't figure it out, please file a bug under "Misc UNIX" requesting assistance.

'''If your website appears to serve hcoop.net do not panic'''. This just means that your dns cache is still pointing toward mire and your site has moved to navajos, causing mire to serve the default virtual host. You can use `domtool-admin describe $domain` to verify. You'll want to look for:

{{{
<VirtualHost 69.90.123.70:80>
}}}
Line 25: Line 91:
To test migration, ... If you are using the `dom` (Easy Domain) type, trying out the new web server is easy.
Line 27: Line 93:
To migrate everything, set `DefaultWebNode = "navajos"` in your `dom` config. To test that your site will work, you can add an empty `webAt` action, which will use the same configuration as your default domain. If you are setting the `WWW` environment variable you must also add a couple of lines to include that configuration. Likewise, you can simply copy and paste any `web` directives as `webAt "navajos" "TESTSUBDOMAIN"` to test if they work.

{{{
dom "yourdomain" where
   ...
with
   ...
   webAt "navajos" "TESTSUBDOMAIN"
      with
        (* The following lines are only needed if you set the WWW environment variable to customize the default vhost *)
        www : [Vhost] <- WWW;
        www
      end;
end
}}}


After you've ensured that things are working (or if you like to jump off of cliffs for fun), set `DefaultWebNode = "navajos"` in your `dom` config to migrate everything. Note that it may take up to an hour for dns changes to propagate (if you have not customized `TTL`). You may also change calls to `web` into `webAt "mire"` if you need to run part of your website on mire temporarily (if you are using e.g. php4 scripts, 32-bit proxied server binaries, postgresql 8.1).

{{{
dom "yourdomain" where
   ...
   DefaultWebNode = "navajos";
with
   ...
   (* To keep a particular subdomain on mire *)
   webAt "mire" "SUBDOMAIN" where ... with ... end;
end;
}}}
Line 31: Line 125:
You're on your own ;-) You're on your own, possibly ;-)

If you use `vhost` or `vhostDefault` to configure your websites, you will need to set the Web``Places environment variable to host them on navajos:

{{{
domain "yourdomain"
with
  vhostDefault where
    WebPlaces = [web_place_default "navajos"];
  with
    ...
  end;
end;
}}}

Any `dnsIP` or `dnsDefault` records pointing toward `mire_ip` or "68.90.123.68" need to be changed to point to `navajos`:

{{{
domain "yourdomain"
with
  ...
  dnsDefault navajos_ip;
end;
}}}

==== Nameservers ====

If you are using our nameservers, check that your `nameserver` declarations are not explicitly mentioning any IP addresses, but rather referencing "ns1.hcoop.net" and "ns2.hcoop.net". No other name servers are valid. If you are referencing IP addresses directly, be warned that at least `ns2.hcoop.net` will be moving to a new IP very soon.
Line 35: Line 156:
Proxied servers must be run on bog. Request ProxiedServer firewall rule. Cron permissions for starting at reboot. Proxied servers must be run on bog. Request a Proxied``Server firewall rule on `bog` and use port above 50000.
Line 37: Line 158:
If you are running your own instance of Apache on mire, please file a bug report explaining why you are doing so. If it's just to run a newer version than available on mire, you can probably just switch to the system-wide version. If you need modules or directives not supported by DomTool, any that can be supported securely will be added during migration. You will also need to re-request cron permissions on node `bog` for starting any services at boot.

If you are running your own instance of Apache on mire, please file a bug report under "HTTP/Apache" explaining why you are doing so. If it's just to run a newer version than available on mire, you can just switch to the system-wide version. If you need modules or directives not supported by DomTool, any that can be supported securely will be added during migration.
Line 41: Line 164:
If you are using php4, you must upgrade to php5. If you are using php4, you must upgrade to php5.

=== Moin Moin ===

Don't panic if your `moinMoin` or `addMoinMoin` directives are now suffixed with `Old`; ClintonEbadi updated your config for you. The `Old` variants of both install your wiki onto mire, using its local machine copy. To support saner upgrades in the future, we haved moved our moin install into afs, upgrading from 1.7 to 1.9.5 in the process. Follow the instructions at [[MemberManual/WebApplications/MoinMoin#Moin1.7.x]] and remove the `Old` to upgrade.

=== SSL ===

If you have requested access to the default HCoop SSL certificate your SSL vhosts will move with no special effort.

If you have an IP Address allocated for SSL, you will need to coordinate with the admins to have the address moved from `mire` to `navajos`. [[https://bugzilla.hcoop.net/enter_bug.cgi|File a bug]] under the "IP Addresses" category and we'll help you.

We do not yet support SNI, but since OpenSSL on `navajos` is new enough, we should be able to once mire is decommissioned to avoid complication.
Line 47: Line 182:
Postgres users will need to dump their databases and upgrade to 9.1. New dbtool "database" `postgres-9.1`. If any trouble is encountered, file a bug under SQL DBs. We are now offering PostgreSQL 9.1 instead of 8.1. Administration is performed using a new dbtool "database" named `postgres-9.1`. See the [[MemberManual/Databases#PostgreSQL|postgresql member manual section]] for full details.
Line 49: Line 184:
== Using the New Shell Server == You can still connect to your 8.1 databases from navajos, but 8.1 is officially deprecated so you should dump and re-create your databases.
Line 51: Line 186:
Just `ssh $user@bog.hcoop.net` If any trouble is encountered, file a bug under SQL DBs.
Line 55: Line 190:
Document temporary squirrelmail/roundcube addresses ''Document temporary squirrelmail/roundcube addresses''

A guide to migrating your services to navajos and bog.

Please read this entire document. Your sites and mail delivery may very well break sometime after the declared transition period if you do not.

1. Explanation of New Machines

Our current machines are hopelessly obsolete, in particular mire. Members may have noticed their sites running slowly for the last few months, things like php are so out of date modern software no longer works, and there are so many warts in the setup upgrading to a newer Debian is impossible.

And so, using fritz's virtualization capabilities, we've created new KernelVirtualMachines running Debian squeeze on 64-bit x86 and with access to roughly four times the processing power as mire.

We've also closed one of our biggest security flaws: the new servers restore the FirewallRules system we had before migrating to Peer1. This means all incoming and outgoing traffic is blocked by default, with access only granted as requested (don't worry, firewall exceptions are granted liberally; the main goal is to prevent malicious outsiders from gaining access via an exploited web service).

Because of openafs and kerberos, this isn't really a "migration": all accounts and data are available automatically on the new nodes. This time around all you have to do is flip a few domtool flags to serve your sites with the new server, and ssh into a different host for shell access. Except for the 32-bit to 64-bit architecture change and newer software, it's the same environment you're used to.

2. Change your Password

If you have not changed your password within the last 30 days or so, take the opportunity to change it now. The version of Kerberos we were running on older machines did not support modern high-strength cryptography methods, so if you haven't changed your password recently you will be unable to forward tickets or access certain services like Postgres from the new server. Changing your password will regenerate your principle with the more secure key types, restoring functionality.

You will need to log in to bog.hcoop.net and use kpasswd to change your password. The normal passwd command will not work.

3. Using the New Shell Server

Just ssh $user@bog.hcoop.net

As with navajos, the environment is pretty spartan. Feel free to liberally request new packages since this is the chaotic shell server. Note that you have little network permission by default; if you're using our shell services for e.g. IRC you'll need to request FirewallRules.

bog will become ssh.hcoop.net in mid-January. Mire will still be accessible through mire.hcoop.net.

4. Changes to Mail

Due to the continued filtering of official messages by Google as spam, we're changing our mail setup. It is now officially recommended that you send mail for domains hosted with us through our mail hub, and required that you send mail from your @hcoop.net address through our mail hub (due to an improved SPF record). The realities of sending mail due to policies by much larger ISPs than us has changed significantly in the last few years.

More importantly: Catch-all addresses will be disabled by default on February 15, 2013. If you are actually using the catch-all and also dom, you can simply set DefaultAlias = true to continue catching all mail for your domain to your primary mail account (the DomTool manual explains how to do it when you use domain instead). To provide equivalent functionality, our mail server now supports sub-addressing in the form mailbox+$subaddress@domain, which delivers mail to mailbox@domain.

If you are forwarding to Gmail, please do not do use a catch-all if you can avoid it all because forwarding all of the spam you inevitably receive is causing Google to flag us as a spam source, and generating apparently unavoidable backscatter that makes us bad netizens.

If you do not actually need a catch-all, you should explicitly set DefaultAlias = false (or remove your defaultAlias action) as soon as possible to reduce the burden on admins following up with members before making the switch (ideally, all current members will explicitly set the value so that the admins are certain no one will begin losing mail).

If you are already sending all of your mail through mail.hcoop.net, you may want to use addDefaultSPF to indicate that mail for a given domain will only be sent from our mail exchangers.

dom "yourdomain" where
  ...
with
  addDefaultSPF;
  ...
end;

See MemberManual/Email for more details, in particular MemberManual/Email#Forwarding.

4.1. Ensuring You Receive Official Mail

If you are using Gmail, scan your spam folder for messages from hcoop.net, and delete the spam flag from any you find. This might help Google's spam filter recognize our messages as misclassified ham. Make sure to review the email section of the MemberManual for additional tips including a filter that will whitelist official mail.

5. DNS

If your records at your registrar are using nameservers other than ns1.hcoop.net and ns2.hcoop.net (e.g. ns3.hcoop.net), They have been broken for over a year, and no longer resolve. We have not documented any other nameservers for several years so most everyone should not be using them, but if you have been a long term member from the fyodor era you may still have a straggling reference.

6. Firewall

With the restrictive firewall in place, you will have to request rules if you need to access Internet resources from your cgi programs, or want to use irc or similar from bog. The interface for requesting firewall rules is clunky; if you need to request multiple rules or have other questions, file a bug against the Firewall component.

See FirewallRules for full documentation.

You will most likely request rules on node bog. Pretty much the only case to request rules on navajos is for a cgi program that needs to access Internet resources.

The current policy regarding firewall rules is more or less what was done ages ago on Fyodor. Therefore it may not be ideal. If you don't like it, complain loudly! We're democratically organized after all.

7. Moving Web Sites

Before proceeding, re-famliarize yourself with DomTool/UserGuide. Almost all problems you might think require admin help can be solved using various domtool utilities.

Static web sites will work on navajos without any special effort. To minimize any perceived down time, the default TTL for all domains has been dropped to an hour.

Don't be surprised if any CGI programs do not run as expected initially; the new systems have not had many packages installed, and so you will probably have to request packages on node navajos.

Examining your log files using domtool-tail should reveal the missing software (see the member manual for more information). If you still can't figure it out, please file a bug under "Misc UNIX" requesting assistance.

If your website appears to serve hcoop.net do not panic. This just means that your dns cache is still pointing toward mire and your site has moved to navajos, causing mire to serve the default virtual host. You can use domtool-admin describe $domain to verify. You'll want to look for:

<VirtualHost 69.90.123.70:80>

7.1. Easy Domain Users

If you are using the dom (Easy Domain) type, trying out the new web server is easy.

To test that your site will work, you can add an empty webAt action, which will use the same configuration as your default domain. If you are setting the WWW environment variable you must also add a couple of lines to include that configuration. Likewise, you can simply copy and paste any web directives as webAt "navajos" "TESTSUBDOMAIN" to test if they work.

dom "yourdomain" where
   ...
with
   ...
   webAt "navajos" "TESTSUBDOMAIN"
      with
        (* The following lines are only needed if you set the WWW environment variable to customize the default vhost *)
        www : [Vhost] <- WWW;
        www
      end;
end

After you've ensured that things are working (or if you like to jump off of cliffs for fun), set DefaultWebNode = "navajos" in your dom config to migrate everything. Note that it may take up to an hour for dns changes to propagate (if you have not customized TTL). You may also change calls to web into webAt "mire" if you need to run part of your website on mire temporarily (if you are using e.g. php4 scripts, 32-bit proxied server binaries, postgresql 8.1).

dom "yourdomain" where
   ...
   DefaultWebNode = "navajos";
with
   ...
   (* To keep a particular subdomain on mire *)
   webAt "mire" "SUBDOMAIN" where ... with ... end;
end;

7.2. Low-level domain users

You're on your own, possibly ;-)

If you use vhost or vhostDefault to configure your websites, you will need to set the WebPlaces environment variable to host them on navajos:

domain "yourdomain"
with
  vhostDefault where
    WebPlaces = [web_place_default "navajos"];
  with
    ...
  end;
end;

Any dnsIP or dnsDefault records pointing toward mire_ip or "68.90.123.68" need to be changed to point to navajos:

domain "yourdomain"
with
  ...
  dnsDefault navajos_ip;
end;

7.2.1. Nameservers

If you are using our nameservers, check that your nameserver declarations are not explicitly mentioning any IP addresses, but rather referencing "ns1.hcoop.net" and "ns2.hcoop.net". No other name servers are valid. If you are referencing IP addresses directly, be warned that at least ns2.hcoop.net will be moving to a new IP very soon.

7.3. Proxied Servers

Proxied servers must be run on bog. Request a ProxiedServer firewall rule on bog and use port above 50000.

You will also need to re-request cron permissions on node bog for starting any services at boot.

If you are running your own instance of Apache on mire, please file a bug report under "HTTP/Apache" explaining why you are doing so. If it's just to run a newer version than available on mire, you can just switch to the system-wide version. If you need modules or directives not supported by DomTool, any that can be supported securely will be added during migration.

7.4. PHP

If you are using php4, you must upgrade to php5.

7.5. Moin Moin

Don't panic if your moinMoin or addMoinMoin directives are now suffixed with Old; ClintonEbadi updated your config for you. The Old variants of both install your wiki onto mire, using its local machine copy. To support saner upgrades in the future, we haved moved our moin install into afs, upgrading from 1.7 to 1.9.5 in the process. Follow the instructions at MemberManual/WebApplications/MoinMoin#Moin1.7.x and remove the Old to upgrade.

7.6. SSL

If you have requested access to the default HCoop SSL certificate your SSL vhosts will move with no special effort.

If you have an IP Address allocated for SSL, you will need to coordinate with the admins to have the address moved from mire to navajos. File a bug under the "IP Addresses" category and we'll help you.

We do not yet support SNI, but since OpenSSL on navajos is new enough, we should be able to once mire is decommissioned to avoid complication.

8. Databases

MySQL users should not need to do anything.

We are now offering PostgreSQL 9.1 instead of 8.1. Administration is performed using a new dbtool "database" named postgres-9.1. See the postgresql member manual section for full details.

You can still connect to your 8.1 databases from navajos, but 8.1 is officially deprecated so you should dump and re-create your databases.

If any trouble is encountered, file a bug under SQL DBs.

9. HCoop Services

Document temporary squirrelmail/roundcube addresses

NavajosBogMigrationGuide (last edited 2013-02-16 21:19:41 by ClintonEbadi)