welcome: please sign in

Diff for "DigitalOceanMigrationGuide"

Differences between revisions 1 and 9 (spanning 8 versions)
⇤ ← Revision 1 as of 2018-10-13 16:54:50 →
Size: 1574
Editor: ClintonEbadi
Comment: stub for migration guide
← Revision 9 as of 2018-10-25 02:33:34 → ⇥
Size: 7325
Editor: ClintonEbadi
Comment: mention cron changes
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
A guide to moving your services to our new virtual infrastructure at digital ocean. A guide to moving your services to our new virtual infrastructure at DigitalOcean.
Line 4: Line 4:

== Changes Requiring Action ==

If you...

 * have a crontab
 * are running any daemons on `bog`
 * use postgresql
 * are using the low level `domain` domtool type instead of `dom`

...you will need to take some manual actions during the migration or '''your services may break'''.

Migration should be otherwise transparent.
Line 9: Line 22:
 * All domains converted from suphp/cgi php to fastcgi based php: 2018-10-10 (done!)
 * mailman list migration
Line 10: Line 25:
 * database migration  * database migration (incurring downtime)
   * TODO: estimated max downtime window for migrating databases after doing test run
Line 15: Line 31:
The new machines that members will directly interact with are:
Line 16: Line 33:
=== Networking Change: IPv6 is Supported ===  * marsh, the new shell server. replaces [[ServerBog|bog]]. This is the server you will login to.
 * shelob, the new web server. replaces [[ServerNavajos|navajos]]
 * minsky, the new mail server. replaces [[ServerMcCarthy|mccarthy]]

For a full list of servers at the new host and their purposes, see [[Hardware#Digital_Ocean]]

All servers are now running Debian Stretch (the latest stable release). Packages that were requested through the members portal on both bog and navajos have been installed on marsh and shelob so if your software works now, it ought to work on the new servers. If you are using compiled binaries that link against system libraries you might need to recompile.

=== New Features ===

==== Networking Change: IPv6 is Supported ====
Line 20: Line 47:
By default, domtool will not generate `AAAA` (IPv6) DNS records for your domains, but this will be enabled for the `dom` type after all sites are migrated. By default, domtool will not generate `AAAA` (IPv6) DNS records for your domains, but this will be enabled for the `dom` type after all sites are migrated. {{{#!wiki comment

note: not quite, since domtool set the vhost to listen on the ipv4 address only

If you're feeling like living on the cutting edge, you can set an `AAAA` record to the web server manually using `dnsIPv6` and `dnsDefaultv6`, for example:

{{{
dom "yourdomain" where
  DefaultWebNode = "shelob";
  ...
with
  dnsDefaultv6 shelob_ipv6; (* sets AAAA for "yourdomain" *)
  dnsIPv6 "www.yourdomain" shelob_ipv6;
  ...
end
}}}

==== Web Server ====

All sites are now using mod_fcgid based php 5.6, which should provide a significant improvement in performance.

The new webserver has the latest Apache 2.4.x release, so the `sslCertificateChainFile` directive will no longer be required for https -- the certificates can contain the full chain now.
Line 24: Line 73:
The new shell server may be accessed using `ssh marsh.hcoop.net`. Thanks to [[AndrewFileSystem|openafs]], both the old and new infrastructure share the same volumes and you can access your data from either.

=== Using Cron: Permissions No Longer Needed ===

All members have cron enabled by default on the new login server and no longer need to specially request permissions.

Members with crontabs on bog will need to re-create the crontab on marsh. This should just be a matter of copying the crontab contents to marsh, and removing it from bog afterward.
Line 25: Line 82:

We are upgrading from apache 2.2 to 2.4, but have a configuration that should behave identically the one currently used on ServerNavajos. We are currently using `mod_access_compat` (Allow/Deny/Satisfy directives) instead of the newer `Require` access framework so that existing configurations do not need to be updated. At some point in the future we will update domtool and convert member configurations to the new access control directives.

If you are using the `dom` type, the move should be transparent; we will update `DefaultWebNode` to the new web server and reconfigure all domains on the date scheduled for transitioning to the new servers. If you would like to migrate your domains early, you may set the environment variable `DefaultWebNode = "shelob";` in your configuration to force the domain to be configured on the new webserver.
Line 28: Line 89:
The new webserver is running php 5.6, with a configuration matching the existing production configuration.

We have supported a simple fastcgi based php for a while now, but have not widely deployed it. Our current method of supporting php-cgi based php (suphp) has been removed from debian stretch, and `shelob` only supports fastcgi based php.

All domains will be automatically upgraded to fastcgi based php, and plain cgi php will be disabled before migration.

After we migrate all domains, we will be able to enable php 7.2 with minimal effort (due to domtool limitations, it's not feasible to support it while ServerNavajos is in production).
Line 29: Line 98:

=== Low-level domain users ===

You're on your own, possibly ;-)

If you use `vhost` or `vhostDefault` to configure your websites, you will need to set the Web``Places environment variable to host them on `shelob`:

{{{
domain "yourdomain"
with
  vhostDefault where
    WebPlaces = [web_place_default "shelob"];
  with
    ...
  end;
end;
}}}

Any `dnsIP` or `dnsDefault` records pointing toward `navajos_ip` or "69.90.123.70" need to be changed to point to `shelob`:

{{{
domain "yourdomain"
with
  ...
  dnsDefault shelob_ip;
end;
}}}
Line 34: Line 130:
todo: example of proxied server config and update.
Line 36: Line 134:
Postgresql is upgraded to 9.6. 9.1 no longer supported, data automatically migrated. == Postgresql ==

Postgres users '''must''' take action! Due to our usage of gssapi and ident for authentication, we cannot set up a simple stunnel for secure connections between the datacenters. To ensure the security of your data, connections from one datacenter to the other will require ssl be enabled in postgres. Applications based on `libpq` ought to negotiate ssl automatically, but php applications using the PDO library will not automatically negotiate, and require `sslmode=require` be added to the connection string.

Postgresql is also being upgraded to 9.6 as 9.1 is not longer supported. There should be no major compatibility issues, and all databases will be automatically migrated. Postgres will still listen on port `5433`; some time after migration is complete we will enable postgresql 10 on port `5432`.
Line 40: Line 142:
MySQL is still using Percona 5.6. === MySQL ===

The MySQL migration should be transparent. We are staying on Percona MySQL 5.6, and are using an stunnel to transparently/securely proxy connections between the datacenters during migration.

A guide to moving your services to our new virtual infrastructure at DigitalOcean.

Contents

  1. Changes Requiring Action
  2. Important Dates
  3. Overview of New Machines
    1. New Features
      1. Networking Change: IPv6 is Supported
      2. Web Server
  4. Using the New Shell Server
    1. Using Cron: Permissions No Longer Needed
  5. Moving Web Sites
    1. PHP
    2. Low-level domain users
    3. Proxied Servers
  6. Changes to Databases
  7. Postgresql
    1. MySQL
  8. Changes to XMPP
  9. Features Coming After Migration

1. Changes Requiring Action

If you...

  • have a crontab

  • are running any daemons on bog

  • use postgresql

  • are using the low level domain domtool type instead of dom

...you will need to take some manual actions during the migration or your services may break.

Migration should be otherwise transparent.

2. Important Dates

TBD

  • All domains converted from suphp/cgi php to fastcgi based php: 2018-10-10 (done!)
  • mailman list migration
  • email migration
  • database migration (incurring downtime)
    • TODO: estimated max downtime window for migrating databases after doing test run
  • volumes migration

3. Overview of New Machines

The new machines that members will directly interact with are:

  • marsh, the new shell server. replaces bog. This is the server you will login to.

  • shelob, the new web server. replaces navajos

  • minsky, the new mail server. replaces mccarthy

For a full list of servers at the new host and their purposes, see Hardware#Digital_Ocean

All servers are now running Debian Stretch (the latest stable release). Packages that were requested through the members portal on both bog and navajos have been installed on marsh and shelob so if your software works now, it ought to work on the new servers. If you are using compiled binaries that link against system libraries you might need to recompile.

3.1. New Features

3.1.1. Networking Change: IPv6 is Supported

Core HCoop services (ssh, email, dns, ...) are now IPv6 enabled. Members with native IPv6 are encouraged to test the new services and report any problems.

By default, domtool will not generate AAAA (IPv6) DNS records for your domains, but this will be enabled for the dom type after all sites are migrated.

3.1.2. Web Server

All sites are now using mod_fcgid based php 5.6, which should provide a significant improvement in performance.

The new webserver has the latest Apache 2.4.x release, so the sslCertificateChainFile directive will no longer be required for https -- the certificates can contain the full chain now.

4. Using the New Shell Server

The new shell server may be accessed using ssh marsh.hcoop.net. Thanks to openafs, both the old and new infrastructure share the same volumes and you can access your data from either.

4.1. Using Cron: Permissions No Longer Needed

All members have cron enabled by default on the new login server and no longer need to specially request permissions.

Members with crontabs on bog will need to re-create the crontab on marsh. This should just be a matter of copying the crontab contents to marsh, and removing it from bog afterward.

5. Moving Web Sites

We are upgrading from apache 2.2 to 2.4, but have a configuration that should behave identically the one currently used on ServerNavajos. We are currently using mod_access_compat (Allow/Deny/Satisfy directives) instead of the newer Require access framework so that existing configurations do not need to be updated. At some point in the future we will update domtool and convert member configurations to the new access control directives.

If you are using the dom type, the move should be transparent; we will update DefaultWebNode to the new web server and reconfigure all domains on the date scheduled for transitioning to the new servers. If you would like to migrate your domains early, you may set the environment variable DefaultWebNode = "shelob"; in your configuration to force the domain to be configured on the new webserver.

5.1. PHP

The new webserver is running php 5.6, with a configuration matching the existing production configuration.

We have supported a simple fastcgi based php for a while now, but have not widely deployed it. Our current method of supporting php-cgi based php (suphp) has been removed from debian stretch, and shelob only supports fastcgi based php.

All domains will be automatically upgraded to fastcgi based php, and plain cgi php will be disabled before migration.

After we migrate all domains, we will be able to enable php 7.2 with minimal effort (due to domtool limitations, it's not feasible to support it while ServerNavajos is in production).

(fastcgi php is mandatory now)

5.2. Low-level domain users

You're on your own, possibly ;-)

If you use vhost or vhostDefault to configure your websites, you will need to set the WebPlaces environment variable to host them on shelob: 

domain "yourdomain"
with
  vhostDefault where
    WebPlaces = [web_place_default "shelob"];
  with
    ...
  end;
end;

Any dnsIP or dnsDefault records pointing toward navajos_ip or "69.90.123.70" need to be changed to point to shelob: 

domain "yourdomain"
with
  ...
  dnsDefault shelob_ip;
end;

5.3. Proxied Servers

Will need to be moved to marsh, but will still work when connecting from the new webserver to bog. Connections will be going unencrypted over the Internet however!

todo: example of proxied server config and update.

6. Changes to Databases

7. Postgresql

Postgres users must take action! Due to our usage of gssapi and ident for authentication, we cannot set up a simple stunnel for secure connections between the datacenters. To ensure the security of your data, connections from one datacenter to the other will require ssl be enabled in postgres. Applications based on libpq ought to negotiate ssl automatically, but php applications using the PDO library will not automatically negotiate, and require sslmode=require be added to the connection string.

Postgresql is also being upgraded to 9.6 as 9.1 is not longer supported. There should be no major compatibility issues, and all databases will be automatically migrated. Postgres will still listen on port 5433; some time after migration is complete we will enable postgresql 10 on port 5432.

dbtool commands for postgres will now use version postgres-9 instead of postgres-9.1.

7.1. MySQL

The MySQL migration should be transparent. We are staying on Percona MySQL 5.6, and are using an stunnel to transparently/securely proxy connections between the datacenters during migration.

8. Changes to XMPP

We are now using ejabberd 18.06, which brings ...

9. Features Coming After Migration

Once migration is completed, a few features will be implemented as soon as feasible:

  • PHP 7.2 support
  • Postgresql 10.x support

  • Automated integration with letsencrypt in DomTool.

CategoryMemberManual

DigitalOceanMigrationGuide (last edited 2018-12-09 22:30:35 by ClintonEbadi)