|
Size: 12322
Comment:
|
← Revision 114 as of 2012-12-17 21:12:48 ⇥
Size: 9794
Comment: this is leftover information from the fyodor -> peer1 transition!
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| For the purposes of this page, we'll use the name New to refer to the servers hosted at Peer 1 (which are deleuze, mire, and eventually abulafia) and Old to refer to any servers that we've used previously. | #pragma section-numbers off {{{#!wiki warning This page is historical information from the migration to Peer1 in 2006. For information on migrating to Navajos and Bog, see NavajosBogMigrationGuide instead. }}} This page describes the steps that members using the old machines need to take in order to migrate to the new machines. For the purposes of this page, we'll use the name New to refer to the servers hosted at Peer 1 (which are deleuze, mire, and eventually abulafia and krunk) and Old to refer to any servers that we've used previously. <<TableOfContents>> |
| Line 4: | Line 14: |
| '''14 September 2007''': Migration as begun! Use this page to learn how to create a new account and migrate your data. A user creation script will be ran periodically each day. | |
| Line 6: | Line 15: |
| [[TableOfContents()]] | Everyone has been migrated to the new machines. People who haven't finished this yet are holding up the works and should get their acts together. A deadline of Monday, April 14, was announced by e-mail. If you aren't migrated by then, your service may be degraded arbitrarily. |
| Line 10: | Line 19: |
| We are now offering limited-access accounts on the new infrastructure (see NewServersSetup) on a "beta test" basis to all users who have accounts on fyodor. These accounts come with no uptime or service guarantee; during the next few weeks we may need to temporarily disable them from time to time. These accounts will allow you full access to your space in AFS (currently 400MB per user) and the ability to log in to mire.hcoop.net via ssh. Currently NO OTHER SERVICES are officially supported on the new infrastructure (for example, email or serving HTTP), although we hope to make these available soon. |
Having an account on our new machines will allow you to have full access to your space in AFS (currently 400MB per user) and the ability to log in to {{{mire.hcoop.net}}} via ssh. |
| Line 18: | Line 25: |
| == Step 1: Get a mire account == | == Step 1: Get a New account == |
| Line 20: | Line 27: |
| We have created a file in each fyodor user's home directory called `.mire-password`. This file is readable only by you and the admin who will be processing the account creations. | 1. ssh to `hcoop.net` as usual. 1. Run this command line: `migrationpw` 1. Follow the on-screen directions. 1. Wait for an e-mail from the user creation script. (This stage requires that a human run the script periodically to watch for failures, but one of us should run it several times a day.) |
| Line 22: | Line 32: |
| If you would like to get a mire account, please put your desired password in the file `~/.mire-password`. Please be VERY careful not to delete this file or change its permissions. The recommended procedure is to type:{{{ cat > ~/.mire-password}}} |
The password you set will go into our new Kerberos database, allowing log-in to mire and any other of our servers that we choose to enable for non-admin shell access. You will also use this password for authentication to other services, like e-mail and members-only HCoop web sites. |
| Line 25: | Line 34: |
| and then type your desired password, press enter, and then press control-D. After you do this, please run the command:{{{ ls -l ~/.mire-password}}} and make sure that the permission bits are `-rw-rw----` (user and group can read and write) and the group for the file is set to `megacz`. We will be running an account-creation script approximately once a day. After you put your password in `~/.mire-password`, your account will be created the next time the script is run, typically no more than 24 hours after you specify your password. An email will be sent to your hcoop account to let you know that your account has been created. Your .mire-password file WILL BE DELETED when your account is created, so please make sure you memorize it before writing it to the file. |
An e-mail will be sent to your HCoop account to let you know that your account has been created. Be sure to memorize your password, as it won't be saved anywhere unencrypted once the account creation script runs! |
| Line 35: | Line 38: |
| Now you may attempt to login using your favorite SSH client or the new AJAX SSH service at http://ssh.hcoop.net. It requires a modern browser that cooperates with AJAX. | Now you may attempt to login to {{{mire.hcoop.net}}} using your favorite SSH client or the new AJAX SSH service at [[http://ssh.hcoop.net/]]. The latter requires a modern browser that cooperates with AJAX. === SSH Public Key is Obsoleted === You can no longer use SSH public key authentication. Kerberos authentication ("`ssh -K`") ''is'' supported, for passwordless log-in. Some day, someone might implement the Kerberos support needed to make SSH public key auth work again. See MemberManual/DistributedSecurity for more information on all of this. |
| Line 39: | Line 46: |
| === SSH Public Key is Obsoleted === | === DenyHosts === |
| Line 41: | Line 48: |
| You can no longer use SSH public key authentication anymore. ["Kerberos"] authentication ("`ssh -K`") ''is'' supported, for passwordless log-in. Some day, someone might implement the Kerberos support needed to make SSH public key auth work again. See RealSecurity for more information on all of this. If you fail to log in correctly a few times the DenyHosts scripts will lock you out. Currently any blocked IP's are purged after a week, so if you don't want to wait you'll need to submit a ticket, or if you can't access the portal to do this you'll need to send an email to admins@hcoop.net. |
If you fail to log in correctly quite a few times, the Deny``Hosts scripts might lock you out. Currently any blocked IP's are purged after a week, so if you don't want to wait you'll need to submit a ticket, or if you can't access the portal to do this you'll need to send an email to <<MailTo(admins AT hcoop DOT net)>>. |
| Line 47: | Line 52: |
| [https://members2.hcoop.net/ The new portal] uses the same password you use to log in to mire. That is, if you haven't created a mire account yet, then you can't access the new portal. | [[https://members.hcoop.net/|The new portal]] uses the same password you use to log in to mire. That is, if you haven't created a New account yet, then you can't access the new portal. |
| Line 49: | Line 54: |
| You should use the new portal for all administrative requests related to the new servers, except for... | You should use the new portal for all administrative requests, except for the specialized request types (e.g., domains, firewall rules, etc.) when they relate to fyodor. |
| Line 53: | Line 58: |
| We recommend that you tell fyodor to ''dual-deliver'' all of your mail so that one copy goes to deleuze and one copy goes to fyodor. That way you can start reading your email via deleuze, but if anything goes wrong you can just switch back to fyodor. | We recommend that you tell fyodor to ''dual-deliver'' all of your mail so that one copy goes to deleuze (our new main server) and one copy goes to fyodor. That way you can start reading your email via deleuze, but if anything goes wrong you can just switch back to fyodor. |
| Line 67: | Line 72: |
| You can also copy the contents of your mailboxes from fyodor to deleuze. To do this, log in to fyodor and type: | You can also copy the contents of your mailboxes from fyodor to mire (actually to our shared AFS filesystem by way of mire). To do this, log in to fyodor and type the following. |
| Line 73: | Line 78: |
| = Bugzilla = | Then log into mire and '''remove the {{{~/Maildir/shared-folders}}} directory, if it exists'''. Also, change the contents of {{{~/Maildir/shared-maildirs}}} on mire to: {{{ SpamAssassin /var/local/lib/spamd/Maildir }}} |
| Line 75: | Line 82: |
| We have a [https://bugzilla.hcoop.net/ Bugzilla] that we are using for managing support requests that don't fit into the special categories handled by the portal. If you've completed the migration steps above that create your mire account, then you can use the same username and password to access Bugzilla. You can also start from the portal, which links to Bugzilla from the support page. | = Migration strategy = |
| Line 77: | Line 84: |
| = Databases = | == Making a subdomain on fyodor and pointing it at mire == |
| Line 79: | Line 86: |
| ''Here lie interim dbtool docs until migration is done, at which time they will probably move to UsingDatabases.'' | It is possible to test out your setup on the new servers by making a new subdomin on the old machine that points to the new machine. This way you can hone your new setup until it's as good as the old, while still having access to the old. |
| Line 81: | Line 88: |
| To manage your database user and databases, the basic syntax is `dbtool <DBTYPE> <COMMAND>`, where `<DBTYPE>` is `postgres` or `mysql`. | First, make a directory in your {{{/etc/domains/TLD/DOMAIN}}} folder on the old machine. '''TLD''' is the Top-Level Domain of your domain. For example, it might be `com`, `net`, `us`, `in` etc. '''DOMAIN''' is your domain, and '''SUB''' is the new subdomain that you would like to use. '''SUB''' should not include any of the text in '''DOMAIN''', and should have no periods. |
| Line 83: | Line 90: |
| The `adduser` command creates a database user for you, with the same name as your UNIX log-in name. In the case of `mysql`, you will be prompted for a password and confirmation re-entry in the usual manner. | {{{ mkdir /etc/domains/TLD/DOMAIN/SUB }}} |
| Line 85: | Line 94: |
| The `passwd` command allows you to reset the password. (Useless for `postgres`, where we use only ident authentication) | In that directory, make a file called {{{.dns}}} with the following contents. |
| Line 87: | Line 96: |
| The `createdb <DBNAME>` command creates a database named `<USERNAME>_<DBNAME>`, and `dropdb <DBNAME>` drops a database. For security/accounting reasons, you won't be granted permissions to drop a database in the usual way, through an SQL session. | {{{ Primary ns ns Default 69.90.123.68 }}} |
| Line 89: | Line 101: |
| Sometimes we update our policies for which permissions users are granted on their databases. To re-set the permissions for a database after such a change, run `grant <DBNAME>`. At the moment, this can only ever be necessary for MySQL databases. | Then, run the `domtool` command to finalize your changes on Fyodor. |
| Line 91: | Line 103: |
| To access your database use the following on mire: `mysql -p -h mysql <USERNAME>_<DBNAME>` or `psql -h postgres <USERNAME>_<DBNAME>` | Now request control of the '''DOMAIN''' using the new portal (http://members.hcoop.net). When you receive notification of control, you can then log into mire.hcoop.net and configure DomTool so that Apache knows it can serve your '''SUB'''domain. Please take a look at [[MemberManual/UsingDomtool| using DomTool]], the [[DomTool/UserGuide| DomTool user guide]], and [[DomTool/Examples| DomTool examples]] to learn how to do this. You'll probably want to take a look at the `vhost` directive. |
| Line 93: | Line 105: |
| = DNS = | = Quickies = |
| Line 95: | Line 107: |
| We are purposely not sending any DNS data from New to Old, which means that you need to change domains at your registrar if you want New to be authoritative for them. The proper nameservers are ns1.hcoop.net and ns3.hcoop.net, in that order. Keeping ns.hcoop.net and ns2.hcoop.net '''will not work'''. | Be sure to read through the chapters of the MemberManual that interest you. The following are some very quick overviews of things that have changed. |
| Line 97: | Line 109: |
| = Domains = | == DNS == We are purposely not sending any DNS data from Old to New, which means that you need to change domains at your registrar if you want New to be authoritative for them. The proper nameservers are `ns1.hcoop.net` and `ns3.hcoop.net`, in that order. Keeping `ns.hcoop.net` and `ns2.hcoop.net` '''will not work'''. == Domains == |
| Line 101: | Line 117: |
= Home = |
== Home == |
| Line 106: | Line 121: |
| = Email = | == OpenAFS and permissions == First of all, UNIX permissions carry no weight with AFS -- therefore they are useless to you. Instead, use Access Control Lists (ACL), which are a far more powerful way of restricting access to parts of a file tree. Read MemberManual/GettingStarted for further information on AFS concepts. |
| Line 108: | Line 124: |
| == .forward == | See the [[../TransferringFiles/OpenAFS|OpenAFS]] subpage to find installation directions for various operating systems. |
| Line 110: | Line 126: |
| {{{~/.forward}}} files should have the same effect that they do with our old setup, but on the new setup they are located at {{{~/.public/.forward}}} instead of {{{~/.forward}}}. See [https://bugzilla.hcoop.net/show_bug.cgi?id=81#c2 this bug] for more. | == Mailman == |
| Line 112: | Line 128: |
| == IMAP == | See the [[MemberManual/Email/MailingLists|Mailing Lists page]] for details, including how to migrate existing lists to the new machines. |
| Line 114: | Line 130: |
| SSL IMAP is available via SSL at port 993, using hostname {{{deleuze.hcoop.net}}}. | == MoinMoin == |
| Line 116: | Line 132: |
| STARTTLS IMAP is available on port 143, using hostname {{{deleuze.hcoop.net}}}. | See the [[MemberManual/ServingWebsites/MoinMoin|MoinMoin configuration page]] for details on how to set up MoinMoin and how to migrate data to match the new version of MoinMoin that we have installed. |
| Line 118: | Line 134: |
| == POP3 == POP3 access is available via SSL at port 995, using hostname {{{deleuze.hcoop.net}}}. If you're using Thunderbird, make sure to uncheck "Use secure authentication". Do not use port 110; it is not available, because no good way of securing normal POP3 has been found by the admins. == procmail == The page ProcmailExample has been updated for the new setup. Basically: * Use the file {{{~/.procmail.d/procmailrc}}} instead of {{{~/.procmailrc}}}. * Write any procmail logs in {{{~/Maildir}}} rather than elsewhere. * Use appropriate values for the HOME, MAILDIR, and DEFAULT options, based on those in ProcmailExample. == Virtual mailboxes == The `vmail` program from fyodor has been updated for the new servers. Here's a quick run-through of how to invoke the new version. Like before, you always invoke it with `vmail $DOMAIN $COMMAND`, which indicates that you are configuring the virtual mailboxes for domain `$DOMAIN` for which you have DomTool permissions. The valid commands are: * `list`: Print the mapping from usernames to mailbox directories for `$DOMAIN`. * `add $USER $MAILBOX`: Add a mapping from `$USER@$DOMAIN` to a Maildir directory `$MAILBOX`. You'll be prompted to enter a password for the user, which he can then use to access IMAP, POP, or restricted SMTP services. * `passwd $USER`: Reset a virtual user's password. * `rm $USER`: Remove a mapping. The mailbox directory remains for you to deal with as you like. = rsync = |
== rsync == |
| Line 143: | Line 138: |
| = Security = See RealSecurity for some technical notes on security. == Securing Directories == First of all, UNIX permissions carry no weight with AFS -- therefore they are useless to you. Instead, use Access Control Lists (ACL), which are a far more powerful way of restricting access to parts of a file tree. That said, when a new directory is created inside $HOME, its ACL defaults to allow listing by any authenticated party on HCoop. Note that ACLs cannot be set on individual files. They inherit the ACL from its parent directory. If you wish to make a directory within your $HOME completely private so that only you can list, read, and write, do this: {{{ mkdir ~/private fs setacl -clear ~/private <USERNAME> all }}} Note that {{{-clear}}} removes any previously set ACLs and {{{<USERNAME> all}}} sets full access to the directory's contents to the specified user. Therefore, if you have a directory in $HOME that you wish to make only accessible to you (such as ~/.ssh or ~/documents), use: {{{fs setacl -clear ~/<DIRECTORY> <USERNAME> all}}}. '''NOTE''': ''do not'' apply any ACLs to ~/domtool or ~/public_html, as they may become inaccessible to necessary services such as a webserver. If you wish to view the ACLs on a specific directory, such as any you have just applied an ACL, use: {{{fs la <DIRECTORY>}}} == Log-In Security == We use the [http://denyhosts.sourceforge.net/ DenyHosts] package to help prevent user accounts from brute-force attacks. If a user fails to login within several attempts, then the offending originating IP will be blacklisted in order to prevent additional attempts. If the individual attempts to login again, then he will see something similar to the following: {{{ssh_exchange_identification: Connection closed by remote host}}} The blacklist expires IPs after a predetermined period if time. Typically, most users will not be affected by the blacklisting, but if you are, you will want to contact the hcoop-sysadmin list to get your IP address removed from the list. = WebDAV = |
== WebDAV == |
| Line 175: | Line 141: |
| Note that you can only use WebDAV on directories that have {{{system:anyuser rl}}} as part of its ACL. You'll be able to write even if {{{system:anyuser}}} does not. See Securing Directories on this page for additional details on directory ACLs. | Note that you can only use WebDAV on directories that have {{{system:anyuser rl}}} as part of its ACL. You'll be able to write even if {{{system:anyuser}}} does not. |
| Line 177: | Line 143: |
| = public_html = Your public_html directory under USER/public_html is available via http through http://deleuze.hcoop.net/~USER/ |
== webmail == A Squirrelmail instance for reading your email on the new servers is available at [[https://mail.hcoop.net/]]. == Web sites == Your {{{~/public_html}}} directory is available via HTTP through {{{http://deleuze.hcoop.net/~USER/}}}. Eventually this will change to {{{http://hcoop.net/~USER/}}}. Due to consequences of AFS authentication, we don't plan to allow dynamic content (CGI, PHP, etc.) via hcoop.net/~you/... on New. If you don't have a domain hosted at HCoop, but want to serve dynamic content, then you can request an hcoop.net subdomain (example: {{{USER.hcoop.net}}}, where USER is your username) via [[http://bugzilla.hcoop.net/]]. See the chapter on [[MemberManual/ServingWebsites|Serving Websites]] for more details. ---- CategoryHistorical |
This page is historical information from the migration to Peer1 in 2006. For information on migrating to Navajos and Bog, see NavajosBogMigrationGuide instead.
This page describes the steps that members using the old machines need to take in order to migrate to the new machines.
For the purposes of this page, we'll use the name New to refer to the servers hosted at Peer 1 (which are deleuze, mire, and eventually abulafia and krunk) and Old to refer to any servers that we've used previously.
Contents
Status of Migration
Everyone has been migrated to the new machines. People who haven't finished this yet are holding up the works and should get their acts together. A deadline of Monday, April 14, was announced by e-mail. If you aren't migrated by then, your service may be degraded arbitrarily.
Summary of what exactly is going on here
Having an account on our new machines will allow you to have full access to your space in AFS (currently 400MB per user) and the ability to log in to mire.hcoop.net via ssh.
Requesting an account on the new infrastructure will not affect your fyodor account. You will have access to both accounts until after all migration is complete.
Getting started
Step 1: Get a New account
ssh to hcoop.net as usual.
Run this command line: migrationpw
- Follow the on-screen directions.
- Wait for an e-mail from the user creation script. (This stage requires that a human run the script periodically to watch for failures, but one of us should run it several times a day.)
The password you set will go into our new Kerberos database, allowing log-in to mire and any other of our servers that we choose to enable for non-admin shell access. You will also use this password for authentication to other services, like e-mail and members-only HCoop web sites.
An e-mail will be sent to your HCoop account to let you know that your account has been created. Be sure to memorize your password, as it won't be saved anywhere unencrypted once the account creation script runs!
Step 2: Try logging in
Now you may attempt to login to mire.hcoop.net using your favorite SSH client or the new AJAX SSH service at http://ssh.hcoop.net/. The latter requires a modern browser that cooperates with AJAX.
SSH Public Key is Obsoleted
You can no longer use SSH public key authentication. Kerberos authentication ("ssh -K") is supported, for passwordless log-in. Some day, someone might implement the Kerberos support needed to make SSH public key auth work again. See MemberManual/DistributedSecurity for more information on all of this.
That being said, if you've always been typing a password to log in via SSH and don't care to do otherwise, then you don't need to bother reading this section!
DenyHosts
If you fail to log in correctly quite a few times, the DenyHosts scripts might lock you out. Currently any blocked IP's are purged after a week, so if you don't want to wait you'll need to submit a ticket, or if you can't access the portal to do this you'll need to send an email to <admins AT hcoop DOT net>.
Step 3: Visit the new portal
The new portal uses the same password you use to log in to mire. That is, if you haven't created a New account yet, then you can't access the new portal.
You should use the new portal for all administrative requests, except for the specialized request types (e.g., domains, firewall rules, etc.) when they relate to fyodor.
Step 4: Have your mail dual-delivered
We recommend that you tell fyodor to dual-deliver all of your mail so that one copy goes to deleuze (our new main server) and one copy goes to fyodor. That way you can start reading your email via deleuze, but if anything goes wrong you can just switch back to fyodor.
To do this, put the following lines in your ~/.forward file on fyodor. Note that the comment on the first line is mandatory -- it tells exim that this forward file uses special exim features. If your username was fred, you would put this in your ~/.forward:
# Exim filter deliver fred deliver fred@deleuze.hcoop.net
and you mail will be dual-delivered.
Step 5: Copy your existing email
You can also copy the contents of your mailboxes from fyodor to mire (actually to our shared AFS filesystem by way of mire). To do this, log in to fyodor and type the following.
rsync -are ssh --no-g --progress --verbose ~/Maildir/ mire.hcoop.net:Maildir/
Then log into mire and remove the ~/Maildir/shared-folders directory, if it exists. Also, change the contents of ~/Maildir/shared-maildirs on mire to:
SpamAssassin /var/local/lib/spamd/Maildir
Migration strategy
Making a subdomain on fyodor and pointing it at mire
It is possible to test out your setup on the new servers by making a new subdomin on the old machine that points to the new machine. This way you can hone your new setup until it's as good as the old, while still having access to the old.
First, make a directory in your /etc/domains/TLD/DOMAIN folder on the old machine. TLD is the Top-Level Domain of your domain. For example, it might be com, net, us, in etc. DOMAIN is your domain, and SUB is the new subdomain that you would like to use. SUB should not include any of the text in DOMAIN, and should have no periods.
mkdir /etc/domains/TLD/DOMAIN/SUB
In that directory, make a file called .dns with the following contents.
Primary ns ns Default 69.90.123.68
Then, run the domtool command to finalize your changes on Fyodor.
Now request control of the DOMAIN using the new portal (http://members.hcoop.net). When you receive notification of control, you can then log into mire.hcoop.net and configure DomTool so that Apache knows it can serve your SUBdomain. Please take a look at using DomTool, the DomTool user guide, and DomTool examples to learn how to do this. You'll probably want to take a look at the vhost directive.
Quickies
Be sure to read through the chapters of the MemberManual that interest you. The following are some very quick overviews of things that have changed.
DNS
We are purposely not sending any DNS data from Old to New, which means that you need to change domains at your registrar if you want New to be authoritative for them. The proper nameservers are ns1.hcoop.net and ns3.hcoop.net, in that order. Keeping ns.hcoop.net and ns2.hcoop.net will not work.
Domains
See the DomTool page for instructions on managing your domains with the new setup. The configuration files are in a vastly different format, but they have a better-defined syntax that should be relatively easy to understand.
Home
Your home directory is now managed by AFS. You will enter it by default when logging in to mire.hcoop.net via ssh. Type pwd to see what the path is. It will look like /afs/hcoop.net/user/u/us/username. Some directories have been created for you already, so that they have the correct permissions for things like serving web pages and delivering mail.
OpenAFS and permissions
First of all, UNIX permissions carry no weight with AFS -- therefore they are useless to you. Instead, use Access Control Lists (ACL), which are a far more powerful way of restricting access to parts of a file tree. Read MemberManual/GettingStarted for further information on AFS concepts.
See the OpenAFS subpage to find installation directions for various operating systems.
Mailman
See the Mailing Lists page for details, including how to migrate existing lists to the new machines.
MoinMoin
See the MoinMoin configuration page for details on how to set up MoinMoin and how to migrate data to match the new version of MoinMoin that we have installed.
rsync
If you're using rsync to transfer data to the new servers, the "-a" option by itself won't work properly because rsync attempts to chgrp the transferred files. Use "-a --no-g" instead of "-a".
WebDAV
WebDAV is accessible at https://dav.hcoop.net/. WebDAV is useful when working on a website using systems that cannot mount an AFS share. For details on how to setup WebDAV, take a look at http://research.cs.berkeley.edu/doc/dav/
Note that you can only use WebDAV on directories that have system:anyuser rl as part of its ACL. You'll be able to write even if system:anyuser does not.
webmail
A Squirrelmail instance for reading your email on the new servers is available at https://mail.hcoop.net/.
Web sites
Your ~/public_html directory is available via HTTP through http://deleuze.hcoop.net/~USER/. Eventually this will change to http://hcoop.net/~USER/.
Due to consequences of AFS authentication, we don't plan to allow dynamic content (CGI, PHP, etc.) via hcoop.net/~you/... on New. If you don't have a domain hosted at HCoop, but want to serve dynamic content, then you can request an hcoop.net subdomain (example: USER.hcoop.net, where USER is your username) via http://bugzilla.hcoop.net/. See the chapter on Serving Websites for more details.