welcome: please sign in

Diff for "NewServerDiscussion2014"

Differences between revisions 5 and 6
Revision 5 as of 2013-01-20 08:32:26
Size: 3300
Editor: ClintonEbadi
Comment:
Revision 6 as of 2013-01-26 08:04:00
Size: 3572
Editor: ClintonEbadi
Comment:
Deletions are marked like this. Additions are marked like this.
Line 15: Line 15:

This means we need hardware featuring something like:

 * 8+ cores
 * At least 16G RAM
 * Room to add more drives
 * Space for a number of VMs
 * Space for afs storage
 * Space for on-site staging of backups
Line 34: Line 43:
As feared, it turns out that we cannot do an afs volume dump to the afs volume partition of the fileserver without grinding hcoop to a halt. ClintonEbadi's early test resulted in fritz thrashing itself into oblivion, and the entire site losing access to the fileserver (never fear! services recovered flawlessly on their own mere minutes after killing the dump). Neither deleuze nor mire nor hopper have the free space required for backing up all of the data we have in afs now, never mind future data. The new server is at least a month and a half away, and we really need backups to have not stopped working six months ago. As feared, it turns out that we cannot do an afs volume dump to the afs volume partition of the fileserver without grinding hcoop to a halt. ClintonEbadi's early test resulted in fritz thrashing itself into oblivion, and the entire site losing access to the fileserver (never fear! services recovered flawlessly on their own mere minutes after killing the dump). Neither deleuze nor mire nor hopper have the free space required for backing up all of the data we have in afs now, never mind future data.
Line 36: Line 45:
Thus: we need a pair of drives for a third RAID1 on fritz, dedicated to backups. We have nearly 200G in data that needs backing up now, so 500G is the absolute minimum. However, much larger drives are not much more expensive and it's not unrealistic that we would have well over 500G of data in need of backing up 18 months from now (inserting new stuff at the data center is a pain).

Alternatively, we could put these drives into the new machine.
Thus: we need a pair of drives for a third RAID1 on fritz or the new machine, dedicated to backups. We have nearly 200G in data that needs backing up now, so 500G is the absolute minimum. However, much larger drives are not much more expensive and it's not unrealistic that we would have well over 500G of data in need of backing up 18 months from now (inserting new stuff at the data center is a pain).
Line 44: Line 51:
RichardDarst claims that we actually have two extra caddies, possibly in the rack, purchased with the initial pair of drives for openafs. @sig@ RichardDarst claims that we actually have two extra caddies, possibly in the rack, purchased with the initial pair of drives for openafs. -- ClintonEbadi <<DateTime(2013-01-26T04:03:59-0400)>>
Line 48: Line 55:
 * Caddy: http://www.amazon.com/Original-SASTu-Drive-Caddy-Poweredge/dp/B004IYFC4I/ref=pd_cp_pc_3 == OpenAFS Drives ==

We're only using a bit under 200G of data now, and have a 1TB partition on fritz for afs. So we probably only need another RAID1 of 1TB disks, although we should investigate larger drives to make sure we hit the optimal price/GB vs predicated needs.

1. New Virtualization Host

Now that we are using KernelVirtualMachine, getting systems into use without weeks or months of delay is finally feasible. We're making good strides toward getting rid of mire, and now the time has come to allow deleuze to retire as well. Unfortunately, fritz is effectively at capacity with the web server and member shell vms, so we need a new machine.

The immediate uses for the new machine:

  • Base system as secondary KDC and openafs file server (probably a good idea to split user and mail volumes across the machines)
  • VM for misc services and admin tasks (dns, domtool-server, the place where admins build software)
  • VM for mail services
  • Probably migrate navajos over to the new machine to make fritz less cramped

We then need it to support at least two more reasonably powered virtual machines so that we can smoothly upgrade to Debian wheezy.

This means we need hardware featuring something like:

  • 8+ cores
  • At least 16G RAM
  • Room to add more drives
  • Space for a number of VMs
  • Space for afs storage
  • Space for on-site staging of backups

1.1. Possible Hardware

Find a price quote, and list any machines you think would suit the task.

1.1.1. Power Edge R515

  • Up to 12 3.5" hot swap SATA drives (think of the possibilities)
  • Dual 6-core Opteron 4170HE (2.1GHz)
  • 32G RAM (4x8G Low voltage 1333MHz)
  • Fancy sliding rails, no cable management arm
  • Non-redundant power supply
  • Only BMC
    • Since we don't have a fancy ipmi console, we might want to consider getting the $100 DRAC Express (basically equivalent to the service processors on our other systems)
  • On 2013-01-20, $2413: pe-515.pdf

2. Backup Drives

As feared, it turns out that we cannot do an afs volume dump to the afs volume partition of the fileserver without grinding hcoop to a halt. ClintonEbadi's early test resulted in fritz thrashing itself into oblivion, and the entire site losing access to the fileserver (never fear! services recovered flawlessly on their own mere minutes after killing the dump). Neither deleuze nor mire nor hopper have the free space required for backing up all of the data we have in afs now, never mind future data.

Thus: we need a pair of drives for a third RAID1 on fritz or the new machine, dedicated to backups. We have nearly 200G in data that needs backing up now, so 500G is the absolute minimum. However, much larger drives are not much more expensive and it's not unrealistic that we would have well over 500G of data in need of backing up 18 months from now (inserting new stuff at the data center is a pain).

2.1. Drive Options

We have a single hot swap caddy on hand because of the failed drive. We need a single 3.5" caddy for the PowerEdge 2970.

RichardDarst claims that we actually have two extra caddies, possibly in the rack, purchased with the initial pair of drives for openafs. -- ClintonEbadi 2013-01-26 08:03:59

If we installed the drives when installing the new systems, we could come with extra caddies for the new machine and decided which machine to put the drives in at that time (if we have caddies, probably fritz, otherwise...).

3. OpenAFS Drives

We're only using a bit under 200G of data now, and have a 1TB partition on fritz for afs. So we probably only need another RAID1 of 1TB disks, although we should investigate larger drives to make sure we hit the optimal price/GB vs predicated needs.

NewServerDiscussion2014 (last edited 2014-05-17 23:46:16 by ClintonEbadi)