1. General Specifications

1.1. All Machines

• RAID1
• Dual Socket
  • Initially install one four-core processor in each (six-core processors are dramatically more expensive)
• Remote reboot and console ability
  • Most of the servers I have quickly speced out appear to have minimal remote reboot and console ability built in with fancier addon cards for web interfaces and other things; we should be ok with just the baseline module.

1.2. Core Services Machine

• 2U
• 8GB RAM (initially)
  • We should probably use 2G or 4G modules to ensure we can upgrade to 16/32G without having to replace memory
• Dual RAID1 Arrays (ideally room for 6 drives; 2 hot spares?)
  • Large (750G - 1TB?) array for AFS and only AFS
  • Smaller (250G?) array for OS images / databases
• Redundant power supplies

1.3. User Services Machine

• 1U
• 8G RAM
  • Mostly allocated to user daemon image
• Single RAID1 Array
  • Not too large (250G? Any smaller does not seem cost effective)
  • Local VM image disk space
  • Some amount of space for users (80G?)
    • Users who need fast local disk could request a portion of this (enforced using quotas etc.)

1.4. Serial Console Server / IPKVM

We need some type of worst-case access to the physical consoles of the servers. IPKVM/KVM units are fairly expensive, and potentially don't really need everything they give us since we are not running X or anything remotely. Given that we have a nice IPKVM and KVM setup now we may want to ship that to the new data center, but then we will be running for a period of time with no equivalent to physical access remotely on our setup that is known to occasionally go down and be inaccessible.

Alternatively we could procure a serial console for a bit less money and have access to the serial consoles of every machine, which ought to be just as good as having physical keyboard/monitor access via vnc. Additionally we would gain access the the IPMI capabilities of the connected machines (which may lower the cost of each machine by $200-$300 since we could avoid buying service processors for them). If we got a fancy switch it might also have a serial console for configuration.

1.4.1. General Specs

• 1U/2U
• Access to 8 machines ideally, four minimally

1.4.2. Console Server

1.4.2.1. Avocent Cyclades CS 8-Port Console Server

• Cyclades CS 8-Port Console Server Product Page

• Price: $482 (Ecomeletronics)

Does not support IPMI commands it appears; unless the BMCs of the servers we get have some type of text console interface over serial this is suboptimal.

1.4.2.2. OpenGear CM400x

These are not rack mount units, but they seem to be more in line with what we need from a console server. It appears (need to check the docs more thoroughly) they support connecting to IPMI devices via the network (which it seems we can secure by restricting IPMI access to the IP of the console server) in additional to supporting direct serial consoles.

These devices also are running entirely Free Software and there is a dev kit that look reasonably easy to use so we can customize them. ClintonEbadi contacted Opengear and learned that th CM400x consoles cannot be reconfigured to coexist on multiple protocol based vlans making them a bit less useful than they initially appeared.

• OpenGear CM4000 Series Product Page

• Firmware / Source Code Downloads

1.4.2.2.1. OpenGear CM4008

• 8 Serial Console Ports

• Price: $495 (OpenGear Store)

1.4.2.2.2. OpenGear CM4001

• 1 Serial Console Port

• Price: $225 (OpenGear Store).

If we use Serial-over-LAN (assuming it can be secured without a dedicated management lan) for everything the CM40001 should be fine for our use.

1.4.2.3. OpenGear IM4004-5

• OpenGear IM4004-5 Product Page

• OpenGear CM/IM Manual

• 4 serial console ports
• 1 uplink ethernet port
• 4 port integrated switch for Management LAN

• Price: $495 (Opengear Store)

The CM4001 cannot coexist on multiple protocol-based vlans so this looks like our best bet for a console server -- we can connect eth0 of each server to the console server's management lan and eth1 to the primary lan. The hardware is fairly powerful and we could later daisy chain the dumb switch to this and add six more devices to the management lan. This has a few other useful features e.g. it can act as a tftp server (pxe boot!). The mostly Free Software and semi-customizable userspace are a definite win. Even better is that the fancy remote console application is also Free Software, runs on GNU/Linux (Java),

• SDT Connector

• okvm project

1.5. Network Switch

• 1U
• Gigabit
• 8- or 16- ports

If we can get by with a CM4001 we should spend a bit more on a proper smart switch so that we can setup multiple vlans. Initially at least a public vlan and a private IPMI-only protocol based vlan. Later on we may want to experiment with routing database and afs traffic locally on a vlan with jumbo frames enabled (according to a cursory google this would increase database throughput but would likely have little effect on afs until openafs 1.6 is released with the new RxTCP transport layer).

1.5.1. Unmanaged

1.5.1.1. US Robotics 8-port Gigabit Switch

• USR997930A Product Page

• Price: $65.46 (Amazon.com Free shipping)

This looks like it will be an acceptable switch until we can afford (or need) a managed switch.

2. Shopping list

2.1. Non-Dell Vendors

From other vendors, systems comparable to the Poweredge 2970 cost...

• ASA Computers: over $2000 for a reasonably configured ASA2001-O2Q-S2-R

• Penguin Computing Altus 2701

  • Dual, AMD Opteron 2380, 2.5GHz, Quad Core, 4x512KB L2, 6MB L3 Cache
  • 2GB DDR2-667MHz ECC RAM (2 x 1GB)
  • 250GB, SATA2, 7200RPM
  • LSI 3Gbps 8-port SAS controller, no RAID
  • Intel 9402PT Dual Port Copper Gigabit PCI-E Ethernet Card
  • DVD-ROM / CD-RW
  • Preload, CentOS, Version 5
  • Standard 3-Year Warranty
  • Keep your hard drive support option

  • Price: $3,440.32

• IBM's build-your-own tool appears to be broken, but a System x3655 machine with one CPU, 2GB memory, and, AFAICT, no disks costs $1,332.00

2.2. Option A

• Dell poweredge R410

  • Intel® Xeon® E5520, 2.26Ghz, 8M Cache, Turbo, HT, 1066MHz Max Mem

  • 1GB ram
  • Only one of the cheapest 160GB hard drives
  • Baseboard Management Controller
  • no CD drive (we'll boot from USB)
  • 1-year warranty

  • Price: $1050

• PowerEdge 2970

  • Quad Core AMD Opteron™ 2372HE 2.1GHz 4x512K Cache 1Ghz HyperTrnsprt

  • SAS 6/iR Integrated, x6 Backplane, 1x6 Backplane for 3.5-inch Hard Drives
  • One default 160GB HD
  • No CD/DVD
  • 3Yr Basic Hardware Warranty Repair: 5x10 HW-Only, 5x10 NBD Onsite (cheapest)

  • Price: $709

• Ram

  • 4GB DRR3 x 2 for user services (2 x $121.49)

  • 4GB DRR2 x 2 for core services (2 x $101.49)

• Drives

  • 1TB x 2 for core services(2 x $74.99)

  • 500GB x 4 (4 x $54.99)

• Serial console server
  • $400

• Total price: 2974.9 + various other shipping/tax/small things we might need to pay

2.3. Option B

• 2xDell PowerEdge 2970

  • 2U
  • 2xQuad Core Opteron 2372HE (2.1GHz / 4x512K L2)
  • 8G RAM (4x2G 800MHz DDR / Dual Ranked)
  • Base 160G SATA drive
  • No CD-ROM (initial OS install from USB stick)
  • Price: $1031*2 = $2061
• Opengear IM4004-5 Management Gateway
  • Price: $495
• US Robotics Unamanged 8-port GigE Switch
  • Price: $65.46
• Drives

  • 1TB x 2 for core services(2 x $74.99)

  • 500GB x 4 (4 x $54.99)

• Cables
  • Price: ?
• Rack mounting hardware

  • Price: $129.99 * 2 = $259.98

    • Open box, but two sets of rails for the price of one (with a 90 day return warranty)

• Tenative Total Price: $3252.38 (more likely ~$3500 when all is said and done)

2.3.1. Why Two PowerEdge 2970s

Although this setup would use 6U rather than 5U, the PowerEdge 2970 offers a much better price/performance ratio than the 1U R410. For a bit less than a single processor R410 we could have eight cores on both machines (avoiding a difficult/time consuming processor upgrade later on).