welcome: please sign in

Migration2009 / HardwareUpgrade

1. New purchase

We decided to expand the list of HCoop machines with one Dell PowerEdge 2970 as listed on Hardware page under "Awaiting purchase".

2. Hardware reshuffling

With the new purchase, we will rotate our setup in the following way:


Current role

New role



Dell PowerEdge 2970


Main server (Krb5, AFS, LDAP, ...)


Intended as high-power, high throughput, high capacity system with spare disk slots for future capacity increase


Main server

Web server, DB server, local user disk**


Databases and websites will in this way be on the same machine, and databases will use local disk storage. Local user disk access may be possible for users who want to use Krb5/AFS, but still have access to local disk.


Web server

Shell server***, local user disk, web server


shell server which can kinit and aklog (Krb5 and AFS terms) if needed, but is basically intended as a standalone, regular Linux machine that offers local login (makes SSH keys possible), local disk and local web server for people who do not want to use Krb/AFS infrastructure


Email processing

Email processing


3. Old information

Here's information collected prior to our final decision.

4. General Specifications

4.1. All Machines

4.2. Core Services Machine

4.3. User Services Machine

4.4. Serial Console Server / IPKVM

We need some type of worst-case access to the physical consoles of the servers. IPKVM/KVM units are fairly expensive, and potentially don't really need everything they give us since we are not running X or anything remotely. Given that we have a nice IPKVM and KVM setup now we may want to ship that to the new data center, but then we will be running for a period of time with no equivalent to physical access remotely on our setup that is known to occasionally go down and be inaccessible.

Alternatively we could procure a serial console for a bit less money and have access to the serial consoles of every machine, which ought to be just as good as having physical keyboard/monitor access via vnc. Additionally we would gain access the the IPMI capabilities of the connected machines (which may lower the cost of each machine by $200-$300 since we could avoid buying service processors for them). If we got a fancy switch it might also have a serial console for configuration.

4.4.1. General Specs

4.4.2. Console Server Avocent Cyclades CS 8-Port Console Server

Does not support IPMI commands it appears; unless the BMCs of the servers we get have some type of text console interface over serial this is suboptimal. OpenGear CM400x

These are not rack mount units, but they seem to be more in line with what we need from a console server. It appears (need to check the docs more thoroughly) they support connecting to IPMI devices via the network (which it seems we can secure by restricting IPMI access to the IP of the console server) in additional to supporting direct serial consoles.

These devices also are running entirely Free Software and there is a dev kit that look reasonably easy to use so we can customize them. ClintonEbadi contacted Opengear and learned that th CM400x consoles cannot be reconfigured to coexist on multiple protocol based vlans making them a bit less useful than they initially appeared. OpenGear CM4008 OpenGear CM4001

If we use Serial-over-LAN (assuming it can be secured without a dedicated management lan) for everything the CM40001 should be fine for our use. OpenGear IM4004-5

The CM4001 cannot coexist on multiple protocol-based vlans so this looks like our best bet for a console server -- we can connect eth0 of each server to the console server's management lan and eth1 to the primary lan. The hardware is fairly powerful and we could later daisy chain the dumb switch to this and add six more devices to the management lan. This has a few other useful features e.g. it can act as a tftp server (pxe boot!). The mostly Free Software and semi-customizable userspace are a definite win. Even better is that the fancy remote console application is also Free Software, runs on GNU/Linux (Java),

4.5. Network Switch

If we can get by with a CM4001 we should spend a bit more on a proper smart switch so that we can setup multiple vlans. Initially at least a public vlan and a private IPMI-only protocol based vlan. Later on we may want to experiment with routing database and afs traffic locally on a vlan with jumbo frames enabled (according to a cursory google this would increase database throughput but would likely have little effect on afs until openafs 1.6 is released with the new RxTCP transport layer).

4.5.1. Unmanaged US Robotics 8-port Gigabit Switch

This looks like it will be an acceptable switch until we can afford (or need) a managed switch.

5. Shopping list

5.1. Non-Dell Vendors

From other vendors, systems comparable to the Poweredge 2970 cost...

5.2. Option A

5.3. Option B

5.3.1. Why Two PowerEdge 2970s

Although this setup would use 6U rather than 5U, the PowerEdge 2970 offers a much better price/performance ratio than the 1U R410. For a bit less than a single processor R410 we could have eight cores on both machines (avoiding a difficult/time consuming processor upgrade later on).

Migration2009/HardwareUpgrade (last edited 2009-12-04 18:31:26 by DavorOcelic)