Of rats' nests and fans

[altivo]

I decided to activate the new server by moving devices around in virtual space rather than by moving patch cords around in real space. In order to do that, though, I needed a map of the preset state of our switches and patch panel. No, there never was such a map. I've been lamenting that for years, but the previous network management team never made one even though they were being paid an outrageous amount (more than I make, on an hourly basis, by far.)

Doing such a chart, which is really just a spreadsheet affair, requires standing at the patch panels and tracing individual cables through the rat's nest to find out which panel jack is connected to each port on each switch. We have only about 150 ports, but the patch cables are routed very "imaginatively" and about 95% of them are the same color. None are tagged in any way. This took a couple of hours.

I happened to glance down at the server itself, since it's now in the same communications rack, and saw a flashing amber light. This didn't seem right. On the back there was a bright red flashing light. No label on either, but I went to look up the documentation on Dell's web site. There's a laundry list of possible causes, but a fan failure is right at the top. OK, I was warned that fans tend to conk out (it has four of them) and even received a bag o' fans to be used as replacements. I've not worked with rack-mounted servers of this sort before, and I have to say it's pretty slick. No interruption in service, no need to kill the power. The encapsulated fans can be replaced while the server is running. Just flip open the top, find the fan with a flashing red light next to its power connector (the others are all steady green,) unplug it and slide it out. Slide in replacement, plug it onto the power posts, the flashing red light turns green, and you can button up the case. Not even a screwdriver is needed. Worked like a charm.

Comments

[corelog.livejournal.com]

I do find the Dell servers to be pretty slick. I haven't yet found the courage, though, to attempt failing one of my hard drives and testing the RAID. I'm sure it'll work fine when I do test it, but I'm more concerned with it syncing back up successfully.

[altivo.livejournal.com]

Oh, RAID works fine. A couple of years ago the primary server that supported the library catalogs and circulation system developed a bad drive. It was part of a RAID 5 array, and the server just kept running with a drive out of commission. Nothing was visible to users, while behind the scenes there was a scramble to get a new drive shipped in. When the new drive was snapped into place, the server still kept running, but there was a noticeable slowdown for a couple of hours while the array rebuilt its redundancies.

[duskwuff.livejournal.com]

Good, high-quality server hardware is truly a pleasure to work with. While some of their consumer machines leave something to be desired, Dell's higher-end kit (business desktops and servers) is rather nice. :)

[altivo.livejournal.com]

Yep. I particularly like their clever case designs and the ease of access and replacement in most of those, with no tools required in many cases even to replace drives or power supplies.

[duskwuff.livejournal.com]

That's exactly it. Even their lower-end business desktops use "rails" on the hard disks, which makes changing disks a lot easier than it'd otherwise be.

Thats almost too hard to believe!

[bladehorse.livejournal.com]

Its so rare anymore to see engineering that actually thinks of the future, and puts in conveniances like that. Very cool:) Especially after the new fan install:p

Re: Thats almost too hard to believe!

[altivo.livejournal.com]

This unit is designed with redundant power supplies too. If a supply goes bad, you can actually replace it without ever shutting down the power. Makes sense for a big datacenter that provides services 24/7, but it's a level of service I haven't had to handle since the 80s when I worked on IBM mainframe equipment.

[megadog.livejournal.com]

IBM have been doing similar things for years: I suspect they subcontract some aspects of their design to LEGO.

IBM also have *THE BEST* rack-mounting kits in the business - you can rack up a 1U IBM server in about 30 seconds. Sun, H-P, Dell gear takes orders-of-magnitude longer to rack up and often draws blood in the process.

Thought: perhaps Dell should spend money on better quality more-reliable fans rather than making the cheap-and-nasty ones easier to swap?

[altivo.livejournal.com]

Well, this server is very used. It's been in service continuously for six or seven years before it was given to me. A datacenter that uses such equipment as a matter of course would have retired it some time back, and in fact that's why I have it. The spare fans were tossed in by the donor because, as he said, it's long past its expected lifetime and they're the most likely thing to fail.

IBM does build stuff to last, at least sometimes. They also suffer from keyhole vision at times. The last IBM server I bought was purchased with Linux rather than Windoze, since I much prefer Linux on network servers. One would think that they'd supply hardware suitable for full Linux operation if they were offering Linux as an out of the box option, but no. It had a Matrox video card for which there was no Linux driver (because at the time Matrox was still trying to keep its stuff "secret" and the Linux folks hadn't yet managed to reverse engineer it.)

[megadog.livejournal.com]

I'm having deep cognitive dissonance with "server" and "video card".

The low-end servers I deal with have the console on either a 9600-baud serial-port or a dedicated Ethernet management-port [See Sun's ALOM]

Mid-range and top-end servers from the likes of IBM have daisy-chaining capabilities on their 'console' ports: I can manage four 43U-racks of the things from a single physical console. If I need to.

Truth is, once I've installed a rackful of servers it generally doesn't see daylight again for at least a year.

[altivo.livejournal.com]

This was not a rack mount unit, it was a tower. The time frame is roughly ten years ago. Tower servers even today are often configured with video cards, and especially those sold with Windoze server software pre-installed. As far as that goes, this Dell PowerEdge 1U rackmount has a graphics chipset and SVGA connector on the motherboard. Obviously not needed but these days it adds little expense.

The thing was that IBM preconfigured the Linux on the HD to expect a graphical console, but put it on a box in which the graphics device was not compatible with the software. The solution was to log in through a serial port, reconfigure to disable the graphical console and make it text based, then reboot again. This was not the solution IBM prescribed, however. Theirs was to install a second graphics card in the tower, one that would be recognized by their Linux configuration, use that as a console to disable the graphical interface, then remove the extra card.

[schnee.livejournal.com]

*smiles* I just can't help but chuckle every time I read "bag o' fans". ^^

[altivo.livejournal.com]

Just like a barrel o' monkeys. It's a construct I acquired from my brother, no idea where he caught it.

[schnee.livejournal.com]

*noddles* It actually reminded me of a Sam & Max story - I'd have to look up the exact title again, but it's one where Sam and Max are called to a supermarket to exorcise a demon that haunts the cereal aisle. They check out all sorts of products, too, of course, and at one point, Sam can be seen holding a giant "Bag O' Ham", stating "I want this. I have no idea what it is, but it feels neat".

Anyhow, outside of that, it's not so much the "o'" that I find funny but rather the concept of a bag of fans, or the associated sort of going into a (computer) hardware store and telling the clerk that you'd like two pounds of fresh fans, please... ^^

[cozycabbage.livejournal.com]

And a 2L carton of cabling with that, sir?

[soanos.livejournal.com]

I'd like to have three deciliters of RAM. :)

[soanos.livejournal.com]

Ooooh... Now look what you did. you made me wish I had a server too :D

Glad to hear thigs are going smoothly. ;)

[altivo.livejournal.com]

Went "live" with it this morning. I had a little issue at first with two patch cables that were swapped, but it's running nicely now.

[soanos.livejournal.com]

That is what I meant. ONLY 2 patch cable and a fan :)

[cabcat.livejournal.com]

I love things like that, I just wish I could power up my E-sata without turning the machine off.