time to go raise goats on a mountain

Yesterday, the provider that hosts one of my personal machines fell off the network for about 12 hours, and topped it with a power outage from which my machine didn’t come back up on its own. (Never heard why, hardware was fine when it did come up; suspect some mismanagement in the colo space.) The network that came back after the outage was materially slower than before, making it increasingly hard to actually migrate the remaining services off of that machine, onto the new one, which now had replacement fans.

I left, as you might imagine, quite a few voice mails on the tech support box, mails to the support@ address, and even a voice mail on the cell phone of the president of the hosting provider (he had insisted previously that I should call him if there were ever a problem). When I finally heard back from someone at the hosting provider, I was told that I could move my machine to their cabinet at 151 Front for improved reliability. They were doing this for me as a favour!

“No, that’s fine, I’ll take care of it.” Well, actually…they were doing this to reduce the load on the now-weaker uplink from the colo space.

“I won’t be running out of there for long, and I can keep the bandwidth down just fine for a few days, but thanks.” Well, actually…they were doing this because they were shutting that colo space down because their remaining colo customers didn’t need all that space.

“Cool, I’ll be done in a week or so and then I can remove the box entirely, don’t worry about it.” Well, actually…they need to be out of that space in 48 hours or they’ll have to pay rent, or something. I was unable to avoid mentioning that I’d been told I would need to give 60 days notice to cancel my own hosting, a few weeks earlier; I was weak in that regard.

So a wonderful friend of mine moved the old machine to some rack space he has, and then he and other wonderful friends worked all night to get the routing unscrewed. In the process, one of my new machines had to be rebooted, and my friends’ respect for my sleep trumped the possibility of calling me to help fix things when the services on that machine didn’t talk to the network post-reboot. Once I got up this morning, I spent about 30 mins figuring out what state the various bridges were in and then Google led me to victory in about another 30.

For a period of a few hours, I had no DNS service, as the machine I was using as a secondary DNS server was experiencing related but not identical failure; a failure of redundancy that I will soon remedy. That led to a bit of mail bouncing, if it was sent to shaver@off.net or shaver@mozilla.org, so please do resend if you were thusly victimized. My blog was also down for a bit, not that anyone is likely to care about that.

I’m not sure if it’s irony or demand creation, but the same company that I have now put behind me offers a high-availability network access service — clearly, I should have availed myself of it.

with vacations like this, who needs work?

I took a day off today to get some stuff done around the house, and to put some new drives in the server that hosts this blog, among other wonders.

Once we got into the server hosting space, which was mildly exciting, things went downhill in a hurry:

I had 4 drives in my backpack, and 4 drives in the machine, but only capacity for 6 drives of this size (3.5″). I did not have the right kind of screwdriver to manipulate the front panel of the machine — and to fit the 5th and 6th drives, even, I would have to remove the CD-ROM and media-bay stuff. And get more sleds.

While I was removing the CD-ROM, and discovering that the chassis probably wanted an adapter of sorts to enjoy these new drives at all, I managed to stick my left pinky a little into the whirling maw of pain that is a chassis fan. That little mother was really spinning, because the impact sent a blade flying off the fan, and sent me to St. Mike’s to get a tetanus shot and have a doctor inspect my now-oddly-hamburger-like pinkytip.

So now I need:

  • 2 drive sleds,
  • possibly, some conversion kit to go from ’4 x 3.5″ drives plus media bay’ to ’6 x 3.5″ drives without media bay” on a Dell Poweredge 2950,
  • two new chassis fan modules,
  • someone to work the shift, control and command keys on my computer for a little while,
  • a picture of the blood spatter on the sticker that says “don’t put your finger in here when the power’s on, moron”.

And I’m totally going to need another day off at some point soon, after this mess.

(Oh, I also lost my book somewhere today, possibly at lunch, so I had to buy another copy.)