time to go raise goats on a mountain

Yesterday, the provider that hosts one of my personal machines fell off the network for about 12 hours, and topped it with a power outage from which my machine didn’t come back up on its own. (Never heard why, hardware was fine when it did come up; suspect some mismanagement in the colo space.) The network that came back after the outage was materially slower than before, making it increasingly hard to actually migrate the remaining services off of that machine, onto the new one, which now had replacement fans.

I left, as you might imagine, quite a few voice mails on the tech support box, mails to the support@ address, and even a voice mail on the cell phone of the president of the hosting provider (he had insisted previously that I should call him if there were ever a problem). When I finally heard back from someone at the hosting provider, I was told that I could move my machine to their cabinet at 151 Front for improved reliability. They were doing this for me as a favour!

“No, that’s fine, I’ll take care of it.” Well, actually…they were doing this to reduce the load on the now-weaker uplink from the colo space.

“I won’t be running out of there for long, and I can keep the bandwidth down just fine for a few days, but thanks.” Well, actually…they were doing this because they were shutting that colo space down because their remaining colo customers didn’t need all that space.

“Cool, I’ll be done in a week or so and then I can remove the box entirely, don’t worry about it.” Well, actually…they need to be out of that space in 48 hours or they’ll have to pay rent, or something. I was unable to avoid mentioning that I’d been told I would need to give 60 days notice to cancel my own hosting, a few weeks earlier; I was weak in that regard.

So a wonderful friend of mine moved the old machine to some rack space he has, and then he and other wonderful friends worked all night to get the routing unscrewed. In the process, one of my new machines had to be rebooted, and my friends’ respect for my sleep trumped the possibility of calling me to help fix things when the services on that machine didn’t talk to the network post-reboot. Once I got up this morning, I spent about 30 mins figuring out what state the various bridges were in and then Google led me to victory in about another 30.

For a period of a few hours, I had no DNS service, as the machine I was using as a secondary DNS server was experiencing related but not identical failure; a failure of redundancy that I will soon remedy. That led to a bit of mail bouncing, if it was sent to shaver@off.net or shaver@mozilla.org, so please do resend if you were thusly victimized. My blog was also down for a bit, not that anyone is likely to care about that.

I’m not sure if it’s irony or demand creation, but the same company that I have now put behind me offers a high-availability network access service — clearly, I should have availed myself of it.

one comment to “time to go raise goats on a mountain”

  1. entered 5 December 2007 @ 4:35 pm

    If DNS did not resolve I imagine the emails bounced, which may not be nice but at least notified senders appropriately, so chances are any important mail will be resent. (At least so I’d hope).

  2. entered 5 December 2007 @ 4:53 pm

    Yeah, but if those people are like me they get a few hundred “returned to sender” spam backscatter messages a week, so it might have been missed. :(

  3. janice
    entered 6 December 2007 @ 1:56 am

    “My blog was also down for a bit, not that anyone is likely to care about that.”

    Your Mom called. She said: “Do too!” [care about your blog being down].

    She’s also hoping that does not make her a ‘no one’. She’s perversely logical and ‘not anyone’ == ‘no one’ in her very different world.

    Oh, she called back… (such a pest!) “OMG. Did you get my ‘thank you’ email? I’ll resend it. Just in case.”

    [ Ed: Good thing your Mom reads your blog or she wouldn't know that you may not have gotten her message. She's leaving now, muttering something about responds or maybe it was response times? Something like that anyway. And not being able distinguish between red and not red without them. Can't make out much else. Might be time for her nap. ]

    Wait. She’s back. She says: “Almost forgot. Love you! [kiss kiss kiss]“

  4. entered 6 December 2007 @ 11:19 am

    [...] I had the same experience that Mike had with the colo that hosts my machine, since they were in the same room. Our mutual [...]

  5. entered 10 December 2007 @ 12:50 am

    Thanks for the all the heroics, Shaver! I happened to be in an intensive period of scheduling meetings with students when it happened, so I almost completely spazzed out after the first few hours, knowing I was missing many meetings. But then it came back up; what joy!

    I can’t quite figure out from your post where the thing is now hosted. I hope somewhere with either a business plan or a physical access panel…