Yesterday, the provider that hosts one of my personal machines fell off the network for about 12 hours, and topped it with a power outage from which my machine didn’t come back up on its own. (Never heard why, hardware was fine when it did come up; suspect some mismanagement in the colo space.) The network that came back after the outage was materially slower than before, making it increasingly hard to actually migrate the remaining services off of that machine, onto the new one, which now had replacement fans.
I left, as you might imagine, quite a few voice mails on the tech support box, mails to the support@ address, and even a voice mail on the cell phone of the president of the hosting provider (he had insisted previously that I should call him if there were ever a problem). When I finally heard back from someone at the hosting provider, I was told that I could move my machine to their cabinet at 151 Front for improved reliability. They were doing this for me as a favour!
“No, that’s fine, I’ll take care of it.” Well, actually…they were doing this to reduce the load on the now-weaker uplink from the colo space.
“I won’t be running out of there for long, and I can keep the bandwidth down just fine for a few days, but thanks.” Well, actually…they were doing this because they were shutting that colo space down because their remaining colo customers didn’t need all that space.
“Cool, I’ll be done in a week or so and then I can remove the box entirely, don’t worry about it.” Well, actually…they need to be out of that space in 48 hours or they’ll have to pay rent, or something. I was unable to avoid mentioning that I’d been told I would need to give 60 days notice to cancel my own hosting, a few weeks earlier; I was weak in that regard.
So a wonderful friend of mine moved the old machine to some rack space he has, and then he and other wonderful friends worked all night to get the routing unscrewed. In the process, one of my new machines had to be rebooted, and my friends’ respect for my sleep trumped the possibility of calling me to help fix things when the services on that machine didn’t talk to the network post-reboot. Once I got up this morning, I spent about 30 mins figuring out what state the various bridges were in and then Google led me to victory in about another 30.
For a period of a few hours, I had no DNS service, as the machine I was using as a secondary DNS server was experiencing related but not identical failure; a failure of redundancy that I will soon remedy. That led to a bit of mail bouncing, if it was sent to email@example.com or firstname.lastname@example.org, so please do resend if you were thusly victimized. My blog was also down for a bit, not that anyone is likely to care about that.
I’m not sure if it’s irony or demand creation, but the same company that I have now put behind me offers a high-availability network access service — clearly, I should have availed myself of it.