return of the king

“Thank god.”:

chain of causality

I’ve learned some more about the events that led to the “impromptu system reinstall”:, and they’re not entirely amusing, or entirely surprising. Let me lay out a scenario for you.

Let us define E and N as two computer systems, not equal to bitchcake ([B]).

E was compromised, and a “trojan ssh”: was installed on their system. Via one or more users shared between E and N, N was eventually compromised. (I suspect, though have no evidence to support, that one of the recent flurry of “privilege”: “escalation”: bugs in the Linux kernel let the intruder up the ante on E, N and eventually B.)

N and B also share at least one (likely precisely one) user, and it’s not at all unlikely that this was the vector through which B (and, transitively, one additional machine) was compromised.

This wouldn’t be all that bad, as These Things Do Happen, and I could have certainly done a better job of keeping B‘s update, well, up-to-date, but it turns out that E‘s administrators knew quite some time before the N {->} B attack that they had this problem, and didn’t bother to tell people. A-frigging-hem. Given that the N {->} B user is conscientious about such things to a fault, and generally the sort of responsible user that every system administrator would like to clone throughout his or her @shadow@ file, it seems not unlikely that we’d have at least discovered the intrusion on B earlier, and quite possibly avoided it in the first place. Alas.

B is pretty sad about the whole thing, apparently, because it just killed another drive in its angst:

@hdc: dma_intr: error=0×40 { UncorrectableError }, LBAsect=120582, high=0, low=120582, sector=120582@

Yay! More drive shopping!

(Further: the “User In Question”: should not be “beating himself up”: about this at all. Stop it right now.)

hey, hey, you, you

‘Tis the season, I suppose. “Matej”: is starting to get back into driving again, and I passed my first-ever in-car driving test on Friday. It wasn’t a “Ministry test”:, so I’m still not really a very well-licensed driver, but the combination of the YD collision-avoidance skills test and the “simulated Ministry test” left me feeling pretty good about the whole thing. Time to schedule my road test, if you can believe that.

(I haven’t mentioned, I suppose, that we have a car now. Mike and Kristen very, very generously donated their pasture-bound Escort — affectionately known as “Kermit” in spite of a slight colour mismatch — for my practice and errand-running pleasure, when they bought their spiffy new Kia. It’s been outrageously helpful to actually have a car around in which I can shuttle Tyla about the city like the princess she is, and also, coincidentally, get in lots of wheel time. Thanks, guys!)

We’re going to drive up to Markham tomorrow to visit the family for Dad’s birthday celebration, and from his enthusiasm on the phone it sounded like he’s at least as excited about the prospect of having me driving as about the actual visit. I’m sure that was just some reception problem with the cordless, though.

utility infielding

I’ve been hard, hard at work on the Lustre Management Tools again for the last few weeks, and it’s really been a pretty interesting experience. More than most of my other software work, these tools really do cover a pretty tremendous range of software domains, and it’s been a lot of fun pulling them all together. (more…)


Ten days can really fly by, if you’re not careful about it. Let’s recap!

like pulling, oh, forget it

For the first time in what is really an irresponsibly long while, I’m off to the dentist to be chided for oral neglect — settle down, Beltzner — and prodded with sharp things. As much as is reasonably possible under the circumstances, I’m actually quite excited about the prospect. I think it’s a tremendous sign of maturity that I did not have a breakfast of “Hot Tamales”: today.

thursday’s child has far to go

So, yeah, birthday.

Got off to an auspicious start, when my driving instructor called to say that he had an excess nail in his sidewall, and would I mind rescheduling. Combined with the phone message from my trainer asking where the hell I was yesterday — did anyone else realize that it was Monday? — my hopes were not high.

I did get some nice well-wishing phone calls from my mother and father, and Madhava. Emily called, not to wish me a happy birthday — though I know she wishes me that, deep down — but to inform me of tickets to tonight’s Boston-at-Toronto game that were dangerously close to affordable. By the time I found someone else to go with, it was too late, which made me feel sort of dumb. I’ve since convinced myself that they were probably spoken for, at their half-off price, long before Em even finished dialing, and I’ll thank you to keep your rebuttals to yourself.

I got slightly more accomplished at work today than I thought I would, which is a very nice change from the usual. Especially, I should add, because I did it by finding a better variant of the problem to solve, rather than just churning through my solution faster.

Tyla’s home soon, I hope. I should tidy a bit before she arrives, because I’m sort of digging this being-married thing. I did do some laundry, at least.

Thanks to everyone who managed to find someone to be nice to today! There was a time when my new year’s resolution every year was to go out of my way at least once a day to help someone, and in 1998 I actually made it until something like June before breaking my streak. It turns out to be relatively easy once you get the hang of it, finding someone with a question on a web forum that you can answer, or walking to the corner with someone seeking directions, to point them to the next landmark.

Hmmm. I wonder if Al and I can find some reasonable scalped tickets…

hippo birdie

For my birthday, I would like

  • 25 hours of my weekend back,
  • everyone who reads this to go out of their way just once today to do something nice for someone they don’t know very well,
  • my wife to come home.

That is all. Thank you for your attention.


I thought I had things fixed when I went to sleep (about this time yesterday, in fact), but there were still some issues to be resolved, such as:

  • reverting from “dovecot”: to @uw-imapd@, which grates on my soul, but which does allow my users to access their email, so here we are. dovecot will be great in the future at some point, but for now it’s really not the right solution for me, especially when I’m on a tight schedule.

  • a bunch of PHP and Apache-config infelicities that broke some users’ apps.

  • a forgotten @spamassassin@ installation, which bounced a bunch of email to “Mike”: It looks sort of like that was all spam, in fact, which would be a nice touch of luck.

  • some unknown problem with @procmail@ that bounced a bunch of Madhava’s mail, in addition to causing a much-less-critical misfiling issue with Phil’s. Confidential to the authors of @procmail@: if you continue to write software for part of a mail-delivery pipeline, please be liberal with your application of @strerror@, so that I have a hope in hell of figuring out why you can’t write to, say, @/var/spool/mail/enros@; many thanks.

  • a classic problem with PINE, which I hate so much I could scream.

  • a billion little tiny permission/group/missing-symlink etc. problems that consumed the rest of the time.

I did get SMTP auth working, though, against all odds, so there’s one bright spot.

Now I just have to figure out how to make up the 20 hours of work-time I missed this weekend so I don’t get fired.

I’m so tired.

callin’ all’a y’all

This weekend was already set up to suck a bit.

  • Tyla is away on Valentine’s Day — about which I am not at all angry, I hasten to add, but it does make the house feel a little empty. Turnabout is fair play, I suppose.

  • There’s a major test shot going on at work this weekend in preparation for the release of the best software to ever bear the Lustre name, so I was going to be doing a lot of test watching and wrangling, probably minimal partying.

  • I’m still behind on some work from this week, which I was hoping to cram into the gaps in testing.

But then I found out that some people had got where they shouldn’t have got, and I had to spend 12 hours buying new drives, installing a fresh OS, copying user data and whatnot over after inspecting it for damage, and generally cursing the world.

At least I know what our near-worst-case scenario is for disaster recovery. And we do have more disk space now…

I need a huge frigging drink now.

next page »