inevitable failure

I wrestled with my test suite today, and I think it came out to a draw. Didn’t find any new core-recovery bugs, and spent a long time tuning the infrastructure, but I did fix a lot of cleanup bugs that have been bugging people for a while. Not bugging me so much, which is why they didn’t get fixed before, but still.

Madhava and Tyla and ALASDAIR and I are going to get some food and then watch a bad movie. The Recruit might do well in that role, if Tyla can be convinced.

I was thinking about Chinese New Year celebrations as an alternative to moviegoing, but now I’m not so sure I’d enjoy that. Snakes are a little iffy for me at the best of times, you see.

January has just about ended, and I’m about to acheive total failure with respect to my New Year’s resolutions. Maybe I’ll do better next month. (Other than the exercise one, all I really hoped to do in January was sign up. How hard is that? Sigh.)

Forgot to publish when we got back from the movies (Catch Me If You Can, because of Tyla’s influence and timing) and dinner (Sottovoce, pretty good). There was a time when LiveJournal would sort of screw itself up if I added to a day’s entry after it had grabbed one copy, so I would try to just publish once for the day. I missed it when Jacob pointed out that they fixed that bug, but now I don’t have to worry. So I’ll just publish all the time, and fill the world with timely joy.

osgoode that ends good

This morning, Tyla got up early, showered, answered her email, ate a healthy (if small) breakfast and walked to U of T, where she participated in an Aquafit class. Meanwhile, I sat in my boxers on the couch, typing on my laptop, watching Muchmusic, and eating Fuzzy Peaches. I swear, sometimes it feels like we’re the same person. When is she going to start living her own life?

Work went well: my test suite is starting to bear fruit, in the form of bugs that I’m finding and fixing before the customer gets a chance. I fixed about ten bugs today, half of which were problems or omissions in the test infrastructure, and the other half of which were things in the recovery space that Just Weren’t Right. And then I got to explain to my good buddy Phil some of the way recovery works — how we deal with replaying opens and closes correctly, to be more specific than any of you really want. Slowly, I’m working off the huge clue-debt I accumulate every time I look into the lock manager and then make Phil’s phone ring.

Coop isn’t coming to visit on the weekend proper, but it sounds like I’ll get to enjoy his company during the week instead. He’ll get to join us for Buffy night, and we can even get some work done together. Much fun.

This evening, Madhava invited Tyla and me along to a production of Un ballo in maschera at the Hummingbird Centre. It was a lot of fun, and I’m now three for three when it comes to enjoying opera performances. I should go more often — especially if Madhava can keep scoring these sweet $15 tickets.

Bedtime. Tomorrow I will spout productivity like a geyser and get the recovery test-train rolling at full steam. Also, I guess I should go and buy some more metaphors. Cripes.

29 January 2003

This was one of those days which, placed in the appropriate other week, would have either been really pretty good, or quite bad. I got most of the driver for the test suite I mentioned yesterday working, until I realized that I’d forgotten a critical piece. On the one hand, it meant that there weren’t — well, weren’t necessarily — a pile of new and terrifying bugs in my recovery processing. On the other hand, it meant that I wasn’t done writing the test driver, which I had really hoped to finish today. Looking at it now, it doesn’t look like a huge pile of work, but you have to understand that I wasted lots of time trying to make it perfectly generic before I realized that we had other people who are a lot better at that than I am. Now it’s just a straight-up monolithic test set, and anyone who wants to add a new test can damned well copy and paste a little. It builds character any way. I was trying to be methodical and thorough about this as a counter to the urge I was feeling to just tear through it and get onto the fixing of recovery bugs. I guess the lesson is to give into my gut and let the panic rule me. Or something. (Confidential to Mike Who Has Made It Very Hard To Link To Specific Entries: I don’t know if this is one of those “Real Real Programmer” things. Maybe the real programmers I work with can tell us.)

In fiscal year 2002, AOL lost nearly $100B. I, to the best of my knowledge, did not — but I should probably send a pile of paper off to an accountant and make absolutely certain. If I managed to lose many thousands of times more money than I made, Tyla will never let me have a baby. Or, you know, will never let me let her have a baby. Or something. (Look, hon: cute baby!).

Coop might — just might —come to visit me in Toronto this weekend. Food will be purchased and prepared. Drinks will be drunk. And boy, do we have our bad movie work cut right out for us.

I coughed maybe twenty times today, and didn’t wake up at all last night from coughing. Might be able to declare victory tomorrow, especially if I get off my ass and go outside.

never test for an error you can’t handle

I work on recovery. This means that I’m responsible for getting the cluster — or, at least the filesystem parts — back into mostly-serviceable shape automatically, if (OK, when) something goes wrong. Could be a network failure, or one of the server or client computers crashing. Maybe a rack loses power, or we hit a fatal bug in our server software and it autoreboots. The combinatorial explosion of failure modes and their effects is really pretty impressive, and makes for some challenging analysis problems. (“OK, but what if the file was created on the other client, and then we have to replay the open on this client?” “Wait a sec, that transaction could make it to disk before we send the reply to the dead client.”)

Today, I embarked in earnest on a test suite to let us test the various combinations of, well, recovery things that we care about. (This is, for the record, the same test suite I told Coop yesterday to not write. After I told him to write it. Tee hee.) I’m very excited, because the quality of tests tends to have a significant correlation with the quality of the code in question, and I think it’s pretty important that recovery, our “software safety net” be robust.

And just in time! Phil and Peter have each put both fists — that’s four flying fists of hacker fury, in total — through the lock management and metadata parts of Lustre, and it’s going to break some parts of recovery like a drunken promise. I don’t mind, though, because it’s improved our stability so much.

When I wasn’t fretting over recovery today — I seem to do that a lot since I started this job, don’t I? — I was reading some pretty entertaining stuff on the interweb. Colby Cosh is a funny guy, and this bit from an article about the Davos conference registered as Officially Funny over here in this armchair that I’m calling an office this week:

For some, life begins at conception, for others at birth. “According to Jewish law,” deadpanned Yossi Vardi, an Israeli software entrepreneur, “life begins when the fetus becomes a lawyer.”

My main man Jacob has acquired hockey tickets for my next trip to Boston. It’s all starting to come together.

sweet, sweet routine

I was a little bit concerned about getting back into the rhythm of work after four days away — my god, four whole days! — so I decided that I’d make myself feel a little less outclassed by sending Coop off on a wild goose chase for a few hours. Sorry about that, bud. (Also, I can’t believe you didn’t like Office Linebacker ad. “Hi, Janice!”)

I still have a nagging chest cough thing, but it’s getting better. It might be my annual bronchitis — no joke, it happens every January — in which case I will end up living la vida penicillin.

I don’t have a lot of trouble believing that this site exists, though at one time I would have spent a lot of time fretting over such an obvious sign of the apocalypse, but I do have trouble understanding why Boing Boing linked to it. You feeling OK, Cory? Are there a lot of people who are wardriving or otherwise flitting about from network to network, but don’t know how to find out their own IP address? If so, is that a huge victory for wireless network usability, or a savage condemnation of the usability of the rest of the network software floating around? I’m sure mpt can tell me.

Four days left in January, and I have already pretty much blown one resolution — 30 minutes of exercise every day — for no good reason. Still have a chance to get some of the other ones underway before I fly past my self-imposed “during January” window of victory.

I’m tired, and I need to shake this cold, so I’m going to go off to bed and read about the discovery of plutonium. February 1 will usher in a whole new month, I predict. And maybe some exercise!

a senior moment

OK, just to be clear, I have no idea why I haven’t been publishing the entries for the past few days. I was reminded on Friday that I hadn’t published Thursday’s, but it really seemed to not stick. Given that “publishing” involved typing one 20-character command, it’s a little disappointing. For me, anyway.

“Mystery” solved, as though you care: I was issuing said command in the wrong directory, thereby republishing December’s last entry over and over. I’m going to be unstoppable at work tomorrow, I can tell.

Tyla and I went grocery shopping. Then we ate sandwiches. Now we’re going to watch some Super Bowl and then probably I’ll play more Shadowbane. It’s so fun!

gentle reintroduction

Played more Shadowbane this afternoon. There’s a big story-arc event going on tonight, with seiging and feature characters and all manner of fun stuff, but I’ll have to miss it: Tyla and I are joining Aven, Mark, Madhava, Hilary, and Rob for a spot of celebration in observance of Rabbie Burns Day. Should be good fun, so off we go!

That was quite fun, though the conspicuous lack of pipers made it seem — other than the haggis, of course — like it was just another evening at the pub with some friends. Which is a quite wonderful way to spend one’s time, but I guess I was hoping for something a little more…experiential.

And, because we’re both morons, Tyla and I are coddling our sick selves — Tyla has a cold-like thing that I am still not taking responsibilty for — by staying up until 5AM playing on various computers. Alas.


As I had hoped, but didn’t dare to predict, I felt a lot better this morning. Probably 80%. There was a time, back in the silly days of my youth, when I would have jumped back into work with both feet, knocking myself out to make up lost time. But I’m older and wiser these days, or at least less hardy, so I decided to take today as a day of rest and additional recuperation. Nobody at work seemed too upset; they’re probably just glad that I’m not breaking anything.

I played some Shadowbane — which is shaping up into quite a fun little diversion, I must say — napped and read a lot, didn’t go out anywhere, all responsible I’ll-get-better-or-die-trying behaviour. Yay for me.

It would have been fun to zip to Ottawa for the weekend and watch the superbowl with Coop and Kev and James, and maybe go to a Sens game with Alasdair, but that would also have been irresponsible.

Time for bed, nice and early. Tomorrow, I get to go out and see people!

home stretch

I had trouble propping myself up in bed last night, such that my lungs would work as designed, so I ended up retiring to the futon with Chester instead, where I fell asleep sitting up for a fair while. When I awoke, I still felt like I’d been beaten with a Buick, so I fished around for my doctor’s phone number and made an appointment for 6pm.

In the interim, I talked to Phil a little bit — he’s a wonderful guy, and goes so far out of his way to make me feel good about my mildly-broken work that it’s quite unbelievable — and then had a nap. When I woke at 5pm, ready to call a cab to hit the doctor’s, I noticed something strange. Something different. What’s this? I can breathe. I’m not coughing every ten seconds. Whoa, whoa, whoa, my nostils are useful.

Well, holy crap. I called the doctor back to reschedule my appointment to tomorrow — if I’m feeling even better then, then I don’t need to waste her time, though I really should get my prescription refilled soon; hmmm… — and she thanked me for not coming in if I was on the mend. It’s the little things, really.

Of course, I’m not yet “hey, anyone want to build a house?” better, or even, as we discovered today, “maybe I should check some code into the source tree right before our next milestone release!” better, but the derivative is positive, and I’m in a much better mood. I can take a bath without the steam sending me into body-wracking fits of coughing!

In fact, I think I’ll do that right now.

(Confidential to Asa: it seems I don’t have Mitchell’s bacterial thing — we compared symptoms on the phone today — but you’re a sweetie nonetheless. Thanks a lot.)

no mercy, it seems

We all die a little every day. I know this, and largely accept it. This week, though, I really seem to have made significant progress. No noticeable improvement this morning, and Tyla thinks I sound worse. If she were a doctor, or even really awake, that would probably worry me a little.

I wrote the rest of that recovery test plan today, and I got in a lot of the old “praying for the sweet release of death”, but that’s about it. The temperature swings have calmed down somewhat, but the chest congestion has picked up the slack and the caulking-of-the-sinuses seems to be here to stay. Rumour has it that my good friend Mitchell, with whom I spent far too little time on my Californian trip, has something very similar, and it’s a bacterial condition that requires treatment. So, yeah, it’s doctor time. (You win, mom.)

I pity the fools that shared AC754 with me.

I’ve been telling Tyla that I love her a lot today, because I want those to be my dying words.

neither snow, nor rain, nor heat, nor gloom of mucus

My battle against the head cold continues largely unabated. It got to the point today that the acid from clementines — sweet, healing citrus though they may be — caused agonizing burning in my throat. And I could really go for some sort of high-pressure sinus attachment for the fire hydrant outside. Dayquil helps, as we always knew it would, but it’s still not enough. Worst of all, I’m now too weak to pick Tyla up with one arm, which removes a key arrow from my domestic-disagreement quiver.

In spite of being congested and headachy and weak and hot-no-cold-no-hot all day, and in spite of being hopped up on pseudofreakingephedrine hydrofriggingchloride — or, more likely, because of said up-hopping — I managed to get a fair bit of work done today. Not much progress on the recovery test plan, because coherent prose was beyond my feeble ken, but I did fix a handful of bugs and analyze a few more. That’s right, my American friends, your nuclear stockpile is going to be protected by code written by me, while I’m high as a kite on poor man’s meth. Sleep tight!

I didn’t eat lunch today, but I had some yummy Japanese food while watching a decent Buffy — no, really, in Season Seven! I figure it’s a wash.

Phil and Peter are in New York right now. I was sort of idly wishing I could join them, since I always learn a lot while they’re hacking hard, but now I’m so very glad that my wishes don’t generally come true. Boy, would I hate to be travelling right now.

I downloaded a copy of Down and Out in the Magic Kingdom today, but I don’t think I’ll be able to read it until I get finished with my current epic. And then I’ll probably go and buy a paper copy anyway, just to lend him support.

next page »