I made a thing

A while ago I made a prototype of a simple, high-performance keyword search for bugzilla, using python and Redis.  It was pretty promising, able to do all/any queries against the summaries of all open Firefox bugs in a handful of milliseconds.  The bugzilla maintainers decided to use the database’s full-text engine instead, but I was nonetheless pretty pleased with it.

A month or two ago, I started rewriting it it atop node.js and gave it a simple websocket-based interface.  (It took a while, due to The Unpleasantness rather than the difficulty of the task itself.) It was blazingly fast, easily able to keep up with real-time matching of entered keywords. The queries on the server side were finished in a dozen milliseconds or less, and the remainder of my 200 millisecond request budget was spent on data transfer back to the client. I had to limit the returned result set to the most recent 100 items to keep the transfer time down. I will probably look at compression in JS to improve things there. (Some requests take as much as half a second if I get unlucky on the network and result-set size.)

Building the index takes about 20 seconds on its wimpy VM, though retrieving all the summaries from bugzilla takes a fair bit longer. (It currently indexes open bugs in Firefox, Fennec, Core, Toolkit, NSPR and NSS.) I could do that on a timer and have it be current within 10 minutes, but it would be pretty wasteful: the rate of relevant change is on the order of a few dozen each minute, and updating that many summaries is literally microseconds.

Enter Pulse, Christian Legnitto’s mozilla-wide message broker.  Wired up to Pulse, my index is up to date cheaply every 5 minutes, and once we deploy the bugzilla extension, the index will be updated within fractions of a second.

To support non-websocket browsers, which is currently all of them, I switched to using socket.io for the transport.  The flashsocket transport adds about 100 milliseconds to the round-trip, which is unfortunate but here we are.

In all, it’s about 500 lines of JS on the server side, plus another ~150 lines for the scripts that do the loading, and I think it’s a great example of what tools like pulse and the Bugzilla REST API are going to make possible in 2011. It’s also an example of how holy-crikey excellent Node and Redis are together.

(There’s an installation running here, which might randomly break as I hack on it.)

Update, 2 minutes after posting:

23 Jan 00:16:00 - searching for sex 23 Jan 00:16:00 - undefined: search 1 -> 1 in 0/1 ms
Hi, Internet.

4 comments to “I made a thing”

  1. entered 22 January 2011 @ 4:28 pm

    That is fast. And pretty useful. Thanks. =)

  2. entered 22 January 2011 @ 4:51 pm

    [...] This post was mentioned on Twitter by Mike Shaver, David Humphrey, Ryan Dahl, Christian Sanz, Christian Legnitto and others. Christian Legnitto said: RT @shaver: I made a thing to search bugzilla. It's pretty quick, because node.js and redis are amazing: http://bit.ly/ifsvkv [...]

    [WORDPRESS HASHCASH] The comment’s server IP (208.74.66.43) doesn’t match the comment’s URL host IP (74.112.128.10) and so is spam.

  3. entered 22 January 2011 @ 11:31 pm

    That is awesome! It would be extra awesome to have this packaged as a Bugzilla Extension so that other installations could use it, too:

    http://www.bugzilla.org/docs/4.0/en/html/api/Bugzilla/Extension.html

    -Max

  4. entered 23 January 2011 @ 2:16 am

    That’s fantastically fast!