I made a thing

A while ago I made a prototype of a simple, high-performance keyword search for bugzilla, using python and Redis.  It was pretty promising, able to do all/any queries against the summaries of all open Firefox bugs in a handful of milliseconds.  The bugzilla maintainers decided to use the database’s full-text engine instead, but I was nonetheless pretty pleased with it.

A month or two ago, I started rewriting it it atop node.js and gave it a simple websocket-based interface.  (It took a while, due to The Unpleasantness rather than the difficulty of the task itself.) It was blazingly fast, easily able to keep up with real-time matching of entered keywords. The queries on the server side were finished in a dozen milliseconds or less, and the remainder of my 200 millisecond request budget was spent on data transfer back to the client. I had to limit the returned result set to the most recent 100 items to keep the transfer time down. I will probably look at compression in JS to improve things there. (Some requests take as much as half a second if I get unlucky on the network and result-set size.)

Building the index takes about 20 seconds on its wimpy VM, though retrieving all the summaries from bugzilla takes a fair bit longer. (It currently indexes open bugs in Firefox, Fennec, Core, Toolkit, NSPR and NSS.) I could do that on a timer and have it be current within 10 minutes, but it would be pretty wasteful: the rate of relevant change is on the order of a few dozen each minute, and updating that many summaries is literally microseconds.

Enter Pulse, Christian Legnitto’s mozilla-wide message broker.  Wired up to Pulse, my index is up to date cheaply every 5 minutes, and once we deploy the bugzilla extension, the index will be updated within fractions of a second.

To support non-websocket browsers, which is currently all of them, I switched to using socket.io for the transport.  The flashsocket transport adds about 100 milliseconds to the round-trip, which is unfortunate but here we are.

In all, it’s about 500 lines of JS on the server side, plus another ~150 lines for the scripts that do the loading, and I think it’s a great example of what tools like pulse and the Bugzilla REST API are going to make possible in 2011. It’s also an example of how holy-crikey excellent Node and Redis are together.

(There’s an installation running here, which might randomly break as I hack on it.)

Update, 2 minutes after posting:

23 Jan 00:16:00 - searching for sex 23 Jan 00:16:00 - undefined: search 1 -> 1 in 0/1 ms
Hi, Internet.

oh snapdragon

I’m late to this particular announcement, as I didn’t really have useful internet during the critical period last week, but that hasn’t dampened my enthusiasm one bit.

The Open Komodo/Snapdragon announcement is very exciting for me, and I suspect for many others in the Mozilla community. Having a basic, extensible IDE built on the same technology as Firefox (and therefore the web) will provide a great focal point for many of the IDE efforts that have been mooted around Mozilla technology over the years. People will be able to focus their energies on the specific tasks they wish to enable or improve, taking advantage of the incredible base that ActiveState has built.

I was lucky enough to be involved in the discussions and planning leading up to this announcement, and I think the most exciting thing about it wasn’t the prospect of the source release, though I am indeed eager to get that technology into the hands of the Mozilla developer community. The most exciting thing for me was seeing the depth of ActiveState’s commitment to the open web, and how enthused they are about helping to create a new open source community. This is a huge, huge investment in the open web by a small, small company, and I can’t wait to see where it’s going to take us — where we’re all going to take each other, in fact.

So thank you to Bart and Shane and David and Erin and all the others at ActiveState who had the vision and courage to make Open Komodo a reality, and to everyone who has already expressed their interest and support for the project. It’s going to be a blast.

the high cost of some free tools

(This is going to be a little long, for which I guess I could apologize, but it’s my blog, so whatever.)

So let’s try this one on for size, because everyone is offering the web developer a set of tools to lure them off the web. You’ve got your silly season participants, and you’ve got your — no, honestly, grown-ups came up with this name — JavaFX Script stuff, and they all want you to give up this archaic web thing for something so much shinier, and faster, and man have you seen the cooking-show demos?

You can point and you can click, and you will get an application, and it will run on the web (says Silverlight, because Microsoft has always been about the web: they were just getting up a good head of steam when they left IE6 dangling for years; they were always planning to come back) and on the desktop (says Apollo, because Adobe is best known for its “web” stuff in this space, but when Silverlight comes to the web, I mean, what would you do?). And it will be glorious. You will have graphics and drag-and-drop widgets and you can bet there will be pretty colours and probably a billion language choices, and if it doesn’t generate the cutest little installer then you can have your money back.

Adobe and Microsoft have always had better tools, in part because they’ve staffed and organized hard against them, but also in large part because their platforms require tools — that’s a big part of their business model. I don’t think you should need to buy, or even use-for-free, any given tool to build the web, and by using and helping to drive open web technologies Mozilla lets people choose the tools they want to use. We don’t force you to use the ones that we make money or marketshare on: you can use Eclipse or Firebug, Rails or J2EE, Komodo or Notepad, YUI or Dojo. If you ask the people who are building the exciting and significant apps on the web today, be they gmail or eBay or twitter or facebook or flickr or even Windows Live Search, I bet you that vanishingly few of them use IDEs. They use the tool that works best for them, for a given problem, sometimes using a bunch at once. I would be very surprised to discover that all of a given team even uses the same text editor, to be frank.

(We also don’t make you sign licensing agreements to get the format specifications, or prevent you from competing with us. We don’t tell you where you can and can’t install the software. We don’t tell you what you can and can’t tell people about your experiences, or that you can’t give it to other people with whom you might want to collaborate.)

If you choose a platform that needs tools, if you give up the viral soft collaboration of View Source and copy-and-paste mashups and being able to jam jQuery in the hole that used to have Prototype in it, you lose what gave the web its distributed evolution and incrementalism. You lose what made the web great, and what made the web win. If someone tells you that their platform is the web, only better, there is a very easy test that you can use:

Is this the web?

When the tool spits out some bundle of shining Deployment-Ready Code Artifact, do you get something that can be mashed up, styled, scripted, indexed by search engines, read aloud by screen readers, read by humans, customized with greasemonkey, reformatted for mobile devices, machine-translated, excerpted, transcluded, edited live with tools like Firebug? Or do you get a chunk of dead code with some scripted frills about the edges, frozen in time and space, until you need to update it later and have to figure out how to get the same tool setup you had before, and hope that the platform is still getting security and feature updates? (I’m talking to you, pre-VB.NET Visual Basic developers.)

Mozilla has always valued and supported web developers, and in turn those who support developers with tools and other assets, and we’ll invest more in this area over the coming year. But we’ll do it in a way that makes sense for the whole web, and brings to bear the human-manipulable power of web technology: a great set of primitives that people combine in very different ways, giving developers a great opportunity to choose tools and toolkits and patterns and technology that suit how they want to work and what they want to build.

The web can eat toolchain bait like this for breakfast. And, if Mozilla has anything to say about it, it will do just that. You won’t have to give up the web to work offline any more, or programmable 2D graphics, etc. Soon you’ll have the power of 3D and great desktop/application integration as well, via projects like canvas3d and registration of content handlers, and you’ll have it in a way that’s built on open specifications and a tool ecosystem that isn’t a monoculture. Why wouldn’t you choose the web, given its record and power and openness?