dealing with the .NET ClickOnce add-on

As a number of people have reported, a recent update to Microsoft’s .NET Framework resulted in an add-on being installed into Firefox. Shortly after this patch was released through Windows Update, we were in contact with Microsoft to see how to resolve this issue, as we were hearing directly and indirectly from users that they wanted to uninstall the add-on, and were unable to do so through the Firefox Add-on Manager.

Until recently, removing this add-on from Firefox required that users manually edit the registry, but I’m pleased to report that Microsoft has made available a downloadable patch, and has now added it to the knowledge base article on the topic. Once this patch is applied, the add-on can be uninstalled per-user. (On Windows 7 Release Candidate, the add-on is already the fixed version, at least in my own testing.)

The add-on that was delivered through Windows Update is not compatible with Firefox 3.5, so we’re still trying to figure out how to make sure that 250M-or-so users aren’t confused or — worse — scared off of the upgrade when they are informed that this add-on will be disabled. I’ll report back when we know how that’s going to work, hopefully before Firefox 3.5 is released!

[Edit: removed reference to "disabling".]

advancing open video

Video is a big part of the modern internet, whether it’s used to communicate, educate, or entertain my daughter. We’re building robust support for video (and audio) into Firefox 3.1, making it straightforward for authors to incorporate audio and video media into their pages and applications. We believe that it’s vital to the health of the web for people to approach video on the web the same way they do images: without needing proprietary plugins or paying license fees for restricted codecs, and with the ability to fully integrate into the rest of the page.

Our commitment to the success of open video on the web requires that we select codecs for Firefox that are usable by everyone, without restriction or licensing fee. To that end, we’ve chosen Theora as the format for Firefox 3.1.

We believe that Theora is the best path available today for truly open, truly free video on the internet. We also believe that it can be improved in video quality, in performance, and in quality of implementation, and Mozilla is proud to be supporting the development of Theora software with a $100,000 (USD) grant. Administered by the Wikimedia Foundation, this grant will be used to support development of improved Theora encoders and more powerful playback libraries. These improvements will benefit future versions of Firefox, and anyone else who supports open video on the web.

[Update: Chris Blizzard, being the awesome evangelist I always hoped I'd be, has a great post with a much deeper discussion of why this all matters.]


OK, Deb, you win. I should be doing something else, but until the Dayquil kicks in I’m not likely to be able to do so.

The Rules

  • Link to your original tagger(s) and list these rules in your post. (see above)
  • Share seven facts about yourself in the post. (see below)
  • Tag seven people at the end of your post by leaving their names and the links to their blogs. (see below)
  • Let them know they’ve been tagged. (you’ll just have to trust me)

The Seven Things

  1. I, too, figure skated when I was younger, to mild success. I don’t believe there are pictures of it on the web, which is OK.
  2. I once wrote a multi-user email system on top of DOS batch files, without the use of a text editor (copy con: represent).
  3. In high school, I was an avid member of the school’s technical theatre (and assembly, and school dance, and gaffer-ball varsity) club — avid to the point that I only narrowly managed to actually graduate from high school.
  4. I married my high-school sweetheart.
  5. I do not like to eat coconut- or banana-flavoured things, with a few exceptions like coconut milk in curries because what sort of animal do you think I am anyway? For the past 18 months or so, though, I have been trying to train myself to tolerate, if not enjoy, suchly-flavoured things so that I don’t unwittingly pass my aversion on to Claire. (See also: snakes, though not in a culinary sense.)
  6. I didn’t get my driver’s license until I was 29.
  7. I have lived at 21 addresses in 10 cities.

7 people, all of whom I believe share my disdain for such things to some degree

  • Vlad, for taking care of me in Serbia.
  • Mom, for taking care of me.
  • Phil, for never disguising his contempt for the stupid.
  • Hoye, ibid.
  • Dave, for teaching me to teach.
  • Kev, for being the photographer, bbqer and responsible adult I’d like to be some day.
  • George, for being George.

onward, nimble monkey

Busy times, busy times. You guys see this browser from Google? A few people sent me that link today, definitely interesting stuff. They have a spiffy new JS engine, which has some genuinely cool stuff in it; I’m enjoying reading through the code, and I think I’m learning some Smalltalk along the way.

Since we landed TraceMonkey 10 (ahem) days ago, we’ve been working mostly on stability and bug fixes, as is usual for the early-but-promising stage of new tech. Jesse’s fuzzers have been as deliciously useful as always, and we’ve been able to make some small performance improvements along the way. Things are looking pretty good.

Pretty soon we’ll be able to dive back in and work on more optimizations, and perhaps even write a paper or two. (We’ve learned a lot in the past couple of months, and it would be great to share those lessons with others who might be looking to make dynamic languages fast.) I have lots to keep me busy, but I’m hoping I’ll be able to sneak off and dabble here and there. Don’t tell my boss.

the birth of a faster monkey

Over the past year, JavaScript performance on the web has undergone a striking revolution. Virtually every browser has improved its engine to produce significant gains in execution speed; Firefox is about 3 times faster than Firefox 2 in various JavaScript benchmarks, for example. But of course, developer and user demand for performance is insatiable, and at Mozilla we demand it ourselves, since our application itself is largely and increasingly written in JavaScript. In addition to improving the performance of web applications, our work on JS performance in Firefox 3 made our own application snappier and more responsive.

We’re not done. In addition to continuing to work on our existing JavaScript interpreter (some 20% improved over Firefox 3 already), we’re also looking farther into the future of JS performance, and have some early news to share. Permit me, if you will, to set the stage:

Core primitives improved by 20-40x

These are the early results from a project we’ve been calling TraceMonkey, which adds native code compilation to Mozilla’s JavaScript engine (“SpiderMonkey”). Based on a technique developed at UC Irvine called “trace trees“, and building on code and ideas shared with the Tamarin Tracing project, a few of us have spent the last 2 months (and most of the last few nights) teaching SpiderMonkey some exciting new tricks.

The goal of the TraceMonkey project — which is still in its early stages — is to take JavaScript performance to another level, where instead of competing against other interpreters, we start to compete against native code. Even with this very, very early version we’re already seeing some promising results: a simple “for loop” is getting close to unoptimized gcc:

for loop: gcc 1070ms, js 910ms

Yesterday we landed TraceMonkey in the Firefox 3.1 development tree, configured off by default. We have bugs to fix, and an enormous number of optimizations still to choose from, but we’re charging full speed ahead on the work we need to do for this to be a part of Firefox 3.1. Depending on the benchmarks you choose, you might see massive speed-up, minor speed-up, or maybe even some slowdown — those latter cases are definitely bugs, and reporting them through bugzilla will be a big help.

Here are the current speedups of some common and uncommon benchmarks, as compared to Firefox 3: Apple’s SunSpider, the SunSpider “ubench” tests added for squirrelfish; an image manipulation demo; and a test of the Sylvester 3D JavaScript library doing matrix multiplication.

4 bench graph: sunspider at 1.8x, ubench at 22.4x, image manipulation at 6.5x, sylvester at 6.2x

There are many wins left in each one of those benchmarks, and we’ll be working on those through Firefox 3.1 and beyond: better code generation, more efficient guards, improvements to some data structures, parallel compilation, use of specific processor features, new optimization passes, tracing more code patterns, and many more. Right now we write all values back to memory at the end of every loop, for example, so there are some easy wins available in avoiding that — perhaps 2-3x on tight loop performance.

There’s lots more to write about why we chose tracing as the path to future performance, what to expect in terms of future work, how these sorts of performance gains can translate into new web experiences and capabilities, and what we’ve learned along the way. (One example out of left field: the static analysis tools are written in JavaScript, and apparently they are immensely faster due to even the current JIT work.) Look for posts about that from myself and other TraceMonkey hackers soon.

If you’re the sort of person who reads computer science papers, you may find the Hotpath paper to be of interest: it’s a great paper, and a great introduction to the art and science of tracing.

Thanks are especially due to Brendan, Andreas and David for making it fun to be the dumbest guy on the team; Andreas’ colleagues at UCI (Michael Franz, Mason Chang, Michael Bebenita, Gregor Wagner) for their advice and help; Ed Smith and the Adobe Tamarin team for their tech and wisdom; Rob Sayre, Vlad Vukicevic, Blake Kaplan, Boris Zbarsky and Bob Clary for testing and timely guidance; and the Mozilla developer community for letting us hold the tree closed for more than a day to get it landed.


As Mike and John announced, Mike Schroepfer is going to be moving on from the Mozilla Corporation, and I will be taking over as our interim VP Engineering. It’s impossible to deny that Mike’s absence will be felt, even though Mozilla is the strongest it’s ever been (in no small part due to his leadership and dedication). Thanks, schrep, for everything you’ve given to Mozilla, and to me personally, over the last few years. It has been an honour and a privilege to work alongside you, and it’s the height of flattery to succeed you in the role you’ve defined.

This is obviously a very exciting opportunity for me as well, even though the circumstances make it bittersweet right now. I’m really looking forward to helping our amazing cadre of engineers and engineering managers continue to produce amazing products, and any engineering lead in the world would give an entire collection of limbs for the team we’ve assembled here at Mozilla. Schrep and I are already working on what we expect will be a smooth transition, and this week’s Summit will be a great chance to get everyone synched up.


Yesterday I was granted the rare and spectacular privilege of receiving an honorary degree at Seneca College. It was an amazing affair, flawlessly and warmly executed, at least until the guy giving the convocation address started blubbering at the mic.

It’s a cliché that it’s an incredible honour to be recognized this way, but it’s no less true for being clichéd — and perhaps more for me than for most recipients because I’ve had the opportunity to work with Seneca students. One of the many wonderful opportunities accorded an honorary degree recipient (at Seneca at least) is being part of the barrage of professors and board-of-governors members and faculty who congratulate each graduate. Being able to congratulate some of the students I’ve worked with over the past few years was fantastic, and an unexpected treat.

I wrote some things down before, and then said them — more or less — on the day. They appear below. There is a video of it as well, and because people seem quite interested in watching a grown man cry, I will probably post it once I make it smaller than 100MB.

Thanks to Johnathan Nightingale, who helped me tweak the words, and to everyone at Seneca who listened to them quite politely. And also, of course, to everyone who’s contributed to Mozilla and our work with Seneca.

Thank you Dr. Miner, Professor Humphrey, fellow Senecans.

This is a tremendous and humbling honour; through my own work with Seneca, I’ve come to know first-hand how hard Seneca’s students, faculty and administration work to learn, teach, and make a difference, and I’m very flattered and proud to be counted among you.

“College is for people who do“; Professor Dave Humphrey taught me that, and that certainly describes the Seneca that I’ve come to know and love over the past years. My own Seneca education, if you will, began with some students working on a project with Prof. Humphrey. They wanted to change Firefox, and while there are many schools and classes around the world that undertake such projects every year, it was obvious from the first meeting that there was something qualitatively different going on here. These students were collaborating with industry to make sure that their work was grounded in reality and had practical applications — and it was really interesting work! Long before Apple’s iPhone demonstrated how a touch-based interface could work well with the web, students here at Seneca were making it happen for a small local company. I was hooked; they were about doing.

The more I got to know the students and staff at Seneca, the more I got to see creative, passionate work being nurtured, and people learning at break-neck speed. Not just fundamentals from a textbook, but often things that had probably never been learned or taught in quite this way before. As I remarked several times to my colleagues, it was a good thing that nobody told them that this work was too hard for students! These days, the Mozilla community is virtually littered with Seneca contributions and contributors, and Senecans — some of whom are graduating with us here today — have already affected the lives of hundreds of millions of people; there’s no sign of that great work slowing down. Today the programme of practical academic collaboration that Seneca and Mozilla began together a little over two years is expanding to include even more open source projects, and that programme is the envy of schools around the world. I consider myself incredibly privileged to have witnessed Seneca’s growth into an open source powerhouse, and I will always, always cherish the lessons I’ve learned and experiences I’ve shared as part of it. I’ve had the opportunity to make friends, make mistakes, make software. I’m a better software developer, teacher, and student myself for that time, and I owe a tremendous debt.

But of course college is also for people who learn, and you’re now coming to the end of the most intensive period of that since you learned not to eat Legos. There’s a pretty fun little exercise you can do, just making a list of all the things you know how to do. Bake a cake, normalize a database, skin a model. Then go back over that list and put a star next to the things you’ve learned in the last year or two; your lists right now would have a lot of stars on them, which is as it should be. I do this every few years — I’d do it more often, but I’m pretty absentminded — and it helps me decide when I need to seek out something new to play with.

Many of you will leave here today looking ahead to new jobs and opportunities, and while the workplace has many things going for it (money, for example, which can be exchanged for goods and services), many workplaces make it too easy for us to forget to learn. You’ll tend to be given work that you already know how to do, which is generally good economics and management, but it means that you’ll have to be much more intentional and deliberate about your own learning. You’ll need to seek out projects that provide an opportunity to do new things, and to seek out mentors and peers to help you learn from the new things you try. And you’ll want to find opportunities to teach as well — a truth that has been proven out dozens of times in my own experiences, especially at Seneca, is that teaching and learning run in both directions when they’re being done well. While my daughter is learning to crawl, she’s also teaching me to never start a game that I am not willing to play for an hour, for example. One of the most wonderful things about working with classes here at Seneca is watching the students teach each other, and share in the joy of each other’s success and discovery. You will certainly use and build on the specific skills and knowledge you’ve learned in your time at Seneca; please also prize and build on the feeling that you should be learning. Remember to learn, remember to teach, and remember to do things that can fail, and you will be on a path to making a difference in the world around you. A bumpy path at times, a path with poor markings to be sure, but I believe the only path that’s worth being on.

Congratulations, graduates, and thank you again to Seneca for this incredible honour.

honoris causa

Tomorrow, a tremendous honour will be bestowed upon me by Seneca College. I’m grateful and flattered to the point of speechlessness, which may pose a problem as I deliver the convocation address tomorrow.

Thanks are due to many people for the wonderful time I’ve had working with Seneca, but one person stands out even in that sea of achievement and dedication: Dave Humphrey, without whom there would truly be no Mozilla@Seneca, and who is a great friend in addition to the best teacher I’ve ever had the pleasure of watching in action.

fsyncers and curveballs

[Update: Hi, interwebs! Had to block my blog for a little bit to deploy the full power of wp-super-cache, everything should be fine now.]

A couple of articles have now been written about the unpleasant behaviour that people encounter with Firefox 3 in certain Linux configurations, related to the flushing of I/O and system lag that can see if there is a lot of other disk activity at the same time. There are a lot of moving parts to this issue, and so it’s not surprising that there’s a fair bit of misunderstanding, though some of it seems less well-meaning that others, which makes me a bit sad.

What’s Firefox doing?

Firefox uses a database called sqlite as the underpinning for many kinds of data storage in Firefox 3, including the browsing history and bookmarks data used to provide the Awesomebar’s awesomeness. sqlite is an excellent piece of software, written by people who take both data integrity and performance very seriously, which makes it a great place to put this sort of data. Lots of people use sqlite these days, and we’re proud to be founding members of the consortium that helps support sqlite development.

Databases, perhaps obviously, usually have complex file formats, and require that different parts of their files agree about things like how many records there are, whether a transaction completed successfully, or how indexes match up with the data to which they refer. This makes them more sensitive to data corruption than some simpler formats, like a basic text file. If you get a chunk of a text file corrupted, you can probably edit around it and salvage the rest of the file, but if you get a chunk of a database file corrupted you can often effectively lose all of the data that’s held there. This is one of the tradeoffs for being able to have efficient access to large sets of data, and it’s common to virtually all databases.

One of the things that sqlite does to ensure that the database is not corrupted in the case of a crash is call a function called fsync. fsync tells the operating system to ensure that this file has been safely written to disk, and waits until that’s complete. This provides what’s known as a “barrier”, and makes sure that we don’t get mismatched parts of transactions (groups of related database operations) when we look at the file after a crash. This is very effective: even if the operating system itself crashes or the computer loses power suddenly, we won’t see the database corrupted.

We don’t want to lose the user’s data, because that makes users sad, and we like to make users happy. So up through Firefox RC1, we set sqlite to its recommended setting of “FULL” synchronization. As release-end-game luck would have it, that made some users sad, because they would find Firefox and sometimes other parts of their system would pause unpleasantly, and it was tracked down to Firefox calling fsync. The bug in question is here, but I caution you to not read it piecemeal; there’s a lot of intertwined conversation there, and some comments are not as correct as they sound.

(There have been other bugs along the way that could cause this, ranging from performance problems with certain sqlite versions to bad interactions with the data set used for malware protection, but those were put behind us by RC1. This bug is the one that has people working weekends at this point.)

Why does that hurt so much?

On some rather common Linux configurations, especially using the ext3 filesystem in the “data=ordered” mode, calling fsync doesn’t just flush out the data for the file it’s called on, but rather on all the buffered data for that filesystem. Because writing to disk is so much slower than writing to memory, operating systems can buffer a lot of data, especially if you’re doing something that involves a lot of I/O, like unpacking a zip file or compiling software. It gets written out in the background, giving you vastly, vastly improved performance. It’s no exaggeration to say that without this sort of buffering your computer would be entirely unusable.

I think you can see where this is going: if there’s a lot of data waiting to be written to disk, and Firefox’s (sqlite’s) request to flush the data for one file actually sends all that data out, we could be waiting for a while. Worse, all the other applications that are writing data may end up waiting for it to complete as well. In artificial, but not entirely impossible, test conditions, those delays can be 30 seconds or more. That experience, to coin a phrase, kinda sucks. Does it suck as much as file corruption wiping out your bookmarks after your computer (not Firefox) crashes? As you might imagine, opinions vary.

This problem with ext3 is well-known to Linux kernel developers, and there’s great work underway as part of the “ext4″ project to remedy it. Other filesystems (like reiser4, I have heard) have similar problems, and I presume that their developers are also working on resolving them.

Why doesn’t other (non-sqlite) software do this?

Actually, a lot of software that’s concerned with data integrity does this, including the editors emacs and vim, and mail clients like mutt and Evolution, as well as bigger databases like MySQL and Postgres. In some cases, those programs are in fact adding more calls to fsync to protect user data better.

In fact, so many programs use fsync to ensure data integrity, and actually writing to disk is so expensive, that some operating systems make fsync not be a “real” fsync: the data is scheduled for (hopefully) immediate write-out, but the call doesn’t wait until it’s all the way to the disk, so it’s not really an effective barrier. This may be permitted by various standards like POSIX, but it’s certainly surprising for programs like Firefox that use it to protect against data corruption in the case of a crash!

Here’s what Apple has to say about fsync, sqlite, and data corruption:

fsync on Mac OS X: Since on Mac OS X the fsync command does not make the guarantee that bytes are written, SQLite sends a F_FULLFSYNC request to the kernel to ensures that the bytes are actually written through to the drive platter. This causes the kernel to flush all buffers to the drives and causes the drives to flush their track caches. Without this, there is a significantly large window of time within which data will reside in volatile memory—and in the event of system failure you risk data corruption.


If this is an operating system bug, why is Firefox being patched?

Because we want to make users happy. Whether Linux should have better fsync behaviour or not isn’t really going to matter to our users — we want to support Linux, which means Linux-as-she-is-shipped, not Linux-as-we-would-like-her-to-be. That means that we need to deal with X servers, with font-selection systems, and with filesystem behaviour as we find it, because that’s where our Linux users are. (It’s not like Windows and OS X don’t have their own annoying things to work around either, though they don’t seem to have this specific one!)

So is it always going to be like this?

No! (Yay!)

In the immediate term, there is a patch that controls how aggressively we sync, and defaults to a slightly less-aggressive state that is equivalently safe on modern operating systems. That patch will be in either Firefox 3.0.1 at the latest, and we’ve been in contact with Linux distributors to make sure they’re aware that the patch is fine to take in their builds — desirable, even. It might be in Firefox 3.0 proper, depending on what happens with an RC2, but either way the vast majority of affected users (Linux users, who usually get their Firefoxes from their distributors) will get the fix right away. This patch also lets users, who might decide that their systems are stable enough and their backups good enough that they don’t need the extra protection, turn off the data-integrity fsyncs almost entirely.

In the medium term, we’re going to be making our database use more asynchronous in Firefox, and batching transactions for things like history. You’ll likely see those effects in the major version of Firefox that follows 3 (might be called 3.1, might be called 3.5, might be called Firefox Magenta, who knows?). This will keep Firefox from pausing in these states, but may not entirely keep the fsync calls from affecting other applications on the system. The sqlite developers are also looking at adding support for a newer, Linux-only API that doesn’t have the system-wide effects as fsync, which could help as well.

Longer term, as I mentioned above, the Linux filesystem situation will improve in this respect, and the work is well-underway. It’s probably a year at least before most Linux users are running systems with fixed fsync behaviour, but at that point application developers won’t have to worry about their data-integrity needs causing pain for the whole system. I, for one, am looking forward to it.

I’d like to thank the sqlite developers for their help analyzing the effects of different fsync patterns, various Linux kernel developers (especially my former colleague Andreas Dilger), and the users who have helped test different settings and I/O loads. We’re going to ship a better Firefox 3 on Linux because of it, and we have even more ways to improve that experience on all operating systems in the future. I would not like to thank the people who have substituted vitriol and dogma for analysis and understanding of the release cycle, but we all get frustrated sometimes. I hope they enjoy Firefox 3 too.

it’s full of bits

Deb’s excellent post about Firefox 3′s bookmarking system hit Digg today, on our shared server, which reminded me that I needed to install some WordPress caching software.

big network spike around 9AM Eastern

No sweat; wp-super-cache, I thank you.

« previous pagenext page »