why facebook?

[I haven't started yet, and what I present here is based on things that are public knowledge, via press or F8 presentations or Facebook's own posts. My impressions are obviously informed by direct conversations, of course.]

As I’ve mentioned before, I’m going to start as an Engineering Director at Facebook some time in November (specific timing is up to the INS). I’m really really excited about it for a number of reasons, even though it means relocating to California. A number of people have asked why I chose to go to Facebook, so I decided to write some of the reasons down.

One reason is that Facebook is probably the most web-influential company in the world on that side of the wire. They’ve consistently invested in the web, from their mobile-client approach, to their APIs, to various tools and whatnot. I have unfinished business with the web myself, and Facebook is a great place for me to continue to have influence over how it evolves.

Another is that the engineering culture at Facebook is simply spectacular. It’s obvious that they’ve invested in it very heavily, from bootcamp and development tools to the testing and deployment model, and it has clearly paid off. It’s going to be a very cool thing to be part of, especially since the world of web-delivered services is so different from the client-side-software one in which I’ve spent the last 6 years.

The third reason is that Facebook’s management team is perhaps the best in all of software right now; Ben Horowitz agrees. (Mozilla operates in such a different way that I wouldn’t really know how to compare, but I’m sure they won’t take offense.) I’m really looking forward to learning a ton working with them (including a very good friend of mine) as well as the other amazing people at FB that I’ve had a chance to meet. In looking around the company while discussing a possible position, I didn’t see anything I didn’t want to work on, or anyone I didn’t want to work with, which was unique in my job-hunting experiences.

And finally, I am by no means an expert on social software and how it can connect people through the web. It’s obvious that personal connections, recommendations, and other shared experiences are going to be central to how the web looks in five, ten, twenty years. I think there’s an enormous opportunity for me to contribute to that, and learn a ton; I think Facebook’s vision of what the web can be is pretty exciting, and will be exciting to help build.

I think Mozilla is a great place, and I would recommend it strongly as a place to work (or a place to volunteer, as I plan to keep doing); it’s unique in the world of software, and changes you forever. I’m thrilled to now go to Facebook, another great place, and see what I can do to change the world again.

such sweet sorrow

One of the hardest decisions I’ve made in my life was the one I made in September to leave my job at Mozilla. And one of the hardest things about that decision was writing the email to my colleagues and friends announcing my decision. Various aspects of timing meant that I announced my resignation during an “all-hands” week — a week-long sync-up for all Mozilla employees — and while it made things much tearier than they might otherwise have been, it was truly wonderful to be able to say goodbye in person to so many of the people I’ve shared the last 6 years with.

This is what I wrote:

People always say that these are terribly hard emails to write, because they are.

When I was 19, I first met Brendan Eich at a conference in NYC. We hit it off (lol nerd-groupie fawning), and it led to me working alongside him at Netscape a year later. The ever-powerful combination of the right time and the right place gave me the opportunity to use my open source experience as part of the founding team for the Mozilla project.

Since that time Mozilla has been a huge part of my life, and a huge part of my career. I’ve decided that it’s time for me to look for another part of my career, and so I’m leaving the Corporation.

I am pretty good at the word thing, but I don’t have any adequate to express how much Mozilla means to me — the project, the people, the changes we’ve made in the world. I love you all, and the things we’ve done together that shouldn’t have been possible.

It’s been wonderful to be surrounded by family here at the all-hands this week. I’m not leaving the family, but I am moving out, so I won’t be around as much as I have been for the past 6 years. Feel free to drop me a line if I can help, or crash on my couch…hmm. You get the idea.

[Some administrivia omitted.] I am leaving with the organization and project in strong, strong hands.

I don’t know what’s next, but you can be sure it will involve the web and trying to make it better. Once that’s in your blood, there’s no getting it out.

Thank you all for many wonderful years; please know that I will always be proud of what we’ve done, and of Mozilla’s incredible, impossible, inevitable successes to come. The vision and courage I’ve seen in this week alone point to a web that won’t know what hit it.

It’s perhaps obvious that I’m tremendously proud of my time at Mozilla, and I feel incredibly fortunate for the opportunities that my work there has provided. Not only did I get to help build great software that changed the web, but I got to do it with brilliant, kind, generous people from all over the world. Looking back at those six years, I wouldn’t want to have to pick out a highlight, so I won’t. I will say that if I had to go back in time, I would definitely do it all over again.

Thanks, Mozilla.

approaches to performance

[This post doesn't have links to anything, and it really should. I'm a bit pressed for time, but I'll try to come back later and fix that.]

Important: I no longer work for Mozilla, and I haven’t yet started working for Facebook, so what I write here shouldn’t be taken as being the stance of those organizations.

Platforms always have to get faster, either to move down the hardware stack to lesser devices, or up the application stack to more complex and intensive applications. The web is no exception, and a critical piece of web-stack performance is JavaScript, the web’s language of computation. This means that improvements in JS performance have been the basis of heated competition over the last several years, and — not coincidentally — an area of tremendous improvement.

There’s long been debate about how fast JS can be, and whether it can be fast enough for a given application. We saw it with the chess-playing demo at the Silverlight launch (for which I can’t find a link, sadly), we saw it with the darkroom demo at the Native Client launch, and I’m sure we’ll keep seeing it. Indeed, when those comparisons were made, they were accurate: web technology of the day wasn’t capable of running those things as quickly. There’s a case to be made for ergonomics and universality and multi-vendor support and all sorts of other benefits, but it doesn’t change the result of the specific performance comparison. So the web needs to get faster, and probably always will. There are a number of approaches being mooted to this by various parties, specifically around computational performance.

One approach is to move computationally-intensive work off to a non-JS environment, such as Silverlight, Flash, or Google’s Native Client. This can be an appealing approach for a number of reasons. JS can be a hard language to optimize, because of its dynamism and some language features. In some cases there are existing pieces of non-web code that would be candidates for re-use in web-distributed apps. On the other hand, these approaches represent a lot of semantic complexity, which makes it very hard to get multiple interoperating implementations. (Silverlight and Moonlight may be a counter-example here; I’m not sure how much they stay in sync.) They also don’t benefit web developers unless those developers rewrite their code to the new environment.

Another approach is to directly replace JS with a language designed for better optimization. This is the direction proposed by Google’s Dart project. It shares some of the same tradeoffs as the technologies noted above (easier to optimize, but complex semantics and requires code to be rewritten), but is probably better in that interaction with existing JS code can be smoother, and it is being designed to work well with the DOM.

A third approach, which is the one that Mozilla has pursued, is to just make JS faster. This involves implementation optimizations and adding language features (like structured types and binary arrays) for more efficient representations. As I mentioned above, we’ve repeatedly seen that JS can be improved to do what is claimed as impossible in terms of performance, and there are still many opportunities to make JS faster still. This benefits not only new applications, but also existing libraries and apps that are on the web today, and the people who use them.

Yesterday at the SPLASH conference, the esteemed Brendan Eich demonstrated that another interesting milestone has been reached: a JavaScript decoder for the H.264 video format. Video decoding is a very computationally-intensive process, which is one reason that phones often provide specialized hardware implementations. Being able to decode at 30 frames a second on laptop hardware is a Big Deal, and points to a new target for JS performance: comparable to tightly-written C code.

There’s lots of work left to do before JS is there in the general case: SIMD, perhaps structural types, better access to GPU resources, and many more in-engine optimizations that are underway in several places I’m sure. JS will get faster still, and that means the web gets faster still, without ripping and replacing it with something shinier.

Aside: the demonstration that was shown at SPLASH was based on a C library converted to JS by a tool called “emscripten”. This points towards being able to reuse existing C libraries as well, which has been a selling point for Native Client thus far.

As Brendan would say, always bet on JS.

what’s next

I’ve now finished up at Mozilla, after a hectic and heart-rending last two weeks. I’m going to take October off to recharge, but I’m already quite excited about what I’m doing next.

On Nov 7, I’ll be starting at Facebook, where my journey will begin with the amazing Facebook engineering bootcamp. It’ll be a pretty different experience for me, no doubt. Of course, my thinking about the web was very much shaped by my experiences at Mozilla and, without speaking for my soon-to-be employer, that thinking is likely a large part of why they were interested in me in the first place.

Throughout my conversations leading up to my decision to join Facebook, I was consistently impressed by the depth and passion of the people I spoke with, from the executive team to the former Mozilla intern who gave me a coding test. I’m going to have a lot to learn, since I haven’t done large-scale computing in half a decade, but I’m really looking forward to pushing the web forward from the other side of the wire.

But yeah, first 5 weeks of vacation, the longest I’ve had since I started professional software stuff in 1992!

moving on

It’s starting to get around, because my co-workers are bad at secrets, so I’ll summarize for “the record”.

I’ve decided that it’s time for me to move on from the Mozilla Corporation, where I have enjoyed 6 years surrounded by incredible people doing incredible things on (and to) the web. I haven’t yet decided what’s next, though I have some exciting opportunities to explore. I am still truly, madly, deeply in love with Mozilla and the web it is building, and grateful for the opportunities that it’s created for me.

I have lots, lots, lots to say about my history with Mozilla and what I think the future holds, but that’s for later. This week is about enjoying the company of my Mozilla family, and celebrating the incredible honour it’s been to work here for the last 6 years. And maybe some sniffling.


Supporting enterprise deployments has always been challenging for Mozilla (and for enterprise deployers), but we’ve nonetheless had some success; one Forrester study from 2010 indicated that Firefox has 20% or more market share within whatever they consider to be enterprises. That success has involved assistance from enterprise IT and OS vendors, both to represent their needs and to contribute actual work. If we want to expand on that success, then we will need to push farther ahead still; we need to be humble enough to know that we need help both understanding and doing.

what do enterprises need?

To my mind, there are two different elements at play here: supporting enterprise users, and supporting enterprise IT.

Enterprise users ask for Firefox to operate fully in their environment. Examples of things that fit in this category include NTLM, proxy auto-discovery, SOCKS proxies, running on Windows XP, and smart card integration.

Enterprise IT ask for a pretty different set of things, though they’re sometimes also the ones who present their users’ requests. Requests include deployment tools like MSI packaging, non-admin updating, preventing updating, pre-installing extensions, filtering/vetting/preventing extension installation and update, control over user settings via things like Group Policy, running quickly from a network share. There are probably many more, since I am mostly inferring now, and details like “control what exactly via GPO?” contain many devils. One proposal was to just provide support for all the knobs that IE (then 7) supported; when I saw that one of the knobs was to disable tabbed browsing, I think I used the word “unlikely”.

I don’t know how much of the enterprise IT checklist is necessary and how much is desirable. Similarly, I don’t know where the value curve and the difficulty curve cross, and it’s been very difficult to find out in detail.

when do enterprises need it?

Enterprise IT and internal app owners (but I think not very many enterprise users) also need varying amounts of time to absorb new versions of software into their environments. I’ve heard tell of 18 month cycles for that sort of thing, which boggles my mind a bit, and makes me weep for the people who are spending a year and a half doing that validation. Adapting to a world where you can get 4-8 browser updates in a year will be challenging for some organizations, though in truth they’ve already been in that world for some time: 3.6.4 with out-of-process plugins on Windows was probably a bigger content-compatibility risk than going from Firefox 4 to Firefox 5.

Chrome and Firefox both now use major version numbers to indicate “checkpoint that we pushed to our users”, rather than a specific amount of nature of product change. Again, though, we’ve been in that grey area for some time, with 3/3.5/3.6 being minor-with-major-characteristics sort of releases.

Our policy prior to FF5 was that we supported releases for a maximum of 6 months after the next release. Some Linux distributors needed to support their customers for longer, and weren’t (yet?) comfortable with bumping along to the next “major”; a couple of them got together and took responsibility for backporting security patches to the branches in question. If an “enterprise-friendly” lifecycle is needed, then that would be one way for enterprises to share the load of supporting their community’s unusual-at-the-scale-of-the-web needs.

why is this so hard?

One aspect is that Mozilla is mostly not composed of enterprise IT staff. This means that we rely on prospective deployers to tell us what their specific needs are, and hopefully to contribute help in meeting them. We’ve tried on a few occasions to collect this information — what sets of features would lead to which deployments with what user impact? — but have had a lot of trouble getting that information into our product planning in a usable way. A surprising (to me) number of institutions will not talk on the record about what they need, which makes it pretty hard for them to join a community conversation about what is worth investing in, and what the results are likely to be.

how can we make progress?

As I mentioned above, we’ve made some slightly successful attempts in the past to engage with enterprise representatives to identify the specific things that are barriers to their deployments. If enterprise deployments will have meaningful impact on the future direction of the web then we need to try another tactic. I don’t know what that is, but we are hearing more about enterprise needs now than ever before, so we have a new opportunity to figure it out.

(People like to point at the long, ugly tail of IE6 as an example of why we should want to make it easy for an administrator to deploy Firefox. Yet administrative ease is not enough, as those on IE6 didn’t even move to IE7 or IE8, which had all the administrative bells and whistles imaginable.)

We would also have benefitted from better communication about the release and update model, and what version numbers now mean (basically nothing). That would have given enterprises an opportunity to involve themselves in the process much earlier, and probably won us some advice about how to explain things even better.

To be successful, I think we’re also going to need enterprises to be more than just consumers of the software and tools. We need to build a framework for enterprises to contribute to the things they care about, and we need for enterprises to make contributions.

Enterprises may also need to change how they think about software rollout, if they want to keep pace with browser development and the evolution of the web platform. This is similar to managing the update cycle of a hosted application like Google Apps or Office 365, which could help the conversation. Some organizations will not be able to adapt, or not willing. If an organization stays on Debian stable for half a decade, they are just going to be left behind by basically all modern software. For less-severe cases, though, there is likely some meeting in the middle that’s possible. It could even lead to more agile deployment of their whole software stack, and more productive and happy users because of it!

Jay has written about this as well, and we’ll post more as we get set up for those conversations.

a three-dimensional platform

Firefox now includes, on all desktop platforms, support for a technology known as WebGL. WebGL allows web developers to use accelerated 3D graphics, including textures and shaders; it exposes the same capabilities used by games like Doom 3 and (optionally) World of Warcraft, and virtually every game that runs on the iPhone, OS X, Android or Linux.

Security professionals, including Context IS, have discovered bugs in the specification (related to cross-domain image loading) and in Firefox’s specific implementation. Both are being addressed, as with security problems in any other technology we ship, but recently the conversation has turned to inherent security characteristics of WebGL and whether it should be supported at all by browsers, ever.

I think that there is no question that the web needs 3D capabilities. Pretty much every platform has or is building ways for developers to perform low-level 3D operations, giving them the capabilities they need to create advanced visualizations, games, or new user interfaces:

  • Adobe is building 3D for Flash in a project called “Molehill“, about which they say: “In terms of design, our approach is very similar to the WebGL design.”
  • Microsoft is doing something similar with Silverlight 5, where they’re bringing XNA Graphics to Silverlight 3D.

Adding new capabilities can expose parts of the application stack to potentially-hostile content for the first time. Graphics drivers are an example of that, as are font engines, video codecs, OS text-display facilities (!) and image libraries. Even improvements in existing capabilities can lead to new types of threats that need to be modelled, understood, and mitigated. We have a number of mitigations in place, including a driver whitelist that’s checked daily for updates; this seems similar to the driver-blocking model used in SL5, based on what information is available. Shaders are validated as legal GLSL before being sent to the driver (or to Direct3D’s HLSL compiler), to avoid problems with drivers mishandling invalid shader text. We’re also working with the ARB and driver vendors on extensions to OpenGL which will make the system even more robust against runaway shaders and the like.

Microsoft’s concern that a technology be able to pass their security review process is reasonable, and similar matters were the subject of a large proportion of the discussions leading to WebGL’s standardization; I also suspect that whatever hardening they applied to the low-level D3D API wrapped by Silverlight 3D can be applied to a Microsoft WebGL implementation as well. That Silverlight supports Mac as well, where these capabilities must be mapped to OpenGL, makes me even more confident. The Microsoft graphics team seems to have done a great job of making the D3D shader pipeline robust against invalid input, for example. (The Windows Display Driver Model in Vista and Windows 7 is a great asset here, because it dramatically reduces the “blast radius” of a problem at the driver level. This likely explains the difference between default-enable and default-disable for WDDM/non-WDDM drivers in SL5′s 3D support. It’s not yet clear to me what the model will be for SL5 on OS X.)

It may be that we’re more comfortable living on top of a stack we don’t control all the way to the metal than are OS vendors, but our conversations with the developers of the drivers in question make us confident that they’re as committed as us and Microsoft to a robust and secure experience for our shared users. Web developers, like all developers, need good 3D support, and — as with Flash and Silverlight — browser implementers will need to be careful and thoughtful about how to expose that functionality securely on all operating systems.

back in a new saddle

I started back at work at the beginning of last week, after more than 4 months of illness. It is very exciting for me. I was like a kid before Christmas the night before my first day back.

I am rarely described as inarticulate, but nonetheless I find it very difficult to describe how and why I’m so happy to be back at work. For more than a third of a year, not only was I learning to recover from and manage a complex condition, but I also had a big piece of me missing: my friends and co-workers and shared passion at Mozilla. Their support was more than I could have imagined, let alone hoped for. There might be something in my eye.

Damon Sicore had been ably managing Engineering in my absence, and has now succeeded me in a permanent capacity. I tremendously enjoyed my job running Engineering, but was also ready to move into a role with a different strategic-versus-operational mix. Given that Damon had been providing so much leadership across all of Engineering before my absence, and even more during it, he was a natural fit as my replacement. I know that he’s going to excel, and that Mozilla Engineering is very fortunate to have him at the helm.

I’ve moved into a new position as VP, Technical Strategy for Mozilla. I have been involved in Mozilla’s technical strategy for some time, perhaps obviously, and I’m pretty stoked about being able to devote my full energy to it.

I plan to write more about my goals in my new position, as well as more about my recovery and related experiences. For now, though, I just wanted to say that I’m back and I’m giddy about it.

I made a thing

A while ago I made a prototype of a simple, high-performance keyword search for bugzilla, using python and Redis.  It was pretty promising, able to do all/any queries against the summaries of all open Firefox bugs in a handful of milliseconds.  The bugzilla maintainers decided to use the database’s full-text engine instead, but I was nonetheless pretty pleased with it.

A month or two ago, I started rewriting it it atop node.js and gave it a simple websocket-based interface.  (It took a while, due to The Unpleasantness rather than the difficulty of the task itself.) It was blazingly fast, easily able to keep up with real-time matching of entered keywords. The queries on the server side were finished in a dozen milliseconds or less, and the remainder of my 200 millisecond request budget was spent on data transfer back to the client. I had to limit the returned result set to the most recent 100 items to keep the transfer time down. I will probably look at compression in JS to improve things there. (Some requests take as much as half a second if I get unlucky on the network and result-set size.)

Building the index takes about 20 seconds on its wimpy VM, though retrieving all the summaries from bugzilla takes a fair bit longer. (It currently indexes open bugs in Firefox, Fennec, Core, Toolkit, NSPR and NSS.) I could do that on a timer and have it be current within 10 minutes, but it would be pretty wasteful: the rate of relevant change is on the order of a few dozen each minute, and updating that many summaries is literally microseconds.

Enter Pulse, Christian Legnitto’s mozilla-wide message broker.  Wired up to Pulse, my index is up to date cheaply every 5 minutes, and once we deploy the bugzilla extension, the index will be updated within fractions of a second.

To support non-websocket browsers, which is currently all of them, I switched to using socket.io for the transport.  The flashsocket transport adds about 100 milliseconds to the round-trip, which is unfortunate but here we are.

In all, it’s about 500 lines of JS on the server side, plus another ~150 lines for the scripts that do the loading, and I think it’s a great example of what tools like pulse and the Bugzilla REST API are going to make possible in 2011. It’s also an example of how holy-crikey excellent Node and Redis are together.

(There’s an installation running here, which might randomly break as I hack on it.)

Update, 2 minutes after posting:

23 Jan 00:16:00 - searching for sex 23 Jan 00:16:00 - undefined: search 1 -> 1 in 0/1 ms
Hi, Internet.

engines half, straight ahead

As I mentioned in my overlong post about my bipolar disorder, the treatment plan informed by my new diagnosis is working quite well. Well enough, in fact, that I was able to give a short talk at the amazing CUSEC conference last week. That success was a major milestone for me, and a proof point that I was doing as well as I had hoped.

So this week, I’m going to be working half-time from the office (mornings), as I build a better structure to some parts of my life, and generally rebuild my brain strength. I still have some days with more anxiety than is usual or healthy, I still cry a little easily (like a bit while writing this post), and my stamina for difficult or frustrating work is about half of where I need it to get back to it. But hot damn, I can work again, and it feels fantastic.

I can’t write about my recovery without mentioning the support that’s made it possible. If someone dear to you is suffering with mental illness, know that your help and support — even just words of well wishing — can literally change their life.

next page »