the missed opportunity of acid 3

Ian Hickson released his Acid 3 test a little while ago, and though there have been a couple of changes recently, it’s pretty much settled down in its final form. There’s been a lot of discussion of how different browsers do on the test, and it’s clear that some projects are really buckling down to get to 100 and the pixel-perfect rendering. You might ask why Mozilla’s not racking up daily gains, especially if you’re following the relevant bugs and seeing that people have produced patches for some issues that are covered by Acid 3.

The most obvious reason is Firefox 3. We’re in the end-game of building what I really do believe is the best browser the web has ever known, and we expect to be putting it in the hands of more than 170 million users in a pretty short period of time. We’re still taking fixes for important issues, but virtually none of the issues on the Acid 3 list are important enough for us to take at this stage. We don’t want to be rushing fixes in, or rushing out a release, only to find that we’ve broken important sites or regressed previous standards support, or worse introduced a security problem. Every API that’s exposed to content needs to be tested for compliance and security and reliability, and we already have some tough rows to hoe with respect to conflicts with existing content as it is. We think these remaining late-stage patches are worth the test burden, often because they help make the web platform much more powerful, and reflect real-web compatibility and capability issues. Acid 3′s contents, sadly, are not as often of that nature.

Ian’s Acid 3, unlike its predecessors, is not about establishing a baseline of useful web capabilities. It’s quite explicitly about making browser developers jump — Ian specifically sought out tests that were broken in WebKit, Opera, and Gecko, perhaps out of a twisted attempt at fairness. But the Acid tests shouldn’t be fair to browsers, they should be fair to the web; they should be based on how good the web will be as a platform if all browsers conform, not about how far any given browser has to stretch to get there.

The selection of specifications is also troubling. First, there is a limitation that the specification had to be suitably finished by 2004, meaning that only standards that were finalized during the darkest period of web stagnation were eligible: standards that predate people actively reviving the web platform, through work like the WHATWG’s <canvas> specification. While this did protect us from tests requiring the worst of SVG’s excesses (1.2, with support for such key graphical capabilities as file upload and sockets, was promoted to CR in 2007), it also means that it includes @font-face, a specification which was so poorly thought of that it was removed in 2.1. I can think of no reason to place such time-based restrictions on specification selection other than perhaps to ensure that there has been time for errata to surface — but in the case of CSS, those errata are directly excluded! Similarly, the WHATWG’s work to finally develop an interoperable specification for such widely-used things as .innerHtml were excluded. (If the test is about “fairness”, I could see not wanting to target specifications that were so new that people just hadn’t had time to catch up to them, but again I think that such a test should be built on more long-term criteria than lining up the starting blocks for a developer sprint. We could be using Acid tests as an “experience index” for the web, if you will.)

I also believe that the tests should focus on areas where the missing features can’t be easily worked around; SMIL’s capabilities are available to interested authors via an easy-to-use library, and if Hixie could stomach digging around in the SVG specification I wish he’d spent his time on things like filters or even colour profiles, which lacks are much harder to work around.

People who misread my disappointment or our lack of last minute Acid-boosting patches as a lack of commitment to standards would do well to study our history as a project. For more than a decade, we’ve been dedicated to support of standards, and improvement of those standards, even though it has often been painful to do so. In 1998 we threw away our entire layout engine in order to rebuild on one that provided, we believed, a better basis for the new standards of the day: DOM, CSS, etc. We’ve changed our implementations of <canvas>, of globallocalStorage, of JavaScript — all to track changes in standards, and to avoid conflicting with them. Last night, we disabled cross-site XHR because we aren’t certain what way the spec was going to go, and if, e.g., we ship something that doesn’t send cookies, we would constrain where the spec could go by building developer expectations about that behaviour. (These developer expectations about what sorts of requests can be triggered from cross-domain content are basically the entire reason that we need special work for cross-site XHR mashup technology.) We will fix standards compliance bugs; it’s what we do. But we won’t fix them all with the same priority, and I hope that we won’t prioritize Acid 3 fixes artificially highly, because I think that would be a disservice to web developers and users. Where Acid 3 happens to test something that we believe is important to fix, we will of course pursue it: surrogate pair handling or some of the selector bugs seem like good candidates.

Acid 3 could have had a tremendous positive effect on the web, representing the next target for the web platform, and helping developers prioritize work in such a way as to maximize the aggregate capabilities of the web. Instead, it feels like a puzzle game, and I can easily imagine the developers of the web’s proprietary competitors chuckling about the hundreds of developer-hours that have gone into adding another way to iterate over nodes, or twiddling internal APIs to special case a testing font. I don’t think it’s worthless, but I think it could have been a lot more, especially with someone as talented and terrifyingly detail-oriented as Ian at the helm.

Maybe next time.

[Update: the WebKit checkin to special-case Ahem wasn't one that used a private OS X API, it was one that used an internal CoreGraphics API on Windows; I should have been reading more closely, apologies to the WebKit folks.]

35 comments to “the missed opportunity of acid 3”

  1. entered 27 March 2008 @ 3:26 am

    So, where is your test?

    I don’t get your argument about how the test is crap. Why test stuff we know works? I agree the test could be bigger, but if you disagree about what 100 items to test, write your own and start promoting it.

    I think Firefox 3 should pass the test, because that is the defacto test suite now. Like it or not.

    Slashdot already post stories about which browser has what score. Who cares if it is 100% relevant, it 100% gold in terms of PR.

    monk.e.boy

  2. entered 27 March 2008 @ 3:26 am

    Heya,

    So actually, Acid2 and Acid3 were created in much the same way, it’s just that Acid3 was much more open and took much longer. It is also much bigger.

    I would love to have tested innerHTML and setTimeout and all kinds of stuff like that, but sadly there is no spec for those yet (other than the very much in-progress HTML5 drafts). We can’t write Acid tests for things that we don’t have a spec for. I’ve been working my ass off for the past few years to write a spec for these things. Hopefully by, say, Acid5, we’ll be able to write an Acid test for them.

    With Acid2, the original “first cut” failed a lot in IE, Mozilla, and Safari, but actually did pretty well in Opera. We (Håkon and I) then went on a hunt for Opera bugs and made Opera fare much worse on the test. With Acid3, IE and Opera ended up doing really badly on the first cut, and Firefox and Safari did well, so we added some more things that failed in Firefox and Safari. (Then we added even more stuff that failed in Safari, because they kept fixing the damn bugs as I was adding them to the test.)

    I really did intend the test to be “fair to the Web”, as you say; the test includes important tests for selectors, media queries, DOM Traversal features, XHTML, SVG, and a number of other important features that Web developers can’t use today due to the poor state of browser interoperability. Maybe I overstretched with the “100 tests” idea, though, and ended up with too much fluff. Acid4, whenever that comes out, will probably be more in the mold of Acid2 (smaller, more visual).

    As far as the SVG tests go, I didn’t write any of them, the SVG working group and its members did, as part of an open call for tests. If you’d wanted to test, say, filters, you were welcome to send such tests. (I originally was against adding SVG tests, but bowed to public pressure.)

    It is unfortunate that Acid3, like Acid2, was timed really badly for the Mozilla team, and for that I apologise. However, in all fairness, it wasn’t timed any better for Microsoft (who are in the final crunch for IE8, and who only just announced that they passed Acid2!), nor for Opera (who are in the final crunch for Opera 9.5), nor, for that matter, for Apple (who were in the final crunch for Safari 3.1, which shipped after Acid3 was released). Håkon and I will endeavour to make the timing suck less for Acid4, but of course, there’s only so much we can do when it comes to lining up the release cycles of four development teams, some of whom have several products with their own independent timelines.

    For what it’s worth, I really wasn’t expecting anyone to work on Acid3 this side of August. I’m of course very happy that Safari and Opera will pass sooner than that, but making any progress at all in the next year or so is still quite reasonable given the timescales that the Web works with. (Of course, one would hope that progress would happen faster than the 2+ years the IE team took with Acid2.) In any case, Acid3 bugs are certainly not fodder for late betas, I agree entirely.

    I’ll add your name to the list of people to contact directly when doing Acid4. I hope you can guide Håkon and I into making a test that is of higher quality and doesn’t miss important opportunities.

    Cheers, Ian

  3. entered 27 March 2008 @ 3:51 am

    I think you miss the point monk.e.boy. Whilst good PR is nice, Mozilla’s top priority is to promote the health of the open Web. If passing certain ACID3 tests is deemed to be less critical to the health of the Web than other Mozilla work, then don’t expect those ACID tests to get undue developer attention.

  4. Bernie
    entered 27 March 2008 @ 3:54 am

    Which version might be able to have all the fixes?

    Fx 4, 3.1, 3.0.1, 3.0.0.0.0.0.1 ? :)

  5. anon
    entered 27 March 2008 @ 4:12 am

    Acid 3 came late in the Fox 3 cycle, everyone gets that. No one was expecting the Mozilla team to be rushing to implement big new features when the focus needed to be on regressions, bug fixes, and release.

    However, yours and Sayre’s complaining about the test, the specs, and the process, comes off as sour grapes. Really, it’s not needed, and I think not intended.

    If, as hinted by your “proprietary competitors chuckling”, you are serious about the ‘open web’ taking on Flash and Silverlight, you must see the importance of some of the things the pair of you are belittling.

    You may not like @font-face but site-defined fonts has been near the top of web designers most-wanted list for years. The new Safari at least presents an option that is better than the old alternatives of text-as-image or, yes, Flash.

    Likewise, animated vector graphics are clearly in demand, and if you read the FAQ for your ‘easy workaround’: http://leunen.d.free.fr/fakesmile/faq.html#aintperfect “Animated SVG should be a first-class citizen on the web as it is an open and standard alternative to Flash. If you are an end user seeking a better experience, I suggest you try Opera. This browser supports SMIL natively and more efficiently.”

    And finally, if Mozilla really does have serious reservations about the merits of some technology, can we have those technical arguments presented, and less of the puerile… Sayer: “SMIL… wtf?”… not needed.

  6. lockoom
    entered 27 March 2008 @ 5:00 am

    Acid 3 was semi-openly developed. Browser vendors were asked to propose subtests. Maybe not all tests are real life problems but as you probably know developers always find “creative” use of the standard. That’s were the egde cases are important. And I see you (and Rob) say that SMIL is not important. 5 years back some people would say the same about SVG itself. Time to move on. Proposing use of libraires instead is plain wrong. When it comes to animation performance is crucial. Lastly, I have strange feeling that if Acid 3 came to public in different stage of Fx 3 development you wouldn’t be so anti-acid3.

  7. Chris Howe
    entered 27 March 2008 @ 5:54 am

    I have to disagree. As deliberately obscure as the test may be, the fact remains that by the time all the major browsers pass 100%, a chunk of web standards that were previously practically unusable are now supported, and that can only be a good thing. Acid 3 in particular was themed around the notion of the dynamic web, which WaSP quite rightly identify as the future direction of web development. It may seek out edge cases, but it’s in the edge cases that the problems often lie, and if it helps focus browser developers on treating the standards with rigour rather than picking the low hanging fruit and leaving the tricky edge cases to be someone else’s problem, so much the better.

    Mozilla may have better things to do now than race to hit 100% just for the PR and that’s fair enough. Hixie’s contributed to Mozilla enough in the past hasn’t he? So I’m pretty sure it won’t be long until you’re up there with Opera and Webkit. You have to admit though that if the timing had been different and Mozilla had got there first you’d have Mozilla devs’ blog posts trumpeting that fact just like they did with Acid 2. So while I can see you don’t need to excuse Mozilla’s third place standing, the criticisms of the benefit of Acid 3 do come across a little as sour grapes, I’m afraid. Sometimes it’s better to say nothing than foster anti-PR…

  8. Hank
    entered 27 March 2008 @ 6:19 am

    I agree with monk.e.boy on this, at least: Where’s your test (or test suite, whatever) that addresses the things you believe are important? And I’m not the only one; Dan_Farina on reddit had the same idea.

  9. entered 27 March 2008 @ 6:32 am

    It’s really useful to read this, and understand what Mozilla is thinking/doing with regard to Acid3. Thanks for sharing it.

  10. Ian
    entered 27 March 2008 @ 6:37 am

    I think one of the good things about Acid 3 is that it tests the DOM a lot, an area in which IE has traditionally been weak, and that might hopefully nudge IE DOM support along a bit.

  11. entered 27 March 2008 @ 6:53 am

    “I think Firefox 3 should pass the test” — We’d all absolutely love FF3 to pass, but that would mean pushing the release back a significant distance while the Mozillians went through the whole development/test cycle for the missing features. Things like @font-face, SMIL and so on would be great features to have, but I don’t think they’re simple additions.

    By the way, the upcoming version of Opera (9.5/Kestrel) is unlikely to fully pass the test for much the same reason — they’ve only got 100/100 on an internal test build of a future iteration of their core engine.

  12. kL
    entered 27 March 2008 @ 7:12 am

    Other developer teams somehow found time and resources to pass the test. If it’s not the case that they’ve already topped Gecko and had nothing better to do, then maybe you could point out weaknesses that you find relevant and would like others to implement?

    Acid4? HTML5 test suite? IE-quirks-and-whatnot-test?

    Watching browsers pass these test is a pure pleasure. The more the better!

  13. anonymous
    entered 27 March 2008 @ 7:17 am

    WALL OF TEXT ZOMG

  14. entered 27 March 2008 @ 7:35 am

    Mike,

    I largely agree with your sentiments about the Acid 3 tests.

    To your SVG 1.2 crack regarding “key graphic capabilities” – when these were first proposed in the SVG 1.2 recommendation, there was no formal W3C HTML effort – the web was effectively stagnating as you have stated. Should we also make the same statements about things in the HTML5 specification (such key hypertext capabilities as Canvas, Video and Audio)?

    Regards, Jeff

  15. entered 27 March 2008 @ 7:35 am

    @jonathon- without popularity, firefox will not be able to push the web anywhere, a healthy direction or not.

    I agree with monkeboy- set up or expose some of your tests in a public forum. If everybody supports Acid3 and you are the only ones supporting WHATWG, none of the things that are possible with either will ever be implemented.

  16. entered 27 March 2008 @ 7:58 am

    “If passing certain ACID3 tests is deemed to be less critical to the health of the Web than other Mozilla work, then don’t expect those ACID tests to get undue developer attention.”

    Remove Mozilla and insert IE8, then think like digg/slashdot, imagine the firestorm.

  17. Smoove Moves Joe
    entered 27 March 2008 @ 8:22 am

    Acid3 could’ve tested if good code would work reliably according to the spec

  18. entered 27 March 2008 @ 8:52 am

    I was thinking the same thing, at least in the way of FF and the latest builds. Right now the only thing that is important are bugfixes so that FF3.0 isn’t blasted by enterprises as being ‘immature’ or even worse ‘insecure.’

  19. Victor
    entered 27 March 2008 @ 8:57 am

    How about a test to force all browsers to support the most commonly duplicated functionality with the same syntax? I.e., no more if (someStandardizedFunction === undefined) { someStandardizedFunctionRenamed(foo) } else { someStandardizedFunction(foo, null, 0, null) }

  20. entered 27 March 2008 @ 9:18 am

    I think the crack at SVG was totally valid. I’ve never understood why sockets or file upload should be anywhere near a vector graphics specification – especially one that relies on other existing specs for animation and linking.

    Canvas, video and audio in hypertext otoh are fine by me. Hypertext isn’t just text.

    Mike, I think your opinion makes total sense with regard to focusing on getting features done for Fx3. I can’t stand when a software project is ready to ship and someone runs in with a pile of new requirements. Of course I’d like to see Firefox 3 pass Acid3 but not at the cost of a pile of security patches for rushed code the week after it’s released. When you look at the test as a collection of things that gecko and other engines didn’t do well then it would make sense that they should perform poorly at it.

  21. entered 27 March 2008 @ 9:45 am

    I am very disappointed to hear you are disabling cross-site XHR. Any chance it will make it in the final release? This is a major loss and much more serious than failing Acid 3.

  22. Diego
    entered 27 March 2008 @ 10:44 am

    If acid3 is pointless, it’s yet another reason why it has not sense not to support it.

  23. entered 27 March 2008 @ 11:38 am

    Interesting to read this Shaver. So if I’ve understood correctly, Acid3 tests for functionality that has been dropped from CSS 2 to CSS 2.1? If so, that’s pretty stupid. Wait for the CSS 3 implementation instead.

  24. Samilar
    entered 27 March 2008 @ 11:55 am

    It’s fascinating how in the wake of Opera and WebKit both announcing 100% scores, only now do statements start emerging from the Mozilla/Firefox stable about how the test isn’t right, and trying to achieve a high score is bad prioritization.

    I don’t believe for one minute that you’re really expressing sour grapes about being towards the rear of the pack (with the possible exception of IE8), but you have to admit, your timing is lousy. Why weren’t you saying all this before the other big browsers made their pronouncements?

  25. cris
    entered 27 March 2008 @ 12:08 pm

    is how webkit did this the way u guys really want?

    it passes the test just for the sake of test. partially implementing tiny portion of standard that is checked by the test, while give up others.

    does this help anything?

    mozilla should indeed gets a standard for itself tho.

  26. entered 27 March 2008 @ 12:32 pm

    Firefox 3 won’t be far behind the other shipping competition when we ship, maybe a few points, and we’re something like 30 points higher than Fx2 was; I don’t think there’s anything shameful there. We’re way ahead of the real competition, which is Internet Explorer, for those people who think that Acid 3 signifies something that it didn’t.

    Why now? Because I finally had time to finish the post, albeit at 2AM. I’ve been hammered trying to get fixes into beta5 and therefore Firefox 3 (and trying to learn more about the history of things like @font-face, or how Acid 3 seems to depend on HTML5-specified parsing rules, but not HTML5-specified .innerHTML rules). I’l confess that when I saw people taking hacks specifically for the test I was newly energized to finish it off, if only to convey my hopes that Mozilla wouldn’t follow the same path. We need to not rush to implement the parts of a spec that are in Acid 3 and leave authors to guess whether having createNodeIterator means that they can use the API as specified, or just as covered by Acid 3. (We left SVG filters turned off in Fx 2 for that reason: the performance wasn’t great, and we didn’t want people to feel that they couldn’t use filters until Fx2 had fallen off the web.)

  27. cris
    entered 27 March 2008 @ 12:52 pm

    agree

    main point being. dont let acid 3 test just become a test, it should be helping web developers and users. the method webkit used to pass the test is just pointless. if u pass that testm but doesn’t support the full standard of that test. why bother? to confuse web developers?

  28. Sean
    entered 27 March 2008 @ 7:14 pm

    It’s funny to see you making the same argument that IE made during IE7 about Acid2: http://blogs.msdn.com/ie/archive/2005/07/29/445242.aspx

  29. entered 28 March 2008 @ 5:06 am

    @samilar: Mozilla people did say similar things about the test as is stated in this blog post from the announcement of this test – on IRC when asked by me and in Bugzilla. Check your facts!

    It is plainly obvious that Apple/Webkit wanted to pass Acid3 (their font “fix” is the final proof). They are still very much an underdog compared to Mozilla – and perhaps even Opera who rule the mobile market. (Yes, I know about the iPhone but its user base is still relatively small, especially outside of the US where more than 95 % of the world population lives.)

    In all fairness the Webkit font “fix” bug has been reopened in anticipation of a better solution.

    I planned to contribute an e4x test to Acid3, but the spec is too new, and when I inquired about the idea Mozilla people discouraged me! Since they did not think their current implementation is good enough. That speaks volumes about a commitment to good standards in preference to winning a race.

    And when it comes to commitment to standards Mozilla proved themselves ahead of Opera and Apple/Webkit on at least one account: CSS 3 selectors. Instead of just implementing them in a way that pleases Acid3, Mozilla postponed implementing a few selectors in order to first discuss the spec per se, since it was ambiguous, knowing it would delay improvement of their score on Acid3. Improving the standard was seen as more important than winning the race. I would almost go so far as call that act sacrificial.

    Now I do think it would be wise for Mozilla to do a Firefox 3.1, perhaps as early as 2 or 3 months after 3.0 and not on a different code branch! The reason? To add this stuff, provided the specs have matured: - - XHR X-site requests - CSS 3 selectors - media queries - “real” border radius And perhaps with a 5-10 point better Acid3 score… And perhaps with this bug fixed: https://bugzilla.mozilla.org/show_bug.cgi?id=424973 ;-)

  30. entered 28 March 2008 @ 12:49 pm

    If you’re referring to me as one of the people who recommended against E4X (you don’t say who, so it’s hard to actually address your points), it’s not because of quality of implementation or competitive issues — we’re way ahead of any other browser on that score, as far as I can tell. It’s that the spec itself is badly flawed, and forcing more people to follow its missteps would be a disservice. Similarly, there are behaviours required by the letter of ES3 that are considered to be bad for security if implemented literally, and so we didn’t propose them for Acid 3, though we’ve had the by-the-letter behaviour for years — pushing other browsers to implement them would be a disservice to the web.

  31. entered 28 March 2008 @ 3:50 pm

    “While this did protect us from tests requiring the worst of SVG’s excesses (1.2, with support for such key graphical capabilities as file upload and sockets, was promoted to CR in 2007)”

    The draft that you link to has not made it to CR, it’s an old working draft of SVG 1.2 Full. I fully agree that testing specs that are not even in CR would be inappropriate for the sake of an acid test. Perhaps you are referring to SVG Tiny 1.2, which is in CR, but which does not include file upload?

  32. entered 28 March 2008 @ 3:50 pm

    And when it comes to commitment to standards Mozilla proved themselves ahead of Opera and Apple/Webkit on at least one account: CSS 3 selectors.

    To be fair, Opera has full CSS3 selectors support in Opera 9.5, and added these long before Acid 3 was announced, so I’m not sure your point holds true. To be fully fair, Konqueror added these even before Opera.

  33. entered 29 March 2008 @ 7:41 am

    It was on IRC, two months ago, even before the competition to contribute to Acid3 had been announced. I do not remember the name of who it was I was talking to. It could have been you – it was a name I recognized. Whoever it was I was told exactly the same thing as you are saying now. And my point does not need to be addressed per se. It was just an example of a brave stance and proof of your commitment to standards!

    I guess the real question is will there be an E4X 2.0 standard? Probably a question that will have to wait until ES 4 is reasonably stable.

  34. Bruce Rindahl
    entered 29 March 2008 @ 5:33 pm

    I have to side with the Mozilla folks on this one. They are rightly focused on getting FF3 out the door, not on Acid3. When Acid3 came out officially less that a month ago, it became the primary focus of bug fixes for Safari and Opera. That is their choice and kudos to their work. However, I need FF3 as my sites will not work in FF2. The Mozilla team has been incredibly responsive to bugs and regressions I report to make FF3 better.

  35. entered 1 April 2008 @ 1:02 am

    Samilar: people have indeed been consistently decrying acid3 on IRC for a long time now. Insinuate as much as you want, but he who has ears and a touch of willingness to actively seek out non-blogged opinions could have heard the complaints.

    Personally, I think the criticism’s a touch overblown — there’s lots of good there (among which I would include font-face, even if we assume the “poorly thought of” part is accurate, because then it’ll force movement on the matter), but there’s also lots of cringe-worthiness. I don’t think SVG should have been in acid3, and many of the tests — including tests I personally fixed, so this isn’t fix-envy or something — were just inane and not relevant to the real web. It’s good we’re mostly past the point where we can point to entire unimplemented features to include in tests, but it reduces us to the edge-casey (sometimes to extremes) tests of acid3 if we’re not willing to work from newer standards (I don’t really understand the 2004 restriction).

    I think that was probably me (Waldo) on IRC warning against E4X, not that it especially matters so long as nobody tries to run with E4X. :-)