AMO and the quality bar

addons.mozilla.org has long occupied a special place in the Firefox software ecosystem. It’s the only site in the installation whitelist by default, the default server contacted for update information about add-ons, and where we send users who are looking for hot add-on leads.

That unique position means that there is a lot of value for some add-on developers in being hosted on AMO. Such hosting involves a review process, which I think both reviewers and developers alike would agree is one of the most frustrating parts of the whole system. The intent of the review process is entirely on the side of the angels: help make sure that add-ons are good for users.

The devil, of course, is in the details here. At times, the review bar has been placed entirely too high, in my opinion: otherwise-fine add-on updates rejected because they cause a strict warning to appear in the JS console, for example. In other cases, we’ve had add-ons approved which send some data to a central server, but don’t have a privacy policy listed. The most common and burdensome cases of this latter example tend to be associated with “toolbar-building” services: the ostensible authors of the resultant toolbars typically know very little about what’s being collected or how it’s being managed, which makes for a predictably unsatisfying conversation with reviewers.

(There are other elements of the review process that are inconsistent and difficult, mostly related to needing to reject items for errors in things that the add-on authors can change after the fact without review, but which can’t be helpfully fixed by the reviewers. These are the “easy” implementation artifacts, though, and not really the topic of this post.)

The trade-offs here are painful: adding a standard of “usefulness” or “implementation quality” to the checklist will not only dramatically slow the review process and require more specialized skills among our reviewers, but will also increase the variability between different reviewers’ decisions. Those are all things that I don’t think we can afford to make worse, and both the history and special position of AMO make me tend towards a much more laissez-faire position: if the description accurately describes what the user will get when they install it, especially as far as the collection and management of private information is concerned, then I think we should let the user make the decision about whether they consider the functionality useful. Some popular add-ons duplicate functionality that is already present in the browser, such as preference settings, adding only an alternate means of accessing it, for example, so requiring “significant new functionality” seems to work against the interests of a fair number of users.

At the same time, of course, I think it’s quite desirable to be able to point users at a more “filtered” view of the enormous add-ons space hosted on AMO. We currently have one such view, the recommended list, but that’s not really much of a solution to the broader problem. (It doesn’t try to be, really.)

A minimum rating threshold would be one way to narrow the default search results returned to a user, though it depends on the reliability and resilience of a rating system. Our current one isn’t sufficient to prevent the sort of gaming and distortion that would plague us in such a world, but that’s not to say that a sufficiently robust one couldn’t be developed. (Not “perfectly robust”, mind; just enough to keep the damage well below the gain.)

A simpler system would simply provide a single piece of metadata that could be set by reviewers or administrators using their judgment and likely via some multi-reviewer discussion. This wouldn’t scale as well as the universal rating by users, but would be more resistant to gaming and abuse (and easier to track and remedy if such nefariousness is detected).

This post is already too long, but you can read and write more about various possibilities for rating and approval schemes in the Remora Idea Dump. We’re thinking about and working on ways to help users find good add-ons, in a way that scales across our community, and I suspect it’s something that we’ll be working to improve for some time!

one comment to “AMO and the quality bar”

  1. Mossop
    entered 1 November 2006 @ 12:31 pm

    I think I agree with what you’re saying. The AMO review process shouldn’t be based on what the extension actually does, or even to a certain extent how it does it. If the extension is useless or complicated for the user then it will just get bad ratings, it shouldn’t be denied review in the first place (and I hope that isn’t happening already). What the review should be doing is ensuring the extension is safe, i.e. it doesn’t kill the browser, and doesn’t transmit personal information without the users knowledge.

    I don’t know how the review process works currently, but it seems that you need an objective criteria, and defined techniques for measuring it. One obvious one is the error console and I’d be inclined to err on the side of caution. You say that add-ons have been rejected based on a strict JS warning, I say good. Warnings are indications of potential problems and where they are simply omitted var’s they are trivial to fix, though maybe the review process needs like a second quicker review when the author is just picking nits like these.

  2. Kevin H
    entered 1 November 2006 @ 1:01 pm

    Mike, I’m sorry if this is a bit tangential, but you mentioned the “recommended list”, and I have a comment to make about that.

    I’ve seen several articles that tout IE7′s new “tab preview” mode. Now FF2 doesn’t have that feature built-in, but I know there are several extensions which provide it including several that won awards in the Extend Firefox competition. How is it that none of these award winning extensions are “recommended”? Wouldn’t it make sense to highlight at least one of them in the recommended list, just to help new FF2 users quickly realize that this is not an exclusive feature to Internet Explorer?

  3. entered 1 November 2006 @ 5:08 pm

    One possible defense against the gaming of user ratings is to allow users to identify others they trust (à la social networking sites) and then weight user ratings according to the length and strength of the trust chains which link them to a set of seed users whom we consider eminently trustworthy.

    I have added this suggestion to the Remora Idea Dump.

  4. VanillaMozilla
    entered 2 November 2006 @ 9:48 am

    The Firefox front page at Mozilla used to say “Browse the Web with confidence – Firefox protects you from viruses, spyware and pop-ups. Enjoy improvements to performance, ease of use and privacy.” I notice that it doesn’t say anything about privacy any more.

    Like it or not, the quality of the browser is limited by the quality of the extensions. Right now extensions are being used for commercial abuse of privacy, and AMO is hosting them. It ought to be harder, not easier, to get an extension listed.

  5. John Prophet
    entered 2 November 2006 @ 10:57 am

    How can you allow a company Conduit (http://www.conduit.com/) to spam your site like they, promoting spyware and adware.

    http://www.conduit.com/privacy/ConduitPrivacy.aspx

    There are dozens of lookalike toolbars that do nothing useful but spying and exibit ads.

    I seen several serious developers frustated by your review process and then you allow these scumbags spread there garbage.

    Why???

    Thank you.

  6. J-Mac
    entered 11 November 2006 @ 2:13 am

    I don’t appreciate AMO being automatically whitelisted as long as they continue to host the Conduit/Effective Brand garbage. Some have posted that extension ratings will level the playing field as far as extension quality goes, but these garbage bars are loaded with five star ratings by the dozens or hundreds within a day or two of their initial posting at AMO – isn’t any of this screened for shilling?

    And some have maligned the old Extensions Mirror! AMO sponsors should be ashamed! Privacy obviously is no longer a concern at AMO or Mozilla.org. If you’re not careful, you’ll start catching up with IE…

  7. Steven
    entered 13 November 2006 @ 5:50 pm
  8. Steve
    entered 14 November 2006 @ 4:12 am

    Shaver AMO is going downhill as every conduit toolbar is allowed to swamp the site and search results as you are giving them (the submitters) free advertising. There have also been several concerns of comments on AMO being deleted every time a remotely negative comment was posted. If you must allow these conduit toolbars on AMO can you not at the very least put them in a separate section?. An example thread is http://forums.mozillazine.org/viewtopic.php?t=465643

    And the link you deleted that I posted earlier here http://byron-adams.com/archives/6-Scammers-turn-Mozilla-traffic-into-revenue.html

  9. jan
    entered 20 November 2006 @ 11:50 pm

    to John Prophet and the others.

    I’m a happy user of torrent-search (https://addons.mozilla.org/firefox/2607/)and so are many of my friends. I know Torrent-search was build on Conduit platform but I see nothing wrong with it. Can you please explain me what risk am I taking by using it? I will not use a spyware software… but as much as I checked (and I did!) it’s totally safe. Please elaborate.

    Thanks, Jan

  10. entered 21 November 2006 @ 6:19 pm

    [...] Last week at the Firefox 2 Summit, we came up with a solution to many of these problems that will raise the quality bar and also increase the transparency of the add-on review process. This new system will divide addons.mozilla.org into two sections: a public area (similar to the current AMO) and a sandbox area. The graphic below is a basic summary of how it will work. [...]