counting still easy, critical thinking still surprisingly hard

Another security study making the rounds today in which someone who purports to know a lot about analyzing security — whose blog tagline, in fact, cautions that “we should try not to simplify [security] to the point of uselessness” — has decided that a product becomes less secure when the developer fixes and discloses vulnerabilities that they find in-house. What Jeff Jones, a director of Security Strategy at Microsoft, has done is simply counted the number of fixed vulnerabilities reported by each of Microsoft and Mozilla, grouping by labelled severity.

What could be simpler? Perhaps nothing. What could be more useless? Again, perhaps nothing.

You can only count what the vendor wants you to see

If Mozilla wanted to do better than Microsoft on this report, we would have an easy path: stop fixing and disclosing bugs that we find in-house. It is well known that Microsoft redacts release notes for service packs and bundles fixes, sometimes meaning that you get a single vulnerability “counted” for, say, seven defects repaired. Or maybe you don’t hear about it at all, because it was rolled into SP2 and they didn’t make any noise about it.

We count every defect distinctly. We count the ones that Mozilla developers find in-house. We count the things we do to mitigate defects in other pieces of software, including Windows itself and other third-party plugins. We count memory behaviour that we think might be exploitable, even if no exploit has ever been demonstrated and the issue in question was found in-house. We open our bugs up after we’ve shipped fixes, so that people don’t have to take our word for our severity ratings.

While Microsoft’s senior technical staff are trying to get severity ratings dialed down (unsuccessfully; kudos to MSRC for sticking to their guns), we are consistently rounding our severities up when there’s any doubt at all.

More fixes means less security?

Even if the scales were the same, and we were living in a parallel universe in which Microsoft even approached Mozilla’s standards of transparency and disclosure, the logic is just baffling: Jeff is saying that Mozilla’s products are less secure than Microsoft’s because Mozilla fixed more bugs. By that measure, IE4 is even more secure, because there were no security bugs fixed in that time frame; bravo to Microsoft for that!

I use Microsoft’s software products myself; I’m typing this on a machine that’s running Vista, in fact. Not only am I pretty upset that we see Microsoft referencing this report without disclosing that it was written by a Microsoft director of Security Strategy, but I’m also concerned for my own safety. Do people in charge of security strategy at Microsoft really believe that aggressively concealing the count of fixes that do make it out makes a product more secure? Shouldn’t they be trying to fix more bugs, rather than writing reports that would “punish” them for actively improving the security of their users rather than hoping that defects aren’t found by someone who they can’t keep quiet?

Microsoft should be embarrassed to be associated with this sort of ridiculous “analysis”. We don’t pretend that hiding the rate of fixes improves our users’ security in any way, and we never will. We’re transparent and aggressive in dealing with security issues, and 130 million Firefox users are safer for it every day.

22 comments to “counting still easy, critical thinking still surprisingly hard”

  1. VanillaMozilla
    entered 30 November 2007 @ 7:01 pm

    Gee, they kind of got an earful of comments on their Web page. I wonder why.

  2. KVB
    entered 30 November 2007 @ 9:24 pm

    The shamelessness of Microsoft passing off such partisanship as an independent study beggars belief. Tangentially there’s nothing wrong with bundling fixes, we also bundle fixes. There are tens of critical fixes listed under some instances of “Crashes with evidence of memory corruption” for example.

    http://www.mozilla.org/projects/security/known-vulnerabilities.html

    (repost due to having to whitelist JS)

  3. entered 1 December 2007 @ 3:44 am

    We stopped using IE long back so it does not matter to us…

  4. entered 1 December 2007 @ 7:48 am

    KVB: when we group fixes into a given advisory, we also provide a complete list of the fixes that are part of that advisory, making them countable and knowable; as you say, there are 27 bugs publicly listed as part of that advisory. That’s in strong contrast to Microsoft’s practice, and that of many other vendors, in which the advisory is the only bug artifact that external folks ever get to see.

  5. entered 1 December 2007 @ 10:42 am

    [...] Updated 12/1/07: Here’s our side of the story. Window Snyder, chief security something-or-the-other: Critical Vulnerability in Microsoft Metrics Mike Schroepfer, VP engineering: Apples, Oranges and the truth Mike Shaver, chief evangelist: Counting still easy, critical thinking still surprisingly hard [...]

  6. Bryan
    entered 1 December 2007 @ 11:23 am

    Hmm. It seems that MS is damned if they do, damned if they don’t. Have and fix a lot of bugs? Clearly it’s a buggy Microsoft product, run away! Have and fix fewer bugs? Clearly Microsoft are hiding stuff, run away run away!

    Critical thought always means Microsoft loses, right? I mean … it’s Microsoft; anything they say must be a lie.

  7. entered 1 December 2007 @ 11:33 am

    No, Bryan, my whole point is that counting fixes doesn’t tell you about the security of the product, or of its users. Jeff Jones is counting badly (due to well-known differences in disclosure), and what he’s counting doesn’t correlate with product security. He’s asking the wrong questions, and answering them badly on top of that.

  8. Bryan
    entered 1 December 2007 @ 1:45 pm

    shaver,

    I looked carefully at how Jones claims to have counted vulns and patches. In his own words, right from the report:

    –quote– 1. Compiled a list of vulnerabilities identified as affecting the respective browser (IE or Firefox) in the National Vulnerability Database (NVD) (http://nvd.nist.gov). 2. Marked a vulnerability fixed if either a vendor advisory noted it as fixed, or if the NVD referenced an advisory as addressing the issue. The latter was necessary because vendor advisories do not always list issues addressed by CVE identifier. For example, MFSA2005-50 does not mention any vulnerability by CVE identifier. However, the NVD entry for CVE-2005-2265 identifies MFSA2005-50 as the patch advisory for that issue. (fyi, this step benefitted only Firefox.) 3. Scrubbed the remaining list to remove rejects and duplicates as acknowledged by the NVD, plus issues where the browser was listed in error or the actual vulnerable product was not the browser. For example, CVE-2007-3657 lists Firefox 2 as an affected product, but it also indicates that other researchers have disputed the issue. Similarly, CVE-2007-1377 lists Firefox as affected, but the flaw is actually in an Adobe plug-in, so I did not count it. 4. Looked at various other references to the issue to try and determine if it applied to the browser version in question. –end quote–

    This, to me, seems to alleviate at least some of your contentions about differences in disclosure and vuln counting.

  9. Bryan
    entered 1 December 2007 @ 1:46 pm

    Apologies for the bad formatting of the above. Without a preview function it’s hard to get it right. If you want to clean that up, I don’t mind.

  10. entered 1 December 2007 @ 4:26 pm

    Mr Shaver,

    1- What is “MSRC”? I would appreciate it if you could use the abbr HTML element from now.

    2- If you want to fight propanganda and poor marketing strategy regarding browser security, browser vulnerabilities/flaws etc, then I ask you to read Firefox FAQ Is Firefox more secure than Internet Explorer?

    http://www.mozilla.org/support/firefox/faq.html#mozvsie

    and then comment on the given answer and submit proposals, suggestions you feel are suitable for this issue. Like what would be a fair, honest and relevant answer to this important question (more secure, better security, etc..). Browsehappy.com is also about this precise issue: so, it’s not a trivial question. Same thing about anti-phishing performance, anti-phishing protection reliability and capabilities.

    Remember that you and I do not work for Microsoft: so, you and I can only improve the product we chose to work/volunteer for.

    Regards, Gérard

  11. entered 3 December 2007 @ 10:55 am

    [...] auch inhaltlich kritisieren die Firefox-Macher die von Microsoft erstellte Studie. In der Studie von Jeff Jones wurde ermittelt, wie viele [...]

  12. Chris
    entered 3 December 2007 @ 5:59 pm

    That’s a pretty funny comparison by Microsoft. I guess since Firefox obviously trumps them in flexibility and speed, they’re grasping at straws from the security angle. It’s like politics or commercials–you’re free to say whatever you want, even if it purposely misleads people. Probably it takes a lot of “politics” to rise to the exec level in MS, so it comes as second nature to Jeff.

  13. LinuX
    entered 3 December 2007 @ 9:37 pm

    spankings for IE. its swiss cheese as far as i’m concerned. Hail to Mozilla!!

  14. Mele20
    entered 4 December 2007 @ 2:31 am

    You didn’t comment on Jones’s remarks regarding Mozilla dropping support for earlier versions of Fx so quickly, whereas, IE6 is still supported and even IE 5.5 for W2000SP4. I use Fx 1.5 and I have no support from Mozilla for security fixes. I think that is awful. I don’t like 2.0 (which I do run on a virtual machine and dislike using that machine because it doesn’t have 1.5 which I LOVE). I will try 3.0 but I probably won’t use it. 1.5 is my choice for several important reasons and I think Mozilla should show the sort of respect toward people like myself who have good reasons for using 1.5 as Microsoft has toward W2000 users and XP users. If Microsoft is smart they will not force IE7 as part of XP SP3 and will support IE 6 for a long time. Mozilla should do the same as far as security patches go.

    I started out with Phoenix and Mozilla many years ago because of TBE. TBE doesn’t work well on 2.0 although I have cobbled together fixes with the help of other TBE lovers for whom, like myself, FX is TBE basically so that it does work on 2.0 but not smoothly as it does on 1.5. I don’t care about the new features in 2.0 or the many privacy violations that were introduced in 2.0 and had me spending hours fixing all those. SeaMonkey still supports TBE beautifully (and TBE works on SuiteMonkey 2.01aPRE also) so I use it more now. There is really not a lot of reasons to use Fx without TBE. There are many users out there who have reasons for why they use an earlier version of Fx or use IE6 on XP instead of IE7. I think Mozilla should still be supporting security fixes for 1.5 as 2.0 is a radical departure in many ways from every version before it and not everyone welcomes that.

  15. tincerbell
    entered 4 December 2007 @ 7:48 am

    I would like to see how Opera does compaired to both IE and Mozilla

  16. entered 4 December 2007 @ 8:50 am

    [...] up in this spat is Mozilla’s Mike Shaver who says flaw count is misleading since Microsoft hides patches in service packs.  That’s a really silly argument since there hasn’t been a Microsoft Windows desktop [...]

  17. Jon Lennart Aasenden
    entered 4 December 2007 @ 9:12 am

    The first thing that came into my mind with regards to this matter, was a real-life, obsolete dinosaur. Microsoft seem to have finally reached it’s evolutionary dead-end. Open-Source is immune to the fangs of capitalism, it’s really that simple. The men in charge of Microsoft represents an era of thought that is becomming more and more alien to our time-frame. They are, to be blunt, stuck in the late 80′s, early 90′s mentality where quantity was always better than quality. It’s like listening to a russian propaganda broadcast from the cold-war.

    I have serious doubts that the current CEO of Microsoft have the abillity, or even the mental capacity to integrate the company into the new values that society fosters. They look more and more like a raging dinosaur that havent understood that it’s already dead. By eating it’s own population beyond the environments abillity to recover -it has effectively made itself instinct. Somthing the Mozilla logo fittingly reminds it of..

  18. entered 4 December 2007 @ 10:22 am

    [...] a rebuttal to the report, Mike Shaver has something this to say.  “Just because dentists fix more teeth in America doesn’t [...]

  19. entered 4 December 2007 @ 10:34 am

    [...] entidade, diz que o erro central do estudo é confundir mais correções com menos segurança. Em seu blog, Shaver cutuca o [...]

  20. entered 4 December 2007 @ 12:55 pm

    [...] entidade, diz que o erro central do estudo é confundir mais correções com menos segurança. Em seu blog, Shaver cutuca o [...]

  21. entered 4 December 2007 @ 4:16 pm

    [...] El enfado de Mike Shaver, trabajador de Mozilla Foundation, ha venido a raíz de que las vulnerabilidades mostradas acerca de Firefox son muy superiores a las de Internet Explorer en todo momento, igual que se dice que Firefox ha tenido que arreglar más problemas, y por lo tanto es más inseguro. También porque ese informe puede haber sido elaborado desde un punto de vista totalmente subjetivo. A causa de todo ello, ha acusado a Jeff Jones de hacer el informe como si fuese un simple estudiante, y no un jefe de seguridad. [...]

  22. entered 4 December 2007 @ 4:30 pm

    [...] raportului au fost imediat disputate de Mike Shaver de la Mozilla, care a afirmat ca “doar pentru ca dentistii repara mai multi [...]