relevance, your honour?

The search engine business is a tough one. People are generally pretty bad at knowing how to phrase queries to give them what they want, to say nothing of dealing with spelling mistakes and synonyms and stemming, and you have to do all that work basically instantaneously. The relevance of search results might be the only thing more important than performance in determining if users will stick with your particular product, or make the trivial switch to another one.

So I was pretty surprised to discover how, er, idiosyncratic the search results were on Live Search for what I — perhaps naively — think of as a pretty straightforward query.

When searching for “Firefox”, the user might want to find the home page for the product, or a description of the history of the project, or maybe even a review of the software. Both Yahoo and Google give you some mix of that, with what seem to me to be pretty reasonable orderings of results.

The Live Search results are a little more difficult for me to understand, since they have the Silverlight developer FAQ as the first result, then an article about cross-site scripting, then an article about ASP.NET, and then the Wikipedia page about Firefox. You have to go to the 8th entry to get the product’s home page, well below the fold on my machine at least. I’ve saved off the results, in case you disbelieve me, or for some reason can’t reproduce them yourself.

Maybe Live Search users really are a different breed, if that’s what they would be most likely to want when searching for Firefox; a ballsy market-differentiation move by Microsoft, if so.

(Canadians don’t call their judges “Your Honour”, and Americans don’t spell honour that way, so the title of this post is a somewhat impossible reference, but I figure you’ll let that slide.)

25 comments to “relevance, your honour?”

  1. Ben
    entered 24 September 2007 @ 4:58 pm

    Those are some interesting ads at the top. Firefox 2008? Sweet.

  2. entered 24 September 2007 @ 4:58 pm

    be afraid of monopolies?…

    Mike Shaver has an interesting post up about search results. I’m going to assume that this is a case of incompetence rather than abusive and anti-competitive behavior — not that they deserve that assumption of innocence…….

  3. entered 24 September 2007 @ 5:25 pm

    It seems their results are very heavily location dependent, with searches for ‘google’, ‘yahoo’, and ‘microsoft’ all leading me to UK regional sites for those companies (despite me being in Ireland). Maybe the bad Firefox results are a symptom of that?

    Incidentally, circuit judges in the UK and Ireland are referred to as ‘Your Honour’, so your title reference isn’t broken.

  4. James Ross
    entered 24 September 2007 @ 6:13 pm

    Interestingly, if you increase the “popularity” slider (Advanced > Results ranking) for the “Firefox” search – even just one point – the results are ordered much more how you expected.

    I’m also mildly interested as to whether this “fixes” it for everyone else (search for “Firefox {popl=51}” if you don’t want to play with the UI).

  5. entered 24 September 2007 @ 6:43 pm

    Being in london, having de_DE as my first country code I get http://www.mozilla-europe.org/de/ http://www.firefox-browser.de/ http://www.firebird-browser.de http://www.netzwelt.de/browser/firefox.htm (Titled: Firefox, the better browser) http://www.firefox2.de Pretty foxy! So it does depend on your settings.

    @Rory: Have you got enUK or enIE in your settings? Tools->Options->Advanced->Languages in FF.

  6. entered 24 September 2007 @ 7:27 pm

    In California I’m seeing pretty similar results to what Shaver sees up in the soon-to-be-frozen northland (with a 1:1 exchange rate), although the ads are a little different (I see one for Firefox-2007 instead of Firefox-2008, and the order of the last 2 is switched), and the top 5 search results are the same except for the third (I get a rather different MSDN page). The product page is 8th; mozilla.com is 9th, but the mozilla.org project page is 6th (and a mozilla.org keybord shortcut help page 7th).

  7. Robert O'Callahan
    entered 24 September 2007 @ 11:11 pm

    I get better results from NZ too.

    I hate to be paranoid, but Microsoft makes it so easy!

  8. entered 25 September 2007 @ 1:27 am

    Shows better results (but still not ideal) here in Australia.

    Mozilla Firefox http://www.mozilla.com/firefox

    Firefox 5 Minute Challenge lachy.id.au/dev/mozilla/firefox/ campaign/5minute/challenge

    Mozilla CEO speaks out on future of Firefox | APC Magazine apcmag.com/6054/firefoxceo speaks_out

    Firefox – Opera – Andrew Gregory’s Web Pages http://www.scss.com.au/family/andrew/opera/ firefox

    Mozilla Firefox Web Browser – Firefox Download for Free firefox.org.nz

    Firefox News firefox.org/news

    Firefox Help: Firefox FAQ http://www.mozilla.org/support/firefox/ faq

    Mozilla Firefox – Wikipedia, the free encyclopedia en.wikipedia.org/wiki/Mozilla_Firefox

  9. entered 25 September 2007 @ 2:47 am

    Silverlight FAQ is first here in Prague as well. Microsoft isn’t doing itself any favors with this type of thing considering that it already has a tough road to hoe to compete with Google in search, without biasing its engine towards its own products (as this result strongly implies).

    Not that Google doesn’t display some idiosyncrasies of its own. For example, “shaver blog” took me to a journal about removing facial hair instead of to this page, as it obviously should.

  10. entered 25 September 2007 @ 4:31 am

    It seems that the search results are dependent on browser locale – despite being in Germany, I see the same results as Shaver in my en-US Firefox. But I must say, the sorting of search results is generally pretty crappy in Live Search, so I might agree with Asa that this is simply a case of extreme incompetence.

  11. Brian
    entered 25 September 2007 @ 5:05 am

    I tried it on the Russian version (live.ru) and it returns the download page first. Odd.

  12. entered 25 September 2007 @ 7:56 am

    Five years ago, I would have read this and immediately assumed that the cause was ‘evil’. Now I’m pretty split between ‘evil’ and ‘incompetent’ as potential causes…

  13. earlpiggot
    entered 25 September 2007 @ 8:13 am

    The results vary if depending on your browser. What this article reports is what IE shows. If you run the same query on Fx or Opera you get the “expected” results…

    Very interesting

  14. Alasdair
    entered 25 September 2007 @ 9:17 am

    What do you mean, Canadians don’t call their judges Your Honour?

  15. Ben
    entered 25 September 2007 @ 11:08 am
  16. Jorge
    entered 25 September 2007 @ 7:13 pm

    Posting from Costa Rica, I see the same results as shaver, with Silverlight leading the result list. Ugh.

  17. IceArdor
    entered 25 September 2007 @ 9:28 pm

    Those are some pretty horrible results from Live.com. Always be specific when you search. Search for [firefox web browser] and you’ll get better results. The Opera web browser, suprisingly, shows up as the first result regardless of whether you search for [opera] or [opera web browser]. It’s strange that Opera, that shares its name with a genre of music, is better understood by the Live.com search engine than Firefox, which is fortunate enough to have a completely unique name.

    I envy the name Firefox sometimes because it’s cool sounding and unique, so people’s first impression isn’t a fat lady screeching in Italian.

  18. entered 27 September 2007 @ 1:31 pm

    I get the same results as shaver and Wladimir here in Germany, but I also have an en-US browser locale.

    @shaver, btw, what do the Canadians call their judges? Germans say “high court”, at least that’s what I heard them say at the supreme court.

  19. entered 28 September 2007 @ 8:20 am

    In the law classes I took, we were told that it was either “my Lord/Lady” or “Mr./Madam Justice”, though it seems that the former is now considered inappropriate:

    http://cas-ncr-nter03.cas-satj.gc.ca/portal/page/portal/fccfen/FAQ#20 for example.

  20. Anonymous Cowherd
    entered 28 September 2007 @ 8:57 am

    Here from Malaysia with EN locale, the result for Firefox is still Firefox website – http://www.mozilla.com/firefox/

    The Firefox 2008 ad is unnerving though. Mozilla advertised in AdCenter?

  21. entered 28 September 2007 @ 9:06 am

    Interestingly, now I get exactly the same search results as before – but the first two hits (Microsoft) are no longer there. The Firefox download page is still rather far down.

  22. Tommy Wales
    entered 2 October 2007 @ 5:20 am

    The Live Search results are a little more difficult for me to understand, since they have.. . No man, you know the answer: “Our products on top, then everything else”. And about MSN search in general I agre with http://mynull.blogspot.com author MSN, Google, Yahoo provide the same results.

  23. David Rolnitzky
    entered 4 October 2007 @ 3:46 pm

    I took a quick look at ranking for a mozilla.com or mozilla.org domain for organic search on MSN Live, Google, and Yahoo for the following search terms:

    “firefox” “Mozilla” “Mozilla Firefox” “firefox download” “firefox.com” “download firefox”

    Basically, all of these terms were #1 in all three search engines for all but one term — “firefox” — which was #4 in MSN live search (seems there is some variation from other folks posting here, but it’s definitely not #1). And for some reason, the listing that came up in MSN at #4 was “www.mozilla.org/projects/firefox” — not crazy though, considering that the url has “firefox” in it, “Firefox” is mentioned in the title tag and throughout the copy of the page, etc.

    It seems to me that MSN Live search algorithms aren’t totally dialed-in yet (or maybe just not that great to begin with) as the other two big kids on the block.

  24. John
    entered 10 October 2007 @ 2:29 pm

    The new version of Live Search does much better. My first result – from Redmond :) – is http://www.mozilla.org/projects/firefox/.

  25. Helen
    entered 15 January 2008 @ 11:02 am

    In Canada, what to call the judge depends on the type of court (Federal, Provincial, Family, etc.) and varies from province to province. In Ontario, they do indeed refer to their judges as ‘Your Honour’. In Nova Scotia, Family Court judges are called ‘Your Honour’.