A law professor is giving Microsoft a statistical smackdown over its “Bing it On” search challenge, but the company isn’t going down without a fight.

microsoftbingitonIan Ayers of Yale Law School is challenging Microsoft’s claims about its “Bing it On Challenge,” the blind “taste test” that pits Google vs. Bing to determine which search engine’s results are preferred by more users. Microsoft’s original commercials for the challenge claimed that people preferred Bing “nearly 2:1” in blind tests.

But a study by Ayers and his students (PDF) declared Google the winner:

“We found that (i) a statistically-significant majority of participants preferred Google search results to Bing search results (53% to 41%); and (ii) participants were significantly less likely to prefer Bing results when randomly assigned to use popular search terms or self-selected search terms instead of the search terms Microsoft recommends test-takers employ on its website. Our findings suggest that some of the claims implicit in Microsoft‘s advertisements warrant legal scrutiny.”

Microsoft says in a statement, “There have been unfair comments challenging the claims used on the Bing It On website, and we’re setting the record straight about Bing It On sample sizes, methodology and more on the Bing blog.”

The situation is complicated by the fact that Microsoft’s original “2:1” claim was based not on the public version of the Bing it On challenge, but on a separate study of 1,000 people conducted by an independent research company on Microsoft’s behalf. After a follow-up study that used different methods, the company revised the claim to, “People prefer Bing over Google for the web’s top searches.”

Ayers says in a post on Freakonomics this week, “To be sure, I’ve often published studies with similarly small data sets, but it’s a little cheeky for Microsoft to base what might be a multi-million dollar advertising campaign on what I’m guessing is a low-six-figure study.”

In the Microsoft blog post, Bing behavioral scientist Matt Wallaert, turns that question around to challenge the methods used in Ayers’ study — noting that the professor split his sample of 1,000 people into multiple groups, including some who were suggested popular search terms and other who were left to select their own search terms.

“A sample of 1,000 people doing the same task has more statistical power than a sample of 300 people doing the same task,” writes the Microsoft scientist.

Addressing Ayers’ comments about the suggested search terms,Wallaert notes that the suggested terms in the follow-up study were pulled from the 2012 Google Zeitgeist report, and the terms used in the public BingitOn site come from trending topics on Bing.

Ayers contends that Microsoft’s ads — inviting viewers to “join the 5 million people who visited the challenge at” — left the misimpression that the “2:1” claim was based on a sample of 5 million people. He says Google might have grounds for a false advertising claim based on this claim.

Here’s the kicker: Microsoft says it doesn’t track the results of the public “Bing it On” challenge for privacy reasons, so the actual results of the larger “taste test” aren’t known.

Like what you're reading? Subscribe to GeekWire's free newsletters to catch every headline


  • Marky Mark and the Funky Bunch

    “Here’s the kicker: Microsoft says it doesn’t track the results of the public “Bing it On” challenge for privacy reasons, so the actual results of the larger “taste test” aren’t known.”

    HUH!?! I tried the “Bing It On” challenge several times. I’m a regular user of both Google and Bing, and i’m interested in search. I tried the challenge specifically hoping that MS would be collecting the data.

    It’s an interesting “Pepsi challenge” theme coming out of the new Penn regime in the Beltway.

    Here’s another kicker for you: I also attempted to use the feature in a foreign country. I even saw banner ads and pop-ups encouraging me to use it.

    Tried to use it and, get this, it didn’t work in the country that I lived in. I’ve got a feeling it was only rolled out in the US – even though if you accessed Bing from other places (say Eastern Europe/Central Asia) you’d see promotions for it all over the place.

    Lame. Bing should encourage more feedback on the feature.

    Bing should also roll out in different languages. Google search takes huge market share (in my opinion) in a lot of countries simply by popping in foreign languages by default.

  • guest


  • macbrewer

    Personally, I stopped believing anything Microsoft said back about the same time they came out with their first real invention: FUD. Fear, Uncertainty and Doubt. That was the 80’s.

    • Guest

      Do trolls go to the same school? Because you all use the same tired talking points.

  • Guest

    A professor who got caught plagiarizing the work of others accuses MS of being dishonest. Erm, okay. Next…

  • Charles

    This article is lame. Bing rocks. Google’s for people stuck in the 90’s and like NKYOB

    • AlsoJohn

      Stupid consumers and their personal preferences. We have no use for your free market economy and choice to use competing products, give us back our monopoly! We must use force on them to use Bing because it is far superior in every way because we say so!

      • Charles

        Word! I love Sarcastaball!

      • The Cappy

        Your attempts at sarcasm seem poorly aimed. In this case, Google is the monopoly and Bing is the competing product. Most people use Google without any sense of their being a choice. Just like most people who use Windows use that without any sense of their being a choice. MS is now in the position of discovering how hard it is to move people from an entrenched monopoly to a competing product. Personally I haven’t been able to discern one of them being especially better than the other, except Bing has a lot fewer ads. Points for that.

  • SeattleMike5

    I use Bing because it works great, and I’m sick of idiots telling people how great Google and Apple are. “Think different”.

    • guest

      Think different! Go to the Starbucks in the other street instead of the 2 on this street! :D

  • Virgil Vergara

    Although I wanted Bing to be a competitor to Google, its still not has good. i still prefer Google and I tested it out myself. Relevancy per search Google pulls out more than Bing and I feel that Bing has a lot of empty white spaces. Of course, all of this is just my opinion. But I’m a stickler for search engine marketing. Therefore, Google wins.

  • Grrtime

    Bing is dogshyt. DuckDuckGo I say

Job Listings on GeekWork