A law professor is giving Microsoft a statistical smackdown over its “Bing it On” search challenge, but the company isn’t going down without a fight.
Ian Ayers of Yale Law School is challenging Microsoft’s claims about its “Bing it On Challenge,” the blind “taste test” that pits Google vs. Bing to determine which search engine’s results are preferred by more users. Microsoft’s original commercials for the challenge claimed that people preferred Bing “nearly 2:1” in blind tests.
But a study by Ayers and his students (PDF) declared Google the winner:
“We found that (i) a statistically-significant majority of participants preferred Google search results to Bing search results (53% to 41%); and (ii) participants were significantly less likely to prefer Bing results when randomly assigned to use popular search terms or self-selected search terms instead of the search terms Microsoft recommends test-takers employ on its website. Our findings suggest that some of the claims implicit in Microsoft‘s advertisements warrant legal scrutiny.”
Microsoft says in a statement, “There have been unfair comments challenging the claims used on the Bing It On website, and we’re setting the record straight about Bing It On sample sizes, methodology and more on the Bing blog.”
The situation is complicated by the fact that Microsoft’s original “2:1” claim was based not on the public version of the Bing it On challenge, but on a separate study of 1,000 people conducted by an independent research company on Microsoft’s behalf. After a follow-up study that used different methods, the company revised the claim to, “People prefer Bing over Google for the web’s top searches.”
Ayers says in a post on Freakonomics this week, “To be sure, I’ve often published studies with similarly small data sets, but it’s a little cheeky for Microsoft to base what might be a multi-million dollar advertising campaign on what I’m guessing is a low-six-figure study.”
In the Microsoft blog post, Bing behavioral scientist Matt Wallaert, turns that question around to challenge the methods used in Ayers’ study — noting that the professor split his sample of 1,000 people into multiple groups, including some who were suggested popular search terms and other who were left to select their own search terms.
“A sample of 1,000 people doing the same task has more statistical power than a sample of 300 people doing the same task,” writes the Microsoft scientist.
Addressing Ayers’ comments about the suggested search terms,Wallaert notes that the suggested terms in the follow-up study were pulled from the 2012 Google Zeitgeist report, and the terms used in the public BingitOn site come from trending topics on Bing.
Ayers contends that Microsoft’s ads — inviting viewers to “join the 5 million people who visited the challenge at Bing.com” — left the misimpression that the “2:1” claim was based on a sample of 5 million people. He says Google might have grounds for a false advertising claim based on this claim.
Here’s the kicker: Microsoft says it doesn’t track the results of the public “Bing it On” challenge for privacy reasons, so the actual results of the larger “taste test” aren’t known.
