MOUNTAIN VIEW, Calif. – One of the key components in any technology business these days is the use of analytics to better understand and serve users. But with the power of analytics comes great risk, as well.
Speaking today at VentureBeat’s GamesBeat conference, Greg Richardson, the CEO of game development house Rumble, said one of the challenges is getting analytics to keep up with the pace of development.
“There’s a real challenge for analytics to move at the same speed as the business,” Richardson said.
But a focus solely on speed can prove problematic, according to Richardson.
Rumble received funding from Google Ventures, and one of the things Google did was set up a meeting between Rumble and Deepak Tiwari, Google’s Head of Analytics and Strategic Insights. According to Richardson, Tiwari told him of an important moment in Google’s history. The company was considering adding another sponsored link to its search results, and they were going to do a 30-day A/B test to see what the resulting change would be.
As it turns out, the change brought massive returns. Advertising revenues from those users who saw more ads doubled in the first 30 days. Google was on the brink of officially adding another ad to its results, when Tiwari “laid across the tracks” to get the company to continue its experiment instead.
So Google waited another 30 days. By the end of the second month, 80 percent of the people in the cohort that was being served an extra ad had started using search engines other than Google as their primary search engine. If the company had rolled the change out broadly, it’s entirely possible users worldwide would have fled.
The moral of the story, Richardson said, is that developers need to be cautious with how quickly they apply insights from their testing, until after at least some of the long-term effects come out. While it’s tempting to take insights that you gather about a new feature and apply them immediately, that path could be a road to ruin.
Blair Hanley Frank is GeekWire’s Bay Area Correspondent. He has also worked for Macworld, PCWorld and TechHive. He can be found on Twitter @belril.