Guest Commentary: At a startup, measuring everything is like prioritizing everything. It’s impossible, frustrating, and a recipe for failure. Instead of chasing every measurable aspect of your business, pause, narrow your focus, and measure what makes it unique. As the saying goes, you improve what you measure. So why not measure what matters the most?
For simplicity’s sake, let’s assume for a moment that you’re engaged in an “average” startup and are ready to determine your first set of proper metrics. Chances are that you already plan to measure baseline metrics like revenue, traffic, and pretty much anything out-of-the-box from Google Analytics. These metrics are important, but often not directly actionable or an accurate reflection of the unique value you provide. What really matters is finding the metrics that correspond with your secret sauce.
Fans of Andrew Chen’s blog have likely read about the importance of understanding your business’s unique value creation – the crucial ingredient that makes your business engine really hum. The catch is that measuring value creation isn’t always easy. Determining what specifically to measure depends heavily on the DNA of your business and requires introspection. Let’s take a look at a couple of hot sectors right now and how they might best measure their unique value.
First let’s look at a new media business. Traditionally media businesses (those who monetize content via ads) have measured success based on reach. Comscore ratings, uniques, and pageviews are their stock-in-trade. However, the landscape of how content is valued is changing.
Google is the arbiter of what’s worth seeing on the web, and while Matt Cutts isn’t about to give away the secret formula, it’s pretty clear that quality and originally authored content is becoming an increasingly large piece of the algorithmic pie. How should a new media company measure their value creation?
If you ask Wetpaint CEO, Ben Elowitz, he’ll tell you it comes down to loyalty. Search drives users in droves, but search-driven users are often “one-and-done,” vanishing after they get what they need. Contrast this with a loyal user who endorses your site again and again with their time. More importantly, loyal users are far more likely to share content and refer others.
Finding the right metrics also sends a big message to your team on what ideas you value. In the media business example, reach vs. loyalty metrics can be the difference between one night stands (more content farming, address scrapers!) and meaningful relationships (exclusive interviews, quality content).
The same principles hold true in the ‘chicken and egg’ world of marketplaces. In a marketplace buyers and sellers are required to tango properly and each constituency deserves their own value metric. For this example, however, we’ll focus on sellers.
While I like knowing how well our marketplace acquires new sellers, what I’m more interested in is how well we retain them. Having a thousand sellers come in through the front door doesn’t do us much good if the same number is leaving through the back. Seller retention is a direct reflection of the value we create.
Fortunately, measuring retention is fairly straight forward with cohort reporting. What’s a cohort? A cohort is a group of users who share a behavior or characteristic over a defined period of time. In the marketplace example, a cohort will show you how many sellers stick around over time. Simply monitoring the acquisition of new sellers each month can paint a rose-colored view of how well your site is performing, but a cohort report brings real health and wellness numbers to light.
Checkout the simplified (and fake) example below. In the first column are the cohorts — new sellers grouped by the month they joined. In Month 1, the sign-in rate is 100 percent as everyone in that cohort joined in that month. The second month is their first opportunity to drop out, meaning that the seller didn’t sign in even once; hardly a recipe for retention. As you can see in the January 2011 cohort, only 70 percent of this group signed in during their second month, and only 65 percent in month three. Clearly, there’s an opportunity here.
(For more definition and examples on cohort studies check out Wikipedia or read this post by Fred Wilson).
In this case imagine that a nice feature enhancement targeted at improving seller retention was introduced in mid-February. According to the cohort report, it’s working. Sellers in their second and third months for the February and March cohorts are now sticking around in increasingly higher numbers. Think of the impact this rate improvement will have as it cascades through future cohorts. More sellers, more items. More items, more traffic. More traffic, more sales. It’s a virtuous cycle — and validation that the new feature is on the right track.
Cohort reports can provide a dashboard-like view of your business’s unique value; surfacing intended and unintended consequences of your actions. While reporting to this degree might seem like a large investment, it’s important as operating blind can quietly kill you. Additionally, cohort data can be impossible to gather retroactively so consider capturing it now. You don’t want to find yourself waiting months for valuable data to accrue.
More than just turning data into useful information, value cohorts can galvanize your team and establish a shared focus in ways that a PowerPoint cannot; especially with data-driven developers. The right metrics can help solve endless feature debates and even be great themes for developer hacking competitions. Want to give your developers freedom to flex their muscles? Give them a value metric to pursue and watch them go nuts.
Do yourself a favor. Grab a trusted colleague and a beer and ask yourselves what value your business really provides and how best to measure it. You might be surprised where the conversation takes you.
Alex Berg is the Chief Product Officer of Bonanza and Bags Bonanza, an online marketplace focused on creating a browse-friendly experience that helps you discover unique items. Prior to Bonanza, Alex served in leadership positions with Wetpaint, Expedia and Blue Nile.
(Tape measure photo via Aussiegal)
