A while back I reviewed a lovely book , Lean Analytics, which was posted by Alistair, called Finding your One Metric That Matters. In paraphrasing the post, the author emphasises whilst t's not wise to neglect all other analytical measures and pigeon-hole yourself on one, giving one metric a focus over another one allows you to derive meaning through sustained measurement. Alistair points out:
Communicating this focus to your employees, investors, and even the media will really help you concentrate your efforts.
Choosing the OMTM falls down to three factors, the industry you are in, the stage of your startup growth and your audience.
Industry you are in
Big businesses track a few vital Key Performance Indicators (KPIs) aligned primarily with the corporation's main goal, based on transactional, collaborative, SaaS-based, media, game, or app-centric.
Someone buys something in return for something.
Transactional sites are about shopping cart conversion, cart size, and abandonment. This is the typical transaction funnel that anyone who’s used web analytics is familiar with. To be useful today, however, it should be a long funnel that includes sources, email metrics, and social media impact. Companies like Kissmetrics and Mixpanel are championing this plenty these days.
Someone votes, comments, or creates content for you.
Collaboration is about the amount of good content versus bad, and the percent of users that are lurkers versus creators. This is an engagement funnel, and we think it should look something like Charlene Li’s engagement pyramid.
Collaboration varies wildly by site. Consider two companies at opposite ends of the spectrum. Reddit probably has a very high percentage of users who log in: it’s required to upvote posts, and the login process doesn’t demand an email confirmation look, so anonymous accounts are permitted. On the other hand, an adult site likely has a low rate of sign-ins; the content is extremely personal, and nobody wants to share their email details with a site they may not trust.
On Reddit, there are several tiers of engagement: lurking, voting, commenting, submitting links, and creating subreddits. Each of these represents a degree of collaboration by a user, and each segment represents a different lifetime customer value. The key for the site is to move as many people into the more lucrative tiers as possible.
Someone uses your system, and their productivity means they don’t churn or cancel their subscription.
SaaS is about time-to-complete-a-task, SLA, and recency of use; and maybe uptime and SLA refunds. Companies like Totango (which predicts churn and upsell for SaaS), as well as uptime transparency sites like Salesforce’s trust.salesforce.com, are examples of this. There are good studies that show a strong correlation between site performance and conversion rates, so startups ignore this stuff at their peril.
Someone clicks on a banner, pay-per-click ad, or affiliate link.
Media is about time on page, pages per visit, and clickthrough rates. That might sound pretty standard, but the variety of revenue models can complicate things. For example, Pinterest’s affiliate URL rewriting model, which requires that the site take into account the likelihood someone will actually buy a thing as well as the percentage of clickthroughs (see also this WSJ piece on the subject.)
Players pay for additional content, time savings, extra lives, in-game currencies, and so on.
Game startups care about Average Revenue Per User Per Month and Lifetime Average Revenue Per User (ARPUs). Companies like Flurry do a lot of work in this space, and many application developers roll their own code to suit the way their games are used.
Game developers walk a fine line between compelling content, and in-game purchases that bring in money. They need to solicit payments without spoiling gameplay, keeping users coming back while still extracting a pound of flesh each month.
Users buy and install your software on their device.
App is about number of users, percentage that have loaded the most recent version, uninstalls, sideloading-versus-appstore, ratings and reviews. Ben and I saw a lot of this with High Score House and Localmind while they were in Year One Labs. While similar to SaaS, there are enough differences that it deserves its own category.
App marketing is also fraught with grey-market promotional tools. A large number of downloads makes an application more prominent in the App Store. Because of this, some companies run campaigns to artificially inflate download numbers using mercenaries. This gets the application some visibility, which in turn gives them legitimate users.
Many businesses fall into more than one categories, as well as 'blocking and tackling' metrics common to all companies, which are captured in lists like Dave McClure’s Pirate Metrics.):
- Viral coefficient (how well your users become your marketers.)
- Traffic sources and campaign effectiveness (the SEO stuff, measuring how well you get attention.)
- Signup rates (how often you get permission to contact people; and the related bounce rate, opt-out rate, and list churn.)
- Engagement (how long since users last used the product) and churn (how fast does someone go away). Peter Yared did a great job explaining this in a recent post on “Little Data”
- Infrastructure KPIs (cost of running the site; uptime; etc.) This is important because it has a big impact on conversion rates.
Second: what stage are you at?
A second way to split up the OMTM is to consider the stage that your startup is at, which includes generating attention to get people to focus on your product or service, through various media campaigns, as well as need discovery which is a qualitative method of finding out through surveys and interviews what fields aren't being answered, different hot trending areas that are or aren't being fulfilled. Finally, whether you are fulfilling the need, through tools such as metric amplification, (how much does someone tell their friends about it?), understanding whether your offering meets the entire need or is it a piecemeal.
Then there’s Feature optimization. As we figure out what to build, we need to look at things like how much a new feature is being used, and whether the addition of the feature to a particular cohort or segment changes something like signup rates, time on site, etc.
This is an experimentation metric—obviously, the business KPI is still the most important one—but the OMTM is the result of the test you’re running.
Another attribute is to question whether your business model is correct, through business model optimization, calibrating the offer slightly, such as how you charge, how that affects the core KPIs, to determine how scalable you are for growth, and how your organic development is progressing.
Later, many of these KPIs become accounting inputs—stuff like sales, margins, and so on. Lean tends not to touch on these things, but they’re important for bigger, more established organizations who have found their product/market fit, and for intrapreneurs trying to convince more risk-averse stakeholders within their organization.
Third: who is your audience?
Who are you measuring the metrics for? Understand the various stakeholders
For a startup, audiences may include:
- Internal business groups, trying to decide on a pivot or a business model
- Developers, prioritizing features and making experimental validation part of the “Lean QA” process
- Marketers optimizing campaigns to generate traffic and leads
- Investors, when we’re trying to raise money
- Media, for things like infographics and blog posts (like what Massive Damage did.)
What makes a good metric?
Let’s say you’ve thought about your business model, the stage you’re at, and your audience. You’re still not done: you need to make sure it’s a good metric. Here are some rules of thumb for what makes a number that will produce the changes you’re looking for.
- A rate or a ratio rather than an absolute or cumulative value. New users per day is better than total users.
- Comparative to other time periods, sites, or segments. Increased conversion from last week is better than “2% conversion.”
- No more complicated than a golf handicap. Otherwise people won’t remember and discuss it.
- For “accounting” metrics you use to report the business to the board, investors, and the media, something which, when entered into your spreadsheet, makes your predictions more accurate.
- For “experimental” metrics you use to optimize the product, pricing, or market, choose something which, based on the answer, will significantly change your behaviour. Better yet, agree on what that change will be before you collect the data.
You can follow Alistair's tweets by clicking here.