Vanity versus Real Metrics : Extract from 'Lean Analytics'

The latest book I have picked up is a book titled Lean Analytics by Alistair Croll, Benjamin Yoskovitz, which is a book aimed at start-up entrepreneurs to understand responsive metric feedback and make better-informed decisions.

Whether you’re a startup founder trying to disrupt an industry or an intrapreneur trying to provoke change from within, your biggest challenge is creating a product people actually want. This book shows you how to validate your initial idea, find the right customers, decide what to build, how to monetize your business, and how to spread the word.

An interesting topic of Vanity versus Real Metrics grabbed my attention, which declared that many companies fail to embrace the 'data' part of the 'data-driven' mantra they self-proclaim to follow. If a company has a piece of data from which one cannot act upon, it is considered vanity metric, which fails to inform and guide one's business model. and help decide on a course of action.

So, when evaluating a metric, it's important to ask yourself: What will I do differently based on this information? and if you can't answer that, you probably shouldn't worry about the metric much, and if you don;t know which metric would change your organisation's behaviour, you are not a data-driven company.

So, if you look at total sign-ups, this is a vanity metric because the number can only increase over time, and thus tells you nothing about what those users are doing and whether they are valuable to you. Total active users on the other hand is a bit better, presuming you have defined what an active user is exactly, but it is still a vanity metric as it will gradually increase over time, as well.

The real metric of interest is the percentage of users active, a real actionable one, and it tells us about the level of engagement your users have with your product, and changes to your product reflects in the metric. So it gives you the visibility to experiment, learn and iterate with the metric. Another interesting metric to look at is the number of users acquired over a specific time period. This is a good metric in allowing you to compare different marketing approaches, such as Facebook or Twitter campaign, a Reddit campaign in the second , LinkedIn in the third and so forth, allowing you to segment experiments by time. It's actionable, so it is quite the opposite of vanity, allowing you to compare various campaigns and work out where to focus your spending on.

Actionable metrics aren't magic, they won't tell you what to do, but at least it allows you to react based on data, and experiment, tweak and the metric is reflected, based on your actions.


I will be reviewing this book in full, in the future, so stay tuned, or better yet, go out and buy this book.


Jump Start CoffeeScript AuthorAlistair Croll, Benjamin Yoskovitz TitleLean Analytics Publisher: O'Reilly Media Year: March 2013