All data analyses start with a specific question to which an answer is needed. The question could be “How many users do we have?” The answer is inevitably in the form of some metric, or measure. Perhaps users who logged in within the last 2 weeks. Or maybe customers who took a specific action (e.g. sent a private message via the platform) twice in the last month.
Since metrics are used to solve business problems, it is evident how important it is that the right metric is chosen. The right metric is a matter of professional judgement – the question, business problem, industry, and market, all need to be considered.
What are the characteristics of good metrics?
Good metrics are
- Actionable: They need to provide information that you can act on. A measure of total visits to a website since its inception provides no actionable insight. A measure of visits to a website this week is better.
- Interpretable: To change working behaviour, attitudes, and actions, the measure of success and failure should be easily understood. A complex performance score makes it difficult for anyone to know what should be done.
- Robust: Metrics should not be robust to manipulation, and be aligned with the best interest of the business. Measuring business success by value of sales closed may lead to hard-selling at the expense of business goodwill in the long term. Metrics should also be robust to noise and unwanted variability. For example, hourly visitor data may be noisy and contain too much variability from one hour to another. Using more aggregate data (e.g. across a week) can help iron this out. A seasonal adjustment can also be applied if there are expected cyclical variability.
- Timely: A measure of what happened last week is better than a measure of what happened last year. A forward-looking predictive metric (e.g. what’s in the sales pipeline) is even better as it means that future problems can be addressed now.
Defining the right metric is often done early in data analysis, but it should not be rushed. It is incredibly important that the right metric is selected to ensure that the analysis is of the greatest business value.