Measuring metrics Part 1
Blog - Measuring metrics Part 1
Dr Mary Wyatt | Published: March 18, 2016
Metrics are one of those things we love to hate. "You can't manage what you can't measure" is often quoted to justify putting a number on every activity in the attempt to increase effectiveness.
But the response noting that there are "Lies, damn lies, and statistics" shows what tension we have about measuring things.
Perhaps it is not the fact that we are looking, but rather the perception that we don't always look with understanding that causes the negative reaction.
Why is it that we often get metrics wrong? Problematic metrics either measures an activity without considering the outcomes, or measures outcomes without considering whether our actions really "caused" them.
In the first instance, we measure activity and think we have done something. This can result in unintended consequences. People react to KPIs based on measurement of activity by performing the tasks faster (which is the point) but too often with less thought or engagement.
We call this "tick and flick" and the results can be very bad – the illusion of work being done with no progress toward the goals of the work. Even worse, people on the receiving end of "tick and flick" know it – and the message of no one caring about their welfare comes through loud and clear.
The opposite can be undesirable as well. Sometimes we measure outcomes and think that we have measured the effectiveness of what we have done to "cause" the change. Politicians love this sort of metrics, because they spotlight the things that matter to the electorate.
This approach can work, but only if the choice of outcomes to be measured is very carefully made. Return to work is an example of a poor choice. When someone returns to work, or fails to return to work, it may be partly caused by the efforts of a particular party or systemic change.
But the results can be partly caused by others, as well. The worker, the employer, the workers support system, legal professionals and the treating medical professionals all can function as blockers or facilitators of return to work.
These influencers are often outside our control. We call them "independent intervening variables". When they are present, we can't be sure whether success or failure is attributable to us, them or a combination of both.
To add to the confusion, there is a lot of talk about "lag" and "lead" indictors – metrics that either shows us something that has already happened or metrics that show us something that we think will predict an outcome in the future.
It's no wonder that metrics have gotten a bad name.
Next time: Constructing metrics that make sense