Your metrics are not your goals

I enjoyed reading Haider Rafique’s blog SLAs are for suckers: Why ITSM orgs should be allowed to be more human, so I tweeted a quote, with a link to the blog.

TWEET: SLAs drive bad behaviour

This is an example of Goodhart’s law in action. Goodhart was an economist and his law can be summarised in a few words. "When a measure becomes a target, it ceases to be a good measure." This is because when people know what is being measured, they adapt their behaviour accordingly, and the results are unlikely to be what was originally intended. And it’s a law applies that applies just as well to IT service management as it does to economics.

I was surprised at the amount of discussion my tweet generated. It quickly became clear that some people thought that I was advocating anarchy.  How could I dare suggest not using service level agreements (SLAs) to document services! Madness!

And, of course, they have a point. IT services are almost always documented in SLAs, precisely because SLAs can and should be a useful tool for helping to frame a discussion between service providers and their customers. With SLAs in place the customer knows what service they’re going to get, and the service provider can plan to put in place the resources needed to meet customer expectations.

Unfortunately, that’s often not the way it turns out in practice. Many SLAs don’t actually help customers understand what to expect in terms of service delivery, and they don’t help providers with their planning either. In fact, the worst SLA’s I have seen set out a list of numbers that relate to technical targets that customers don’t really understand, and that are, in any case, not relevant to their business needs; and then the service provider works to achieve these numbers, regardless of the impact on their customers.

I had a personal experience of this some years ago, in a very different context. I had to take one of my children for treatment at a dental hospital. We researched local hospitals and selected one that had low waiting times for appointments, as well as good outcomes. Our doctor wrote to the hospital to request an appointment. Two weeks later I phoned the hospital to ask why we hadn’t heard and they said that the letter hadn’t arrived yet, so I spoke to the doctor who sent a second letter to the hospital. When the hospital told me that this second letter hadn’t arrived after a further two weeks, I got the doctor to write a third letter, which we hand delivered to the hospital. We talked to the receptionist at the hospital, who showed us a large box of unopened letters, and explained that this was the box of “letters that haven’t arrived yet”.

Letters in a box

The hospital had a metric of “Number of days from receiving doctor’s letter till patient has an appointment”, but letters didn’t count as having arrived until they were opened and date stamped. So the hospital achieved its target by keeping letters in a box until they had the capacity to process them.

This may seem like an extreme case, but I have seen similar things with IT metrics too. One outsource contract included metrics:

  • 98% of P2 issues will be resolved within 8 hours
  • 95% of P3 issues will be resolved within 24 hours

There were significant penalties for failing to meet these targets, so the service provider worked very hard to meet them. One month a senior manager sent an email to service desk staff saying that the target for P2 (highly significant issues) had been breached that month, and that the P3 target (somewhat less important issues) was also at risk; to avoid breaching both targets staff should ignore any P2 issues and focus on the P3 ones. You can imagine how the customer felt when someone showed them this email!

We all know that we need to have SMART KPIs in our agreements, so we focus on making sure they are specific, measurable, achievable, relevant and time-bound; but the thing we forget is that KPIs influence behaviour, and, as Goodhart’s Law tells us, they may drive behaviour that we really don’t want. Each of my examples happened because people treated the things that were being measured as if they were the targets, and forgot to consider the fact that they were performance indicators. To avoid this, we need to remember that the measurement IS NOT the target, it is just an indicator.

I would be the last person to discourage measurement per se. Measurement is essential. It tells us about trends,it helps us to identify improvement opportunities, and it provides a framework for discussion with customers. But we must never forget that achieving the KPI is not an end in itself, and that our work is not done until we meet our customers’ expectations.

So it’s fine to use KPIs for planning, understanding, and discussion, just remember to maintain a focus on your real goals and don’t confuse them with your KPIs!

comments powered by Disqus

Optimal Service Management Ltd.

7 Ingatestone Road, Woodford Green,
Essex, IG8 9AN, UK

Registered No: 8791379 England

Phone: +44 20 8504 2002

Recent Posts

  • 2018 01 02 Incident Management Isnt Just For IT Image
    Incident Management Isn’t Just For IT

    If you define an incident as "... interruption to an IT service..." then you're not really focussing on customers. One of my clients has a much more business-focussed definition, and this drives completely different behaviours...

  • 2017 11 28 Managing IT is just like running a barbershop IMAGE
    Managing IT is just like running a barbershop

    Delivering services isn’t something that is unique to IT. We know good service when we receive it, and so do our customers. If you focus on service levels and processes your customers will never see you as more than an internal cost centre; if you focus on outcomes and customer experience instead then you can become a valued partner.

  • 2017 11 16 You Need a Vision IMAGE 1
    You Need a Vision

    Many IT organizations think that defining a vision is too abstract, they don’t see the point, and they just want to get on with it. So they jump straight into the important work. The results are as predictable as they are sad. IT staff work very hard, but fail to arrive anywhere helpful, with results that are not nearly as good as they could have been.

Latest Tweets