Your metrics are not your goals

I enjoyed reading Haider Rafique’s blog SLAs are for suckers: Why ITSM orgs should be allowed to be more human, so I tweeted a quote, with a link to the blog.

TWEET: SLAs drive bad behaviour

This is an example of Goodhart’s law in action. Goodhart was an economist and his law can be summarised in a few words. "When a measure becomes a target, it ceases to be a good measure." This is because when people know what is being measured, they adapt their behaviour accordingly, and the results are unlikely to be what was originally intended. And it’s a law applies that applies just as well to IT service management as it does to economics.

I was surprised at the amount of discussion my tweet generated. It quickly became clear that some people thought that I was advocating anarchy.  How could I dare suggest not using service level agreements (SLAs) to document services! Madness!

And, of course, they have a point. IT services are almost always documented in SLAs, precisely because SLAs can and should be a useful tool for helping to frame a discussion between service providers and their customers. With SLAs in place the customer knows what service they’re going to get, and the service provider can plan to put in place the resources needed to meet customer expectations.

Unfortunately, that’s often not the way it turns out in practice. Many SLAs don’t actually help customers understand what to expect in terms of service delivery, and they don’t help providers with their planning either. In fact, the worst SLA’s I have seen set out a list of numbers that relate to technical targets that customers don’t really understand, and that are, in any case, not relevant to their business needs; and then the service provider works to achieve these numbers, regardless of the impact on their customers.

I had a personal experience of this some years ago, in a very different context. I had to take one of my children for treatment at a dental hospital. We researched local hospitals and selected one that had low waiting times for appointments, as well as good outcomes. Our doctor wrote to the hospital to request an appointment. Two weeks later I phoned the hospital to ask why we hadn’t heard and they said that the letter hadn’t arrived yet, so I spoke to the doctor who sent a second letter to the hospital. When the hospital told me that this second letter hadn’t arrived after a further two weeks, I got the doctor to write a third letter, which we hand delivered to the hospital. We talked to the receptionist at the hospital, who showed us a large box of unopened letters, and explained that this was the box of “letters that haven’t arrived yet”.

Letters in a box

The hospital had a metric of “Number of days from receiving doctor’s letter till patient has an appointment”, but letters didn’t count as having arrived until they were opened and date stamped. So the hospital achieved its target by keeping letters in a box until they had the capacity to process them.

This may seem like an extreme case, but I have seen similar things with IT metrics too. One outsource contract included metrics:

  • 98% of P2 issues will be resolved within 8 hours
  • 95% of P3 issues will be resolved within 24 hours

There were significant penalties for failing to meet these targets, so the service provider worked very hard to meet them. One month a senior manager sent an email to service desk staff saying that the target for P2 (highly significant issues) had been breached that month, and that the P3 target (somewhat less important issues) was also at risk; to avoid breaching both targets staff should ignore any P2 issues and focus on the P3 ones. You can imagine how the customer felt when someone showed them this email!

We all know that we need to have SMART KPIs in our agreements, so we focus on making sure they are specific, measurable, achievable, relevant and time-bound; but the thing we forget is that KPIs influence behaviour, and, as Goodhart’s Law tells us, they may drive behaviour that we really don’t want. Each of my examples happened because people treated the things that were being measured as if they were the targets, and forgot to consider the fact that they were performance indicators. To avoid this, we need to remember that the measurement IS NOT the target, it is just an indicator.

I would be the last person to discourage measurement per se. Measurement is essential. It tells us about trends,it helps us to identify improvement opportunities, and it provides a framework for discussion with customers. But we must never forget that achieving the KPI is not an end in itself, and that our work is not done until we meet our customers’ expectations.

So it’s fine to use KPIs for planning, understanding, and discussion, just remember to maintain a focus on your real goals and don’t confuse them with your KPIs!

comments powered by Disqus

Optimal Service Management Ltd.

7 Ingatestone Road, Woodford Green,
Essex, IG8 9AN, UK

Registered No: 8791379 England

Phone: +44 20 8504 2002

Recent Posts

  • 2017 08 09 Goldilocks does security Three bears
    Goldilocks does security

    If you are responsible for security try to be like Goldilocks. Not too much security, not too little, but just right.

  • 2017 07 08 Recorded ITSM Presentations IMAGE
    Recorded ITSM Presentations

    I have often written about the need to offer access to knowledge and information in a variety of different formats, so I’m grateful to the people running these ITSM events for making video recordings available for sharing.

  • itSMF NZ
    DevOps and Culture Change in New Zealand

    I really enjoyed the itSMF New Zealand conference, the format and content were as superb as the venue, read on to learn more...

Latest Tweets