Let’s measure the things that really matter

Let’s measure the things that really matterAs technologists, we tend to collect very detailed information and present it in a very technical way.  We talk about things like process maturity levels, and productivity indexes. I have even seen some measurement data about Agile maturity models that attempt to show how agile we are.  The problem with this kind of approach is it doesn’t provide anything that the business and the customers can relate to. 

The business does not really care how agile we are, they want to see results.  They want to be able to set targets, see improvements, and understand the benefits and the quality of the results being generated by IT. Only then can they begin to put the value of Agility into context – the context of the value and improvement generated by the move to agility.

I’ve seen this conundrum many times.  At IJI, we get involved in many Agile transformations –very large scale Agile transformations.  One of the key enablers for success is to be able to demonstrate that things are getting better.  Not just better in the short term, during the first flush of enthusiasm when the A players are adopting the new techniques, but in the long term as we continue on our journey of continuous improvement - seeing improvements in every quarter, every six months, and every year.  Large scale Agile transformations are not things that happen in a few weeks.  You start to put things in place and then you continue to improve, inspect and adapt as you go forward.

You need a way to make visible where you are and where you are going.  You need to do it in a way that everybody understands. 

What we see is that most measurement programs do not really achieve those kinds of goals.  A lot of them add very little value.  People look at the wrong things. They have lots of numbers, lots of quantitative measures but very few qualitative measures. They tend to measure the things that are easy to count rather than the things that are important. They tend to collect secondary rather than primary measures. For example rather than measuring, and communicating, the quality levels of the software produced, they will tell you the defect detection rate. Rather than measuring the effect of agility they will measure the number of people who think they’re agile. This leads to the classic question being asked by the business; “OK, now we’re all agile what difference has it made?”  

There is also too much focus on short-term rather than long-term measures. Remember, the transformation is likely to take more than a few months and you need to be able to show continuous improvements and not just the initial impact of introducing the new ways-of-working.

You need to find a set of measures where things are very intuitive; where everyone can understand the measures without having to have a training course or deep knowledge of IT. You need simple measures where the direction of improvement is clear. The best measures are simple and easy to understand.  They are not complex derived measures; they are not indexes that take into account project length, cost over-runs and things like that.  They are not measures based on theories that are not based on anything.

Let’s measure the things that really matter.  Let’s look at the effects, as well as the causes.  Let’s make sure we have evidence that Agile is a better, faster, cheaper, and happier way of working than what has gone before.

2 Comments
  1. Philippe Back | September 9, 2010 at 4:28 pm Reply

    avatar

    Ian,

    I am curious to see some examples of measures you find useful in a context or two.

    Keep up the good work!