by Luiz “Q” Quintela
Metrics is a subject that has a very polarizing effect. On an average Agile Development shop, there is always a lot of time, effort and money spent at gathering metrics. As a coach, you hear a lot of questions such as “how we can use metrics to improve our Agile roll out?” or “how we can use metrics to improve our Agile practice?” Basically, there are those who demand a series of metrics and targets and those that question the validity of metrics and what you are trying to achieve.
Consider that very often, metrics do not tend to drive the behavior we would expect.
I once worked with CTO that used lines of code as a metric to highlight his department’s achievements. What behavior are we driving here? If you were a developer and found a more efficient way to execute something, would you refactor 200 lines of code down to 30? If you are being evaluated based on lines of code, you would rather have as many as possible. Wouldn’t you?
A few months later, the same individual who by that time had attended the CSM and CSPO classes, wanted to motivate the teams by comparing the story points of one team against the others! Isn’t that a great idea? Of course, it is NOT! How many Scrum Masters and Agile Coaches had to “fight” against that? Or the tendency of management to come with targets for velocity? Just because we did 28 points instead of 24, it does not mean the team got any better.
I know of a large financial services organization whose governance department employed a very large team to come up with new metrics and visualizations for them. What behaviors surfaced from that? One of their dozens of metrics was called commitment reliability and it was used as a punitive metric. The behavior that it created you can easily guess. Teams either become near paralyzed trying to ensure exactness in their estimates or they start to “pad” their estimates just in case something goes wrong. Tell me how this aligns with Agile Manifesto? The part that says, “Individuals and interactions…”
Then I did “some digging” and the first thing I learned was that if the teams didn’t pad the numbers, the Scrum Master would. All to appear “green” in the dashboard because according to them “leadership only looks at the reds.” Further “digging” and I also found that the Scrum Masters eventually found out that the report was generated 3 days after the end of the Sprint.
Instant solution! GO fudge the numbers and make everything fit!
I am sure that you have your own examples of bad behavior driven by metrics. However, the message I am trying to drive, it is not that all metrics are bad. Metrics should not be used to replace thinking and you must use them properly. I am a firm believer that metrics can be used properly and effectively as an incentive to teams and leadership while following the three pillars of Scrum: Transparency, Inspection and Adaptation.
There are, however, several pitfalls to avoid. There is that “little thing” called velocity. Velocity can be based either on user story points or simply how many user stories were completed during a Sprint. Most teams also are mistakenly led to believe they must always increase velocity. Focus completely on velocity likely will lead to shortcuts, such as reduced testing, which in turn leads to decreased velocity later due to rework and fixes.
What a team must do is to focus on delivering quality, working software that has high business value, frequently, as it is stated in the Agile Manifesto. I always suggest reading about the Hawthorn Effect: what is measured will improve, at a cost, as well as Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. Metrics can be used as part of Kaizen, continuous improvement. Continuous improvement it is what you commit to. Things do not turn “perfect” instantly and when they get “perfect” they very likely will not stay that way over a period of time.
When properly applied, metrics are a terrific way to influence behavior, highlight areas of improvement and eliminate waste.
Before you decide on which metrics to adopt, it is important to consider:
How many metrics?
How long should we use them?
Who chooses which metrics?
What behavior are we trying to influence?
What are we trying to improve?
Also think about between metrics a team decides to use internally and metrics that have a broader audience. For example, a team may want to track the elapsed time to build and run unit tests to determine whether that process needs improvement. That metric is internal to the team and has a complete different objective than a business value burn-up chart, which has a broader audience. I am quite aware that there is always a certain reluctance in radiating metrics outside the team or the development department. When this is the case, remember the imperative for transparency! Scrum’s transparency will surface the issues with your organization and processes. It will not fix them, but it will show you what needs to be fixed.
Metrics should reinforce Agile principles. Metrics should also make people have conversations and to provide frequent and regular feedback. Be very aware of how metrics are used or perceived to be used. The misuse of a metric, often by leadership, is likely to promote the intentional manipulation of the metric, a dysfunctional behavior usually referred as “gaming.” See example a few lines ago…
Then there is the issue of productivity. How do you measure that? Most people will say that you don’t or can’t and I agree. Scrum and Lean tell us that the primary measure is value delivered, also referred as return on investment. ROI is a high-level measure that is not subject to “gaming.” You know how much you spent to develop that feature that nobody uses, don’t you?
Can you spell poor product management? I am going to talk about this in another blog entry.
Speaking of value, outcomes, the value created, are not measured in many companies. I read many studies that attribute that to the tendency to measure what is easy. An outcome can be, for example, to increase revenue, improve the usability of a product or high user satisfaction with the Android version of your product. These examples may appear obvious, but many organizations do not measure what is important!
When I run development departments, I tracked the business value results of a release and share them with the whole department. It is far more rewarding to the teams to see what impact the outcome of their work had for the company. It becomes clear why they worked on those features and it does wonders for team morale. When you share the release results, you will notice that the teams become engaged, focused and better at making decisions.
I can tell you one story where a developer got curious about a huge feature and the cost it was going to have to create it. Well, it turned out that very few people would have used it. After “some more digging” we determined that that very expensive feature would really only bring 20K in revenue for the year!
Speaking of poor product management!
Engaged, focused teams also produce a lot of discussion about the items in the product backlog. The example above is just one of the times I saw lots of items that had questionable business value being quickly excised from the backlog by teams. Incidentally, we also excised the bad product owner who came with that feature that had close no value…
I bet you that leadership will be onboard much easier when they see the results of measuring this way.
Before I conclude, the question I often get is what metrics I would suggest?
Obviously, I am going to say that is up to the teams and leadership together to decide but if you are looking for ideas, some of the following are a good starting point for discussion, in no order of importance.
Business Value Per Point
Time to Market
Return on Investment
Whatever leadership and teams decide to track and report, that should not be used to replace thinking!
Concentrate in radiating information and how it influences the outcomes so informed decisions can be made quickly and accurately. As the teams evolve and behaviors change, consider the “shelf life” of some of metrics and replace them. Focus on what behavior you want to encourage and keep the outcomes in mind, because as the Agile Manifesto says, “Working software is the primary measure of progress.”
Get to know Q here.
For more on Agile Metrics, explore these webinars: