I know that both my readers are software developers, so – this is a bit of a departure.

Background

Within a business, there are many financial decisions to be made, and nothing kills a business as fast as costs quietly running away. This is why companies have bureaucracy, to make sure that those who enter into contracts on behalf of the company (primarily sales and procurement) are doing so with due skill and care.

Managers follow up on performance down on the individual level. Commission and bonuses reward top performers, career progression means you are judged by the average scores of your subordinates. If you miss budget you are held accountable. What went wrong? What changes are you implementing, why do you think those changes will make a difference?

CEOs are thinking- why is IT, and specifically IT software development so unwilling to produce metrics and show their work? Are they not monitoring their people? Surely they will want to know who their worst performers so that they can train them or ultimately get rid of them if all else fails?

In this environment, senior developers turned junior managers are treading lightly to try and explain to senior management the ways in which – compared to doing literally nothing – measuring incorrectly can cause much worse problems in terms of incorrect optimisations or alienating and losing the wrong individual contributors, but as far as I know, rarely have any inroads been made into fairly and effectively measuring individual performance in an IT department. You can spot toxic people, or people that plainly lied on their CV, but apart from clear-cut cases like that, there are so many team factors that affect the individual, that getting rid of people instead of addressing systemic or process-related problems is like cutting off your nose to spite your face.

Bad metrics are bad

What do we mean bad metrics? How about a fictionalised example of real metrics out there: LOC. Lines of Code. How many lines of code did Barry commit yesterday? 450? Excellent. Give the man a pay rise! How about Lucy? Oh dear… only 220. We shall have to look at devising a personal improvement plan and file with HR.

However, upon careful review – it turns out that out of Barry’s contribution yesterday, 300 lines consisted of a piece of text art spelling out BARRY WOZ ERE. Oh, that’s unfortunate, he outsmarted our otherwise bullet-proof metric. I know, we solve everything by redefining our metric to exclude comment lines. NCLOC, non-comment lines of code. By Jove, we have it! Lucy is vindicated, she doesn’t write comments in her code so she kept all her 220 lines and is headed for an Employee of the Month trophy if she keeps this up. Now unfortunately after this boost in visibility within the team, Lucy is tasked with supervising a graduate in the team, so they pair program together, sometimes on the graduate’s laptop and sometimes on Lucy’s, and because life is too short they don’t modify the git configuration for every single contribution, so half the day’s contributions fall under the graduate’s name in the repository and the rest under Lucy’s. The erstwhile department star sees her metrics plummet and can feel the unforgiving gaze from her line manager. Barry is looking good again, because he never helps anyone.

So – to the people on the factory floor of software development, the drawbacks and incompleteness of metrics for individual performance are quite obvious, but that doesn’t help senior management that have legitimate concerns for how company resources are spent, and want to have some oversight.

Good metrics are good

After several decades of this situation, leading experts in the field of DevOps decided to research how system development works in big organisations to try and figure out what works and what doesn’t. In 2016 the result came out in the form of the DORA metrics,  throughput (deployment frequency, lead time for changes), and stability (mean time to recover, change failure rate) were published in the State of DevOps report. This measures the output of a team or a department, not an individual, and the metrics help steer improvements in software delivery in a way that cannot be gamed to produce good metrics but negative actual outcomes. Again – the goal of the DORA metrics are to ensure that software design, construction, delivery continuous improvement is successfully undertaken, to avoid pitfalls of failed software projects and astronomical sums of money lost – measuring team or individual performance is not what it’s about.

In the post-pandemic recovery phase a lot of organisations are looking at all of the above and are asking for a way to get certainty and to get actual insight into what IT is doing with all the money, tired of the excuses being made by evasive CTOs or software development managers. How hard could it be?

A proposal, the blowback and a suggestion

Whenever there is a willing customer, grifters will provide a service.

McKinsey took among other things the DORA metrics and NCLOC to cook up their own custom metric to solve this problem once and for all.

Obviously, the response from software development specialists was unanimously critical, and the metric was destroyed up and down the internet, and I must admit I enjoyed reading or watching several eviscerations, especially one of the fathers of DevOps, Dave Farley, had a very effective critique of the paper on YouTube.

There was no solution though. No-one was trying to help the leaders get the insights they crave. There was limited understanding of why. Surely you must trust your employees, why else did you hire them?

Then I stumbled upon a series of writings from Kent Beck and Gegerly Orosz, trying to address this head on, to do what McKinsey hadn’t achieved, which I found so inspiring that this blog exists just as an excuse to post the links:

https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity
https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity-part-2

If you need to know why the proposed McKinsey metrics are bad, look at Dave Farley’s video, and if you what to know what to do instead, read the writings of Beck/Orosz.