In my article “When the metric becomes the target”, I wrote about Goodhart's Law and what happens when code coverage is turned from an indicator into a goal. While writing it, I kept thinking about another metric that has been part of PHPUnit's code coverage functionality for so long that many developers assume it was always there: the CRAP index. Its story is worth telling, not least because it illustrates rather well how I think about metrics.
Origin Story
In 2007, Alberto Savoia and Bob Evans were working at Agitar Labs on the question of how to identify code that is risky to change. Their answer combined two measurements that were each well established on their own: cyclomatic complexity, which counts the number of linearly independent paths through a unit of code, and code coverage, which measures how much of that code is executed by automated tests.
The result was the Change Risk Analysis and Prediction index, CRAP for short, a name that was clearly chosen backwards from the acronym. For a method m with cyclomatic complexity comp(m) and coverage cov(m), expressed as a fraction between 0 and 1, it is defined as
CRAP(m) = comp(m)² × (1 − cov(m))³ + comp(m)
The exponents are the interesting part: complexity is squared, the uncovered fraction is cubed. A fully covered method scores its bare complexity; an untested complex method scores its complexity squared. Complex code is acceptable, the formula says, but only when it is thoroughly tested. Savoia and Evans proposed 30 as the threshold above which a method should be considered, well, crappy.
They introduced the metric in a series of blog posts over the summer and autumn of 2007, culminating in the announcement of Crap4j, an experimental Eclipse plug-in that calculated CRAP scores for Java code. They were refreshingly honest about its status: the metric was explicitly labelled experimental, and the numerical values in the formula were described as a starting point to be refined with experience.
CRAP in PHPUnit
I liked the idea immediately. On 6 August 2007, I implemented the CRAP calculation in PHPUnit's software metrics code, and PHPUnit 3.2.0, released on 19 November 2007, was the first version to ship it. That was before Crap4j itself had properly launched. The PHP ecosystem was an early adopter.
When I extracted the code coverage functionality from PHPUnit into what is now the php-code-coverage library, the CRAP index came along. In December 2009 it found its way into the Clover XML logfile and the HTML code coverage report, where it has lived ever since. When the metric was renamed from “Change Risk Analysis and Prediction” to “Change Risk Anti-Patterns” in 2010, I followed suit, and in 2013 a community contribution added support for the Crap4j XML report format.
The refinement that never happened
Remember that the metric was announced as a starting point to be refined with experience. That refinement never happened. Crap4j was abandoned after a few years, and to this day I am not aware of a single peer-reviewed empirical study that validates the CRAP index: nothing that correlates CRAP scores with actual defects or with maintenance effort, nothing that justifies the exponents in the formula or the threshold of 30. The formula that php-code-coverage calculates today is, character for character, the experimental formula of 2007.
What the research literature does offer is evidence about the two ingredients, and it is sobering. Cyclomatic complexity correlates so strongly with lines of code that several studies conclude it measures little more than size: a method with many paths is, above all, a long method. And code coverage correlates only weakly with the ability of a test suite to actually detect faults. CRAP combines two proxies whose individual validity has been called into question, and the combination itself was never validated at all.
A heuristic, not a quality measure
Why, then, has the CRAP index been part of PHPUnit's code coverage report for almost two decades? Because I have never treated it as a quality measure. For me it is a prioritisation heuristic, and in that role it has served me well.
When you open a code coverage report for a legacy codebase, what you need to know is where to start. Sorting methods by their CRAP index tells you exactly that: it points you at the code that is both complex and untested, which is where changes are most likely to go wrong and where new tests buy the most safety. You lower a CRAP score by writing tests, by refactoring to reduce complexity, or preferably both, with the tests written first so that the refactoring is safe. That is sound advice regardless of whether the exponents in the formula are empirically justified.
What the CRAP index must not become is a target. Everything I wrote in “When the metric becomes the target” about code coverage applies to the CRAP index with full force, because coverage is one of its two inputs. A team that has to keep all CRAP scores below 30 will get them below 30, and Goodhart's Law tells us how: with tests that execute the complex code without really verifying its behaviour. The scores drop below 30, and the risk stays exactly where it was.
Necessary, but not sufficient
This brings me to how I think about code coverage itself. For me, code coverage is a necessary but not a sufficient criterion for test quality. Code that is not executed by any test is trivially untested. High coverage of the code I care about is therefore a precondition for trusting my test suite. But coverage only tells me that my tests reach the code, not that they would notice if the code were wrong.
That second question is answered by mutation testing. A tool such as Infection makes controlled changes to the code that simulate typical programming mistakes and checks whether the tests notice. In “Path Coverage or Mutation Testing?” I described in detail how this works and why it is part of my daily work. Only the combination of the two gives me confidence: code coverage shows that my tests reach the code, mutation testing shows that they would fail if the code misbehaved. Incidentally, mutation testing is also the far better researched of the two ideas: the validity of mutants as stand-ins for real faults has been studied empirically in a way the CRAP index never was.
So the CRAP index remains where it has been since 2007: in the code coverage report, sorting my attention, never grading my work. A nineteen-year-old experimental formula makes for a poor quality gate, but as a pointer to the code I should look at next it has aged remarkably well.
