A video file recently turned up that I could make no sense of at first. Then it came back to me: two hours of conversation with Marco Pivetta and Stefan Priebsch, recorded in April 2022. I meant to do something back then with what I had taken away from it. Nothing ever came of it, until now.
I watched the recording again. The tools have moved on in a few places, and I come back to that below. The questions we circled around have stayed the same.
A question from practice
What set the conversation off was a thoroughly practical question from my daily work. Where does the responsibility of the tests end, and where does that of the type system begin?
Stefan was experimenting with path coverage in PHPUnit at the time and ran into difficulties. The presentation was hard to read. Even a look at the bytecode with VLD brought no clarity. At 100% line coverage and 100% branch coverage, question marks were still left in the path coverage. That is a clear sign that the familiar metrics do not show the whole picture.
This was where I put in that mutation testing ought to pick up. Marco had been working on it intensively and held a position I had not expected: code coverage was of no interest to him whatsoever. That was the moment I wanted to talk the topic through with the two of them properly. What path coverage and mutation testing each measure, I later set side by side in an article of its own.
Types first, then coverage, then mutations
Marco works in a niche he is visibly comfortable in: mutation testing together with static analysis. To him these are not separate tools; they reinforce one another. He calls PHP a "squishy" language, dynamic and with little built-in protection. For him that is the reason to go overboard with tooling.
His approach has three stages. Maximum typing comes first. To the objection that anyone who has types does not need tests, he responded in no uncertain terms: nonsense. Types do not replace tests. What they do is make the code rigid, and that is what matters to him. A function with complete types does one thing and does not wander off left and right with arbitrary variables.
Code coverage follows as a baseline metric. And the real question already arises here: is that coverage meaningful, or merely a by-product? The third stage answers it. Mutation testing does not check whether code was executed, but whether the tests actually check what they are supposed to check.
The tree and the fruit
Marco used an image for mutation testing that has stayed with me ever since. It is like a tree shake. You walk up to a tree and shake it until something drops. Then you know that this fruit was perhaps already a little too ripe.
The image carries in both directions, and the second direction gets overlooked more often than not, in my experience. A surviving mutant can mean that a test is too shallow. It can just as well mean that there is code in that spot which does nothing at all. Dead code, in other words, or a branch that more precise typing made unreachable long ago.
A simple example of this:
<?php declare(strict_types=1);
final class ShoppingCart
{
/**
* @var list<Item>
*/
private array $items = [];
public function numberOfItems(): int
{
return (int) count($this->items);
}
}
The tests for this class can reach full coverage. If Infection now removes the (int) cast, they stay green regardless. That is not a gap in the tests, because count() already returns an int. The cast never did anything. A test that kills this mutant cannot be written at all. The right response is to delete the cast and leave the guarantee to the return type.
This is the point at which the opening question answers itself. Where the return type makes the promise, neither a cast nor a test for it is needed.
Since PHPStan 2.3, there is a related kind of fruit for which the tree no longer needs shaking at all. PHPStan now reports values that are assigned but never read: an assignment that is overwritten before it is first read, a counter that is incremented but never used, a parameter that a private method ignores. These are exactly the places where mutants survive. When Infection turns $imported + 1 into $imported - 1, it changes a value nobody reads, and no test can notice. Until now, it took Infection's report to point at this code; now static analysis does so before a single mutant has been generated. The response is the same as for the cast: delete it. For the time being, the new rules are only active with bleeding edge.
Pointless mutations
The critical objection to this came from Stefan. He had observed that a lot of mutations are simply pointless. His example was a raytracer, a thoroughly mathematical project with matrices, vectors and rays. Many of the operations used there are commutative, so A + B = B + A holds. Mutation testing swaps the operands anyway and expects a test to notice. No test can notice, because the result stays the same.
Mutants like these cost computing time and depress the mutation score without anyone learning anything from them. The question that follows is: how does mutation testing get smarter?
Marco's answer to that was the Roave Infection Static Analysis Plugin, which he had already built by that point. It uses Psalm to check whether a mutant is possible at all under the rules of the type system. A mutation the type system rules out anyway does not also have to be executed. The plugin is still around, and it works. Its README does describe its own approach as decidedly rough and ready, though, because Infection had no interface for extensions of this kind back then.
This is where the most has happened since the conversation. Infection has shipped its own hook for static analysis since version 0.30. With --static-analysis-tool=phpstan, PHPStan runs over the mutants that escaped the tests and kills those that violate the declared types. The reports show which mutants were killed by the test suite and which by static analysis. A plugin that hooks itself into Infection is no longer needed for this.
Work has gone into the other side as well. PHPStan 2.1.34 cut the cost of starting up considerably in January 2026, and explicitly with use during mutation testing in mind. When a tool has to boot once per mutant, every second counts. That was only the beginning. According to the announcement of PHPStan 2.3, analysing the WordPress source code without a result cache now takes just under 8 seconds, down from 60 in December 2025. Without the PHPStan Turbo extension, which reimplements parts of PHPStan in C++, it takes just under 20 seconds. The extension ships with the Composer package and is enabled automatically on most installations.
What mutation testing cannot do
Marco told me about a customer working with CodeIgniter 2. For new code the team introduced a strict regime: a mutation score of at least 60%, complete typing, and exceptions to the rules only after discussion within the team. After a year they were writing remarkably clean and well tested code, in Marco's judgement. The discussions in pull requests had shifted from questions of craft to questions of architecture.
One aside struck me as remarkable. That same team had questions about individual mutation results every week. Mutation testing does not simply run by itself. It needs someone who can interpret the results and who has an overview of the architecture.
For regulatory requirements mutation testing is no use in any case. Path coverage is mathematically defined and can be verified, which is why it is mandated in safety-critical fields. The mutation score, by contrast, is a moving target. Mutations can be invented anew at any time, and the reference point shifts with every new version of a tool. A mutation score of 100% only means that one particular tool in one particular version produces no surviving mutants any more.
The number can be gamed too, only with more effort than code coverage takes. I told Marco about a developer who was dismissed over doctored coverage figures, because a bonus depended on them. Marco's take on that struck me as apt: anyone clever enough to game the mutation score has usually also understood what it is for, and then leaves it alone. What happens when a metric becomes a target I have described at greater length elsewhere.
Small components
Towards the end, Stefan put forward a proposition I kept thinking about afterwards. High figures in mutation testing are realistic only for very small components. With large codebases, combinatorial explosion makes the undertaking hopeless.
Marco agreed and described the architecture that follows from it. He was working on a talk about it at the time, which he gave shortly after our conversation at phpday 2022: How I sliced your app. The idea: every feature is represented by a small, self-contained component. Each of those components has its own branch coverage, its own mutation score and its complete typing. Above it all sits exactly one end-to-end test for the happy path. Dependencies then belong to individual features rather than to the whole application.
That reverses the order of things. Anyone who wants to talk about mutation testing has to have talked about component boundaries first.
Conclusion
Marco summed the conversation up best himself. Mutation testing, he said, is a superset of branch coverage and gets a little more out of it. Taken in isolation, though, it achieves little. It works as the topmost layer above small components, strict typing and branch coverage that has actually been reached, and it works considerably better when static analysis sorts out the pointless mutations beforehand. That last part now sits in the tool itself and still needed a plugin in 2022.
For me this stopped being someone else's position long ago. I have been working exactly the way Marco described it back then for quite a while now. Mutation testing is only one layer among several in that setup, and each of those layers catches different faults than the rest. How they work together, and why none of them is sufficient on its own, I have written up in Everything we have.
Four years on, what stays with me is mainly this: architecture matters more than tools, and no metric takes the decision about what is good enough off your hands. Aiming for 100% is not worth it. A threshold your team considers appropriate, used as a gate in CI, is the better way. What comes after that is manual work: shortening code, sharpening tests, refining types.
Software cannot be fully secured this way, and that was also the note our conversation ended on back then. It is about deliberate decisions rather than perfect numbers. The tools we have for it are unfinished, and they make our work better all the same.
