Not every test, not just once, not one at a time

For most of its history, PHPUnit's test runner did one thing: load every test file, then run every selected test, once, one after another, in one process. Five changes to how tests are executed are undoing each part of that sentence.

A cache of which tests a file contains lets PHPUnit decide not to load a file at all. Test impact analysis records which source files each test depends on, so a change can select the tests it can affect. #[Repeat] runs a test until it fails, because failure is the information you are after. #[Retry] runs it until it passes, because sometimes failure is noise, while making certain that a tolerated failure is reported in every single run. And --parallel spreads a suite across worker processes while producing byte-for-byte the output a sequential run produces.

This presentation covers all five: what they do, what they cost, what they deliberately refuse to do, and why the architecture PHPUnit adopted in version 10 is what made them possible at all.

This presentation currently exists only as an idea and has not yet been accepted by a conference.

About me

I am the creator and maintainer of PHPUnit, the de facto standard testing framework within the PHP ecosystem. Used by millions of developers, it is embedded in the build pipelines of start-ups, Fortune 500 companies, and public sector organisations alike. I serve on the board of the PHP Foundation and am a co-founder of thePHP.cc, where I advise organisations on testing strategy, software architecture, and the performance of large test suites.

I have over 25 years of experience working with Open Source software, having contributed to the PHP language itself as well as the tools on which that ecosystem depends. Which test files have to be loaded, which tests a change can affect, how often a test should run, and how many processes may run tests at the same time are questions I have had to answer inside PHPUnit itself, with millions of test suites depending on the answer being conservative enough to be safe and bold enough to be worth having. That is what I talk about here: what these five changes cost, what they deliberately refuse to do, and why the architecture PHPUnit adopted in version 10 is what made them possible at all.