Saving Time and Tokens with PHPUnit

What it is about

Slow tests are a defect. A suite that takes minutes runs less often than it should. Bugs surface later, and confidence in the tests drops.

In this training you measure where your test suite spends its time and work your way through the levers: configuration, test design, test selection, parallelisation. Each stage becomes cheaper and more effective once the previous one is done. Along the way you get to know the new capabilities of PHPUnit's test runner: the test index cache, test impact analysis, #[Repeat], #[Retry], and --parallel. You also learn what they cost and what they deliberately do not do.

A coding agent runs your tests even more often than you do: before a change, after it, and after every attempt at a repair. Every output ends up in its context window, costs tokens, and displaces code and task there. PHPUnit's default output is made for a human who skims over what they do not need. An agent reads every character. You set PHPUnit up for this reader, and in a way that does not depend on whether the agent remembers to use it.

How you work

The first four sessions follow a formula: feedback time ≈ overhead + (number of tests executed × cost per test) ÷ parallelism. Each of these sessions works on one term, on an exercise project that is deliberately slow and has built-in problems for every term. A series of measurements shows what each change achieves.

In sessions 5 and 6 the reader changes: instead of a human waiting for the suite, an agent reads its output. What you learn there about PHPUnit applies to every coding agent. How to anchor it in your project is shown with Claude Code and Junie, at the level of the three mechanisms of instruction, environment, and guardrail, because agent-specific details change quickly. Every session consists of demonstration, exercise, and discussion, without slides.

Programme

1. 1

   Session 1: Measuring and configuring (overhead)

   Where does the time go, and how do you know?

   • What slow tests cost: a worked example, flow, TDD, and trust in the suite
   • The shape and size of tests: function, size, and implementation, and the test pyramid as a performance budget in milliseconds, seconds, and tenths of a minute
   • Measuring instead of guessing: --log-otr with phpunit-otr-report, --order-by=duration-descending with recordTestRunHistory, profiling and tracing where needed
   • The first stage, configuration: global process isolation, code coverage as opt-in, --order-by=defects, and for every other switch the question of whether it treats a symptom or a cause

   Result: A baseline of the exercise suite, its slowest tests as a ranking, and a cleaned-up configuration, measured again

2. 2

   Session 2: Test design (cost per test)

   Which tests are more expensive than they need to be?

   • Integration tests in disguise that hit a database and call themselves unit tests
   • Fixtures that rebuild the world before every test, and unnoticed access to the file system and the network
   • Overloaded mock setups and framework bootstrapping for pure functions
   • Remedies: lazy fixtures, in-memory databases, data providers
   • Test sizes as a contract: #[Small], #[Medium], and #[Large], time limits with enforceTimeLimit, and small tests without network, database, file system, or clock

   Result: The most expensive tests from the first session, reworked so that they cost less without checking less, and the continued series of measurements

3. 3

   Session 3: Not every test, not just once (number of tests executed)

   Which tests need to run now, and which ones how often?

   • Test index cache: not loading test files in the first place
   • Test impact analysis: which tests a change can affect at all, and what the analysis deliberately does not promise
   • --repeat and #[Repeat]: running a test until it fails, to diagnose tests that are unreliable or depend on the order
   • --retry and #[Retry]: running a test until it passes, with every tolerated failure reported; a tool for the time between the finding and its cause
   • Causes of unreliable tests: shared state, time and randomness, concurrency, external systems, time zones
   • Cadence: test suites and groups for commit, merge, nightly, and release, in the configuration file and through command-line options

   Result: The hidden unreliable tests of the exercise suite, found with --repeat, a justified retry policy, and a cadence that defines which tests run when

4. 4

   Session 4: Not one after another (parallelism)

   What has to be right before parallelisation helps?

   • Why parallelisation comes fourth: otherwise the slowness runs in parallel, and shared state makes tests unreliable
   • --parallel and its guarantee of byte-identical output, compared with ParaTest
   • Database isolation: transaction rollback, template databases in PostgreSQL, one container per worker
   • Infrastructure: RAM disk, caching for Docker and Composer, splitting into fast and slow groups, sharding in CI
   • Why only the event architecture PHPUnit has had since version 10 made the index cache, test impact analysis, repeat, retry, and --parallel possible

   Result: The parallelised exercise suite without the bug that shared state only reveals then, the completed series of measurements, and an action plan for your own suite

5. 5

   Session 5: Output for coding agents

   What does a test run cost in the context window, and how does the agent recognise the result?

   • The context window as working memory: what the default output costs there
   • --compact and PHPUNIT_COMPACT_OUTPUT: errors and failures immediately, everything else after the summary
   • The --display-* options as targeted questions, not as a reflex like --display-all-issues
   • Why | tail and | grep lose content and exit code
   • Why OK in the summary is not a verdict: failOn* and the exit code

   Result: A test run whose output grows with the problems instead of with the tests, a comparison of the context consumption of an agent with and without compact output, and the exit code as the verdict

6. 6

   Session 6: Querying coverage selectively and anchoring it

   How does an agent name the gap instead of optimising a number?

   • Coverage as opt-in: which tasks need it and which do not; --covers selects the responsible tests without a coverage driver
   • --coverage-jsonl with meta.json, coverage.jsonl, and tests.jsonl, queried with grep and jq
   • Restricted runs with --filter and what “uncovered” means then, Covers* attributes, and branch coverage that reports partial although every line was executed
   • A line that was not executed as a question rather than a target
   • What is only in the prompt, the agent forgets. Three mechanisms record it: instruction (AGENTS.md, CLAUDE.md, Junie guidelines), environment (environment variables, Composer scripts, approved commands), and guardrail (hooks that reject filtered test runs)
   • Checking the report of the agent instead of believing it

   Result: Queries against coverage in JSONL format that return the gap instead of a percentage, and your own project, set up with instruction, environment, and guardrail

Who it is for

• Developers whose test suite is so slow that it runs less often than it should
• Teams with unreliable tests that are sometimes green and sometimes red
• Tech leads who are responsible for test execution in CI
• Developers who let coding agents such as Claude Code or Junie run tests

Prerequisites

• Confident practice with PHPUnit, for instance after “Agentic PHPUnit”; “Understanding Testing” teaches test isolation and test doubles as concepts
• Ideally a test suite of your own that is too slow for you
• For sessions 5 and 6, experience of your own with a coding agent; you cover the cost and usage quota of your agent yourself
• For the exercises, PHP 8.4 or newer, Composer, and Git, or Podman or Docker for the container we provide
• For working on your own project, ideally PHPUnit 14, at least 13.4

What you take away

• You establish a baseline of your suite and know which tests consume the time
• You assign each measure to a term of the feedback formula and choose the right order
• You find unreliable tests with --repeat and justify when #[Retry] is acceptable
• You parallelise a suite without shared state making it unreliable
• You set up your project so that a coding agent tests compactly, reads the result from the exit code, and names gaps in coverage instead of optimising a percentage
• You have an action plan for your own test suite

Related training

This training is part of the learning paths Testing with PHPUnit, Working with AI agents.