Testing Beyond Examples

What it is about

Coding agents generate code and tests faster than ever. The tests are green, the code coverage is high. But what about the cases that neither you nor the agent thought of? And would these tests even notice a defect?

Example-based tests verify a known output for a known input. That works as long as someone thought of all the relevant cases. This training goes beyond that with three techniques:

• Properties instead of examples. Property-based testing checks statements about all inputs and finds edge cases you would never have written down.
• Deliberate defects. Mutation testing builds defects into your code and shows you which of them your tests do not notice.
• Unexpected input. Fuzzing confronts your code with input nobody expected and finds where it crashes.

You learn property-based testing with the new support in PHPUnit 14, from the person who designed it. You learn fuzzing with rattle, Sebastian’s fuzzer for PHP, which fuzzes the properties of a PHPUnit test suite as they are. In the last session you apply the techniques to your own code, combine them, and compare notes with the others: what worked where, and what did not?

The techniques are not tied to PHP. Sebastian teaches them using PHP examples with the tools of the PHP ecosystem, and you take the insights with you into any codebase, whether PHP, C#, Go, Java, Python, Rust, or TypeScript.

How you work

Each session starts with a short demonstration; for most of the time you practise yourself, alone or in pairs. Three classes with deliberately planted bugs run through the first five sessions: IniParser, DiscountCalculator, and ListCodec. Their tests are green, their coverage is almost complete, and yet each technique finds something the others miss. The surviving mutants in ListCodec point, in the third session, to the bug that property-based testing found in the first, and the round-trip property for IniParser from the second session becomes the oracle for the fuzzer in the fifth.

Each session closes with a look beyond PHP and names the corresponding tools of other languages.

Programme

1. 1

   Session 1: Property-based testing: what nobody thinks of

   What do your examples miss?

   • Checking an example or making a statement about all inputs: data provider or property?
   • A property is a test method with #[Property]: PHPUnit generates its arguments from the types of its parameters, including PHPDoc types such as list<int> or non-empty-string, enumerations, and objects created through their constructors
   • Each example is a complete test with its own setUp(), its own test doubles, and its own expectations
   • Shrinking: from a random to a minimal counterexample, which PHPUnit prints as PHP code
   • Reproducing: the seed of the test run with --random-seed, remembered counterexamples, and the counterexample as a regression test with #[TestWith]
   • On ListCodec in three acts: a hole in the specification, a real bug, the fix

   Result: A round-trip property for ListCodec that finds a bug, and its counterexample as a regression test that reproduces the failing run

2. 2

   Session 2: Finding properties

   Which statements hold for all inputs of your code?

   • Patterns for properties: round trip, invariant, idempotence, monotonicity, reference oracle, and metamorphic relations, also as a way out of the oracle problem when nobody knows the right answer
   • When the type does not say enough: generators of your own, generators for domain objects that must not be created through their constructor, and dependent values
   • Preconditions, and why many discarded examples are a warning sign
   • Checking what is generated: statistics that show whether the examples hit the interesting cases
   • State: a state machine compares the system with a model that is as simple as possible after every step
   • Limits and costs in everyday use: side effects, slow functions, the number of examples, more examples for the nightly run, and CI runs that produce the same result when repeated

   Result: A monotonicity property for DiscountCalculator with a generator of your own and statistics on whether its examples hit the interesting cases, and the round-trip property for IniParser that the fifth session builds on

3. 3

   Session 3: Mutation testing: the blind spot

   Would your tests notice a real defect?

   • The coverage paradox: 100 % coverage, and still every boundary error goes undetected
   • How mutation testing works: mutation operators, one mutant per run, the test suite as the judge
   • Terms: killed, escaped, timeout, uncovered; Mutation Score Indicator (MSI) and covered MSI
   • Running Infection with PHPUnit, reading its console output and its report
   • Killing mutants, for instance at boundaries and in rounding: a mutant dies from a sharper expectation, not from another test run
   • Back to ListCodec: its surviving mutants point to the bug from the first session, which the example-based tests would not have noticed

   Result: A run of Infection on IniParser whose output you read, and tests that kill its surviving mutants

4. 4

   Session 4: Mutation testing in everyday work

   Which mutants are worth it, and how does mutation testing get into CI?

   • The last survivor: an equivalent mutant points to a design decision
   • Kill it, exclude it, or change the design: which mutants are not worth it, and how you exclude them deliberately and visibly
   • The MSI as a metric: trend instead of threshold, covered MSI instead of MSI, and why 100 % is not a goal
   • Run time: selecting tests via coverage, parallelisation, and a fast suite as a prerequisite
   • Mutating only changed code in CI, with --min-msi as a gate and the full run at night
   • Mutation testing as a touchstone for generated tests: how many mutants does the suite that an agent wrote kill?

   Result: A decision for each survivor you bring along from your own codebase, an equivalent mutant as feedback on the design, an MSI policy, and a CI configuration that mutates only changed code

5. 5

   Session 5: Fuzzing: what nobody expects

   Is your code robust, too?

   • Correct is not the same as robust: defined failure modes instead of warnings
   • How coverage-guided fuzzing with rattle works: corpus, mutation, feedback
   • The target as a contract about failures: expected exceptions with expect(), the maximum length of the input, and the dictionary that rattle derives from the comparisons in your code
   • The triage pipeline: rattle reproduces and minimises every finding, you fix the defect, and rattle regress replays the finding in CI as a regression test
   • Fuzzing and property-based testing: rattle fuzz --phpunit fuzzes the round-trip property from the second session as it is, and a finding has the same failure signature as a counterexample of PHPUnit
   • The security perspective: fuzz everything that touches bytes from outside, and path traversal, code injection, command injection, and unsafe deserialisation, which rattle reports with the input as proof

   Result: A fuzzing target for IniParser and its crashes, found, minimised, and recorded as regression tests until the fuzzer runs for 30 seconds without a finding

6. 6

   Session 6: On your own code: combining and learning from each other

   Which technique finds what your tests miss in your code?

   • The three techniques as a decision map: properties generalise, mutation testing measures the suite, fuzzing checks against reality
   • Questions for choosing: can a property be formulated? How good is the existing suite? Do bytes come from outside?
   • Combinations: a property as the oracle for the fuzzer, mutation testing that checks reproducibly with a fixed seed whether properties kill mutants that the examples let survive, and counterexamples and fuzzer findings as regression tests and as a starting corpus
   • Cost and integration per technique: run time, runs in CI, reproducibility
   • About an hour of work on your own codebase, with the tool of your language, while the trainer supports you individually
   • A round of reports: what did you apply and find, what did not work, and which technique proved itself for which kind of code?

   Result: At least one technique, ideally a combination, applied to your own code with a justified choice, the experiences of the others, and a plan for which technique you use next on which piece of code and how it gets into your CI

Who it is for

• Everyone who already has experience with automated tests and wants to go further
• Developers whose test suite is green and who still see bugs in production
• Teams who let coding agents generate tests and want to know whether these tests check anything
• Developers who do not work in PHP, because the techniques are language-independent

Prerequisites

• Experience in software development with an object-oriented programming language
• Experience with automated tests: you write and run unit tests. “Understanding Testing” teaches the terms
• Basic knowledge of PHP for the exercises of the first five sessions
• For the exercises, a local environment with PHP 8.5, or Podman or Docker for the container we provide; the invitation describes the setup
• A codebase of your own with tests that you may work on in the last session, in another language if you like. For the last session it needs to run, and you install the tools of its language beforehand

What you take away

• You formulate properties and find edge cases you would never have thought of
• You turn a counterexample you found into a regression test
• You know what code coverage tells you and what it does not, and measure instead whether your tests check anything
• You work through surviving mutants and recognise when a mutant points to a design decision
• You systematically check the robustness of your code against unexpected input
• You have tried out and combined the techniques on your own code

Related training

This training is part of the learning path Testing with PHPUnit.