Smaller input, greater insight

In the previous article, I explained the basics of property-based testing. If you have not read it yet, I recommend doing so before reading this article, as it forms the basis for the concepts that follow.

With property-based testing, we can go beyond individual examples and express general invariants about the behaviour of a system. Instead of manually selecting a few inputs and their corresponding outputs, the test framework automatically generates numerous inputs and checks whether the property holds for all of them.

Minimal counterexamples

This follow-up focuses on a second ingredient of property-based testing that is equally important but often receives far less attention in introductory material: shrinking.

Shrinking turns "some random failing input" into a small, understandable counterexample. This is why property-based tests can detect subtle errors and still produce failure messages that humans can understand.

When a property-based test fails, there are typically two phases:

• Generation: The framework draws random values from the generators you defined, looking for an input that falsifies the property.
• Shrinking: Once a failing input has been found, the framework attempts to simplify it step by step while keeping the property failing, until it reaches a minimal counterexample.

Without shrinking, a property-based test might tell you that your code fails for [-12, 5, -7, 3, 19, -2], which is technically useful, but difficult to understand. The bug could be as simple as "the property is wrong because negative numbers exist", yet the failing input appears random.

With shrinking, the very same failure will often be reduced to something like [-1].

Suddenly, the problem becomes obvious: the assumption that "sums are never negative" is simply false. The smaller the counterexample, the easier it is to understand what is actually happening. This is the essence of shrinking: it removes complexity from failing inputs until only the core of the bug remains.

Although each property-based testing library implements its own shrinking strategies, the underlying concept is similar. For each generator, there is a corresponding shrinker. When a failing input is found, the framework asks each generator how to "make this input smaller", tries those smaller variants, and keeps any variant that still falsifies the property. This process repeats until no smaller failing variant can be found.

Therefore, a well-designed set of generators is not only about the values that can be produced, but also how those values are reduced. Eris comes with sensible defaults for common types, enabling you to benefit from shrinking immediately without having to configure anything.

Shrinking in action

To see shrinking in action, consider a deliberately flawed assumption: the sum of a collection of integers is never negative.

This is clearly incorrect as soon as negative numbers are introduced, but consider how this might reflect an erroneous belief in a domain model, for example, "account balances can never fall below zero". Property-based testing is very effective at challenging such assumptions.

The following is a complete, minimal PHPUnit test case using Eris that implements this incorrect property.

<?php declare(strict_types=1);
use Eris\TestTrait;
use PHPUnit\Framework\TestCase;
use function Eris\Generator\int;
use function Eris\Generator\seq;

final class SumIsNeverNegativeTest extends TestCase
{
    use TestTrait;

    public function testSumOfIntegersIsNeverNegative(): void
    {
        $this
            ->forAll(seq(int()))
            ->when(
                static function (array $numbers): bool
                {
                    return count($numbers) > 0;
                }
            )
            ->then(
                function (array $numbers): void
                {
                    $this->assertGreaterThanOrEqual(
                        0,
                        array_sum($numbers),
                        'Minimal input for which this test fails: ' .
                        var_export($numbers, true) . PHP_EOL
                    );
                }
            );
    }
}

Now, let me walk you through what happens when this test is executed:

1. Generation: Finding some failing case

   • int() generates random integers
   • seq() wraps these integers into arrays of varying size
   • when() filters out empty arrays

   Sooner or later, it will generate an array whose sum is negative, for example [-12, 5, -7, 3]. array_sum([-12, 5, -7, 3]) returns -11 which results in the failure of our assertion.

   At this point, a non-shrinking property-based testing framework would simply report that random array and stop.

2. Shrinking: Finding the simplest failing case

   Instead, Eris now invokes the shrinkers associated with the generators used:

   • For seq(), it tries to simplify the array by removing elements and simplifying the remaining element values
   • For int(), it tries to move towards integers with smaller magnitude (closer to zero)

Eventually, Eris concludes that [-1] is a minimal counterexample: any further simplification (for example, [] or [0]) would no longer violate the property. The failure reported to you will then refer to this small array, not the original random one. Running the test shown above yields the (shortened) output shown below:

There was 1 failure:

1) SumIsNeverNegativeTest::testSumOfIntegersIsNeverNegative
Minimal input for which this test fails: array (
  0 => -1,
)

Failed asserting that -1 is equal to 0 or is greater than 0.

Even without looking at the random input that was found first, the minimal counterexample immediately tells you what the actual problem is: the property itself is wrong because negative numbers exist.

Shrinking has distilled the failure into its essential form: "an array containing a single negative integer".

Conclusion

The previous article emphasised invariants and generators as the key ideas behind property-based testing. Shrinking is the other half of the story:

• Generators explore wide swathes of the input space
• Properties tell the framework what must always hold
• Shrinking makes the inevitable failures understandable

Even without shrinking, property-based testing would still find bugs, but many of them would be obscured by large, complex inputs that are difficult to understand. However, with shrinking, property-based tests become sharp diagnostic tools. They do not just tell you that something is wrong. They show you the smallest possible way in which it is wrong.