Property-Based Testing

In a previous article, I explained the concept of test oracles. These are procedures or sources that determine whether a test is successful or not. If you have not read that article yet, don't worry, you can still follow along.

This article focuses on how property-based testing can help solve the oracle problem in practice.

Revisiting the Oracle problem

Establishing correctness in software testing necessitates the involvement of an authority that is familiar with the anticipated behaviour. In traditional unit testing, this authority usually takes the form of explicit assertions about return values, state changes, or interactions with collaborating objects. We write tests that assert specific outputs for specific inputs: for example, when the shopping cart contains two items priced at five euros each, the total should be ten euros.

While this approach works well for known scenarios, it reveals a fundamental limitation: we only test what we think of to test. If we forget to write a test for negative quantities, empty carts, or free items, for example, bugs will remain hidden.

While code coverage tell us which lines of code were executed, as I argued when comparing path coverage to mutation testing, code coverage is a quantitative measure, not a qualitative one. High coverage provides no guarantee that our tests actually validate correctness.

Property-based testing offers a complementary solution, shifting the focus from specific examples to general behaviours. Rather than manually creating static input/output pairs, developers define invariants: universal truths about the system that must always be true, regardless of the input. The test framework then automatically generates test cases to challenge these properties, eliminating the need for hundreds of individual assertions.

The "Hello world" example

Let us look at a very simple function where we can easily see what properties look like: reversing an array. This example is featured in many tutorials because it clearly demonstrates the key concepts while remaining accessible to beginners. To keep things simple, rather than testing a custom implementation of reversing the elements of an array, we use PHP's built-in array_reverse() function instead.

Below is a comprehensive example of how to test array reversal using property-based testing in PHP with Eris, a property-based testing library for PHP that can be used with PHPUnit. Do not be put off by the large amount of example code, I will guide you through it step by step.

<?php declare(strict_types=1);
use function Eris\Generator\int;
use function Eris\Generator\seq;
use Eris\TestTrait;
use PHPUnit\Framework\TestCase;

final class ArrayReverseTest extends TestCase
{
    use TestTrait;

    public function testReversingTwiceIsIdentity(): void
    {
        $this
            ->forAll(seq(int()))
            ->then(
                function (array $input): void
                {
                    $reversed       = array_reverse($input);
                    $doubleReversed = array_reverse($reversed);

                    $this->assertSame($input, $doubleReversed);
                },
            );
    }

    public function testReversingPreservesLength(): void
    {
        $this
            ->forAll(seq(int()))
            ->then(
                function (array $input): void
                {
                    $reversed = array_reverse($input);

                    $this->assertSameSize($input, $reversed);
                },
            );
    }

    public function testArrayReversePreservesElements(): void
    {
        $this
            ->forAll(seq(int()))
            ->then(
                function (array $input): void
                {
                    $reversed = array_reverse($input);

                    sort($input);
                    sort($reversed);

                    $this->assertSame($input, $reversed);
                },
            );
    }

    public function testFirstElementBecomesLast(): void
    {
        $this
            ->forAll(seq(int()))
            ->when(
                static function (array $input): bool
                {
                    return count($input) > 0;
                },
            )
            ->then(
                function (array $input): void
                {
                    $reversed = array_reverse($input);

                    $this->assertSame(array_first($input), array_last($reversed));
                },
            );
    }
}

The ArrayReverseTest test case class shown above uses the Eris\TestTrait trait provided by Eris. This trait introduces methods such as forAll() that can be used to

1. Generate random inputs using generators
2. Exercise the system under test with each input
3. Verify properties that should hold for all outputs

Let us have a look in detail at the testReversingTwiceIsIdentity() test method.

public function testReversingTwiceIsIdentity(): void
{
    $this
        ->forAll(seq(int()))
        ->then(
            function (array $input): void
            {
                $reversed       = array_reverse($input);
                $doubleReversed = array_reverse($reversed);

                $this->assertSame($input, $doubleReversed);
            },
        );
}

We start by reading forAll(seq(int())) from the inside out:

• int() generates random integer values
• seq() generates sequences (numeric arrays) with a random number of elements
• seq(int()) generates arrays of random length that contain random integer values
• forAll(seq(int())) instructs Eris to run the test code (see below) once for each input generated using seq(int())

Finally, we use then() to specify the test code in the form of a callable. This callable is invoked for each input generated using seq(int()) (see above).

When you run this test, Eris will automatically generate arrays such as [], [1], [2, 5, 8], and many more besides, and verify that the double reversal property holds for each one.

Property-based testing shines when you verify multiple properties of the same operation. For our array reversal example, the following additional properties come to mind:

• The reversed array has the same size as the original array
• The reversed array contains the same elements as the original array, just in a different order
• The first element of the original array becomes the last of the reversed array

The other three test methods of the ArrayReverseTest test case class show how these properties can be tested.

The testFirstElementBecomesLast() test method introduces another method provided by Eris: when(). We use it here to "reject" randomly generated arrays that are empty by returning false.

Even with just a few properties, such as "double reversal preserves identity" or "first element becomes last element", we can create a robust oracle for array reversal without ever having to list specific input and output pairs.

A shift in perspective

A single property such as "the reversed array has the same size as the original array" implicitly tests an infinite set of inputs, limited only by the generator's capabilities and the time allocated for test execution. The test framework automatically generates hundreds of test inputs, long arrays, short arrays, empty arrays, and verifies that this property holds for all of them.

This shift from concrete examples to abstract properties addresses a fundamental weakness in example-based testing. We tend to test the cases we think of, which biases us toward common scenarios and away from edge cases. We test positive cases more thoroughly than negative cases. We think to test boundary conditions only after being burned by boundary bugs.

Property-based testing flips this bias by generating inputs we would not think to write, discovering edge cases automatically.

Properties act as partial oracles: they do not provide the complete expected output for every input, but define relationships and invariants that must always apply. Formulating such properties is therefore an oracle design task: you translate your understanding of correct behaviour into verifiable rules that the test framework automatically validates.

The key point is this: for many systems, it is difficult to determine the exact output for arbitrary inputs, but it is often easy to check basic relationships that must apply for correct behaviour. In some domains, it remains challenging to define good properties, because in doing so you encode your domain knowledge and assumptions about correctness in the form of verifiable conditions.

The oracle does not need to have complete knowledge of correctness. It is sufficient if it knows the invariants: properties that must always apply regardless of the specific result. This is precisely where the strength of property-based testing lies: instead of comparing results, you check principles that every valid result must satisfy.

Property-based testing shines in domains where invariants are straightforward to identify and express. Pure functions with clear mathematical or logical properties, such as date and time calculations, string transformations, or mathematical computations, are ideal candidates.

Code that is already deterministic and side-effect-free lends itself well to property-based testing, as does code where side effects can be isolated and mocked.

Conversely, domains with significant external side effects present challenges. For example, generating random inputs for network operations or file system interactions may be costly, destructive, or impractical when it comes to comprehensive testing. Similarly, highly stateful systems where describing invariants across state transitions is difficult require careful design of properties and may benefit more from targeted example-based or integration tests.

Conclusion

Property-based testing is an approach to testing that moves beyond the enumeration of manual examples. By replacing specific expected outputs with general invariants, property-based testing enables input spaces to be explored automatically while maintaining clear verification criteria.

In the next article, we will examine what occurs when a property fails, and how libraries such as Eris automatically reduce failing inputs to their simplest form. This transforms random test failures into actionable, reproducible bugs.

The oracle problem, determining whether a test should pass or fail, becomes tractable when the focus is on invariants rather than complete specifications. Roundtrip properties verify that inverse operations preserve data. Idempotence properties verify stability. Invariant properties ensure that essential characteristics are maintained during transformations. Metamorphic properties facilitate comparisons in domains where exact outputs are difficult to compute.

The Eris library in the PHP ecosystem makes property-based testing accessible and practical by integrating seamlessly with PHPUnit and enabling incremental adoption in existing test suites. When combined with mutation testing for quality assurance and example-based tests for documentation, property-based testing provides a sophisticated approach to testing that discovers bugs that example tests systematically miss.

Knowing when and how to apply property-based testing allows you to develop testing strategies that genuinely improve software quality, rather than merely satisfying metric targets. While the oracle problem may never be fully solved, perfect knowledge of correctness remains elusive, property-based testing provides powerful tools for verifying general truths about software behaviour across diverse inputs.