Security through chaos

In a previous article, I showed how property-based testing can improve our testing. While we define invariants and general truths about our system there, we now change our perspective: We become the adversary of our own software.

In fuzz testing, also known as fuzzing, our code is deliberately tested in such a way that it crashes. Instead of “This behaviour should always be true” as in property-based testing, in fuzzing we ask: “In what situation can we cause this software to fail?”

It is interesting to note that property-based testing and fuzzing appear completely different at first glance, but in fact share similarities. At the same time, they are optimised for completely different problems. In this article, I will explain which testing method is suitable for which situation.

What is fuzz testing?

When building software, we usually write unit tests to ensure that our code handles the expected cases. We may also use mutation testing to verify that our tests are robust enough to detect logic errors. However, there is a class of bugs, the “unknown unknowns”, that often slip through, such as malformed inputs, edge cases in library dependencies, or unexpected environmental behaviours.

Fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as input to a software system, either as a whole or in part, such as a subsystem, component, class, or method. The software is then monitored for exceptions such as crashes, failing built-in code assertions, or memory leaks. Unlike unit testing, which tests for correctness against known inputs, fuzzing tests for robustness against an infinite sea of unexpected inputs.

Let's crash the CSV party

I created a small library called csv-parser a while ago for type-safe parsing of data from CSV files into PHP variables. This library is well tested, with 100% line and branch coverage, over 70% path coverage, and a covered code mutation score of 100%. Nevertheless, fuzz testing uncovered an error in the parser's code.

Let us take a closer look at how fuzz testing can be implemented and examine the “garbage” data that caused the CSV parser to malfunction.

I used the PHP-Fuzzer library developed by Nikita Popov to implement fuzz testing. PHP-Fuzzer is a coverage-guided fuzzer for PHP. It works by:

1. Instrumentation: It modifies the source code to track which parts of the code are executed.
2. Corpus-Based Evolution: It starts with a corpus of valid inputs (like our existing CSV fixtures).
3. Mutation: It randomly mutates these inputs (flipping bits, adding characters, etc.).
4. Feedback Loop: If a mutation reaches a new part of the code, it's saved to the corpus to be mutated further. If it causes a crash or an uncaught exception, it's reported as a bug.

We need a PHP script to be run by PHP-Fuzzer in order to prepare and execute the code that we want to fuzz. Here is what such a fuzzer.php script could look like for fuzzing the csv-parser library:

<?php declare(strict_types=1);
use PhpFuzzer\Config;
use SebastianBergmann\CsvParser\Exception;
use SebastianBergmann\CsvParser\FieldDefinition;
use SebastianBergmann\CsvParser\ObjectMapper;
use SebastianBergmann\CsvParser\Parser;
use SebastianBergmann\CsvParser\Schema;
use SebastianBergmann\CsvParser\Type;

/**
 * This file is meant to be loaded by the php-fuzzer harness,
 * which provides $config before executing this script.
 */
assert(isset($config) && $config instanceof Config);

/**
 * Build a Schema object that describes a 5-column CSV layout:
 * - Column 1: a string field
 * - Column 2: an integer field
 * - Column 3: a float field
 * - Column 4: a boolean field
 * - Column 5: an object field, using a custom inline ObjectMapper
 */
$schema = Schema::from(
    FieldDefinition::from(1, 'string', Type::string()),
    FieldDefinition::from(2, 'integer', Type::integer()),
    FieldDefinition::from(3, 'float', Type::float()),
    FieldDefinition::from(4, 'boolean', Type::boolean()),
    FieldDefinition::from(5, 'object', Type::object(
        new class implements ObjectMapper
        {
            public function map(string $value): stdClass
            {
                $obj        = new stdClass;
                $obj->value = $value;

                return $obj;
            }
        })
    ),
);

/**
 * Register the fuzz target function with the fuzzer.
 * The fuzzer will repeatedly call this function with
 * different randomly generated/mutated byte strings.
 */
$config->setTarget(
    static function (string $input) use ($schema): void
    {
        /**
         * Guard against inputs shorter than 2 bytes,
         * since the function needs at least 2 bytes
         * for the separator and enclosure characters.
         */
        if (strlen($input) < 2) {
            return;
        }

        /**
         * Use first byte of fuzz input as the CSV field separator character.
         * Use second byte of fuzz input as the CSV enclosure/quote character.
         * Everything after the first two bytes is treated as the raw CSV data to parse.
         */
        $separator = $input[0];
        $enclosure = $input[1];
        $csvData   = substr($input, 2);

        $parser = new Parser;

        try {
            $parser->setSeparator($separator);
        } catch (Exception) {
            // Ignore invalid separator
        }

        try {
            $parser->setEnclosure($enclosure);
        } catch (Exception) {
            // Ignore invalid enclosure
        }

        /**
         * Wrap the CSV data in a data:// stream URI.
         * The parser expects a file path or stream, so this converts the in-memory
         * string into a readable stream without writing a temporary file to disk.
         */
        $csvFile = 'data://text/plain;base64,' . base64_encode($csvData);

        try {
            foreach ($parser->parse($csvFile, $schema) as $row) {
                // Just iterate to trigger any potential issues
            }
        } catch (Exception) {
            // Expected exceptions are fine
        }
    },
);

Here is the (beginning of the) output of running PHP-Fuzzer using our fuzzer.php script shown above:

❯ ./vendor/bin/php-fuzzer fuzz fuzzer.php corpus
NEW    run: 1 (3998/s), ft: 2 (7997/s), corp: 1 (1b), len: 1/1, t: 0s, mem: 3469kb
NEW    run: 6 (5565/s), ft: 18 (16696/s), corp: 2 (3b), len: 2/2, t: 0s, mem: 3580kb
NEW    run: 17 (12247/s), ft: 29 (20892/s), corp: 3 (6b), len: 3/3, t: 0s, mem: 3582kb
NEW    run: 18 (12138/s), ft: 31 (20904/s), corp: 4 (9b), len: 3/3, t: 0s, mem: 3584kb
NEW    run: 174 (78449/s), ft: 35 (15780/s), corp: 5 (13b), len: 4/4, t: 0s, mem: 3587kb
NEW    run: 736 (162329/s), ft: 38 (8381/s), corp: 6 (19b), len: 6/6, t: 0s, mem: 3589kb
NEW    run: 1458 (182361/s), ft: 43 (5378/s), corp: 7 (26b), len: 7/7, t: 0s, mem: 3595kb
REDUCE run: 1772 (183473/s), ft: 43 (4452/s), corp: 7 (25b), len: 5/7, t: 0s, mem: 3596kb
REDUCE run: 2014 (184738/s), ft: 43 (3944/s), corp: 7 (24b), len: 6/7, t: 0s, mem: 3598kb
NEW    run: 5821 (204281/s), ft: 53 (1860/s), corp: 8 (34b), len: 10/10, t: 0s, mem: 3599kb
NEW    run: 6659 (204014/s), ft: 56 (1716/s), corp: 9 (54b), len: 20/20, t: 0s, mem: 3603kb
NEW    run: 6870 (203447/s), ft: 58 (1718/s), corp: 10 (74b), len: 20/20, t: 0s, mem: 3606kb
CRASH in /path/to/csv-parser/crash-64f7467e63f83c2267a2e9d6688fa7bc.txt!
Error: [2] The float-string "55555555555555555555" is not representable as an int,
cast occurred in /path/to/csv-parser/src/schema/type/IntegerType.php on line 21

Let us break down the command:

• ./vendor/bin/php-fuzzer is the PHP-Fuzzer executable
• fuzz is the name of the PHP-Fuzzer command we want to invoke
• fuzzer.php is the path to our fuzzing script (see above)
• corpus is the path to the directory where PHP-Fuzzer should store the corpus

The fuzzer generated numeric strings that were far larger than the maximum value a 64-bit integer can hold. Since PHP 8.5, casting such a “float-string” to an int emits a warning. While the code continues to run, this warning can break error-sensitive environments or clutter logs. Most importantly, of course, the numeric string from the CSV file is not parsed correctly into a PHP variable.

These changes were made to the csv-parser library based on the findings from fuzz testing.

How PHP-Fuzzer works

In coverage-guided fuzzing, program inputs are repeatedly mutated and the program is then run while the execution of code branches or basic blocks is measured. Inputs that increase code coverage by reaching new branches or paths are retained as interesting seeds, and future mutations are biased towards them. This feedback loop systematically drives the fuzzer deeper into the program's state space over time, increasing the likelihood of triggering bugs.

Unlike traditional sources of code coverage information such as the PCOV or Xdebug extensions for the PHP interpreter which monitor execution at the engine level, PHP-Fuzzer uses an approach based on code transformation. This allows it to track execution flow without requiring any special PHP extensions to be installed.

PHP-Fuzzer uses code transformation instead of a PHP extension for several reasons. Both PCOV and Xdebug impact code execution performance. Code coverage instrumentation using code transformation can be optimised to meet the specific needs of a fuzzer, which only needs to know whether an edge has been hit rather than requiring full code coverage software metrics. Furthermore, fuzzing often requires more than just line coverage. It also requires edge coverage (tracking transitions between blocks) and, in some cases, value-profile information (tracking the values used in comparisons). Code transformation makes it easy to inject custom logic to track this information.

In summary, PHP-Fuzzer rewrites your code as it is being loaded, inserting hundreds of tiny checkpoints that report back to the fuzzer's engine every time a new path is explored.

Two sides of the same coin?

Both fuzz testing and property-based testing automatically generate inputs and exercise our code with it. They can also simplify inputs that cause failures and reduce them to a minimum. The crucial difference lies in the programming work:

• Property-based testing requires programmers to have a deep understanding of the system. Specifically, they must specify the properties that should always be true and define the form of the “interesting” inputs. While the effort involved is high, the reward is that the tests run quickly, like unit tests, and can easily be integrated into continuous integration pipelines.
• Fuzzing in its basic form is a black-box method in which the fuzzer makes few assumptions about the system and mutates inputs randomly. Coverage-guided fuzzing, such as PHP-Fuzzer, is a grey-box method: it uses code coverage feedback to steer mutations towards unexplored paths without requiring a full specification of the system. To achieve good coverage, the fuzzer must run for a long time – often hours, days, or even weeks.

An important point to note is that, in property-based testing, the test oracle is explicit: we write assertions that define our expectations of the system. In fuzzing, however, the test oracle is often implicit. For example, “the code should not crash” is a simple yet powerful property.

Conclusion

The CSV parser example illustrates the core strength of fuzz testing: despite achieving 100% code coverage through traditional testing, an error was missed that was immediately revealed through fuzzing. Comprehensive code coverage does not guarantee comprehensive security. Input processors sit at the critical intersection of trusted systems and untrusted data, making them prime targets for exploitation. While conventional testing validates expected behaviour using known inputs, coverage-guided fuzzing systematically probes for weaknesses using unexpected vectors that attackers exploit: each input that reaches new code paths seeds further mutation, progressively deepening the exploration of the application's state space.

Rather than preventing chaos, security is achieved by learning from it: by acting as adversaries to our own software and allowing invalid inputs to reveal flaws in the validation logic. Each crash presents an opportunity to strengthen defences. The hardened parser, which now explicitly validates boundaries, is a testament to this principle: resilience is built not through perfect foresight, but through surviving chaos. By integrating fuzz testing into continuous workflows, teams can stress-test defences before attackers do and enable continuous regression detection.

If you seek to deepen your understanding of fuzzing, I recommend these two complementary presentations: “Demystifying Fuzzer Behaviour” provides essential foundational knowledge by examining how fuzzers and programs interact from first principles and helping you build a mental model of fuzzer mechanics. “What the PHUZZ?!” applies these concepts concretely to web application security, demonstrating how coverage-guided fuzzing can discover vulnerabilities like SQL Injection, Remote Code Execution, and Cross-Site-Scripting in PHP web applications. Together, they form a natural progression from theory to practice.