Understanding Testing

What it is about

Writing tests is no longer the bottleneck. A coding agent generates more tests in seconds than you would write in a day. They are green, and the code coverage looks good. But do they check what your software is supposed to do, or do they merely confirm what the code happens to do, including its bugs?

No tool can answer this question for you. To answer it, you have to understand what makes a good test: where it takes its expectation from, how much of the system it covers, when it may replace a dependency, and what a green bar really means.

That is why you write no code in this training; you read it. Sebastian explains the terms and principles and shows them in examples, and you apply them to real tests, many of them written by an agent. The examples are in PHP with PHPUnit; you take what you learn into any language.

How you work in this training

Every session follows the same rhythm:

• Explanation. A term or principle, shown with a small example.
• Demonstration. The principle in running code, with tests turning red and green.
• Assessment exercise. In breakout rooms you and the others assess given tests against an assessment sheet, many of them generated by an agent.
• Discussion. The groups compare their verdicts and justify where they differ.

One question runs through all sessions: “What do you know when this test is green?” The examples come from a small shopping cart in PHP, and at the end of each session we briefly look beyond PHP: what the same concept is called in other languages and tools. The criteria of all sessions grow into a checklist that you apply to a complete generated test suite in the last session.

Programme

1. 1

   Session 1: What a test tells you

   What do you know when a test is green, and what do you not know?

   • A test as a written-down expectation that can be repeated at any time, and its structure: arrange, act, assert, or given, when, then
   • The oracle: does the expected answer come from the requirement or from the code? A test that takes it from the code confirms the bugs of that code as well
   • Finding test cases: equivalence classes, boundary values, error cases
   • Failure or error: the expectation is violated, or the test could not be executed
   • What a green test does not tell you

   Result: Test cases and edge cases for a requirement, determined before you read its tests, and for each test the answer to where it takes its expectation from

2. 2

   Session 2: Properties of good tests: F.I.R.S.T.

   How do you recognise a good test before it lets you down?

   • Fast: a slow test runs less often than it should
   • Isolated: a test does not depend on other tests or their order
   • Repeatable: the same result on every machine, at any time of day, without a network
   • Self-validating: the test decides on success and failure itself, nobody has to read any output
   • Timely: when is a test written, and what does it mean when an agent writes it after the code?
   • Typical violations in generated tests: the real time of day, randomness, network access, assertions that say nothing

   Result: Ten given tests, each assessed by which principle it violates and what that would mean in day-to-day work

3. 3

   Session 3: The life cycle of a test

   What happens before, during, and after a test, and how long does it live?

   • In the test run: set up the fixture, exercise, verify, tear down (four-phase test)
   • Fresh and shared fixtures, setUp() and tearDown(), and what a test framework isolates and what it does not
   • Shared state as the most common cause of tests that are sometimes green and sometimes red
   • In the codebase: a test is written, turns red, is adapted or deleted. When is a red test a regression, and when a changed requirement?
   • Short-lived and long-lived tests: a safety net for a refactoring may go afterwards, a business rule stays

   Result: Shared state recognised in given tests, and a justified decision for every test that a changed requirement of the shopping cart affects: keep, adapt, or delete

4. 4

   Session 4: Test scope

   How much of the system belongs in a test?

   • What is a unit? Solitary and sociable unit tests
   • Unit, integration, end-to-end: what each level can find and what it cannot
   • System boundaries such as the database, the file system, the network, and time, and test sizes by which of them a test touches: small, medium, large
   • Cost per level: run time, stability, precision of the failure message
   • The test pyramid as a first overview

   Result: The expectations for the checkout of the shopping cart, distributed across levels with a justification, and a generated test suite whose tests are classified by their actual scope

5. 5

   Session 5: Test doubles

   When do you replace a dependency, and with what?

   • Why a dependency is replaced: slowness, non-determinism, side effects, unavailability
   • Dummy, stub, spy, mock, and fake, and whether a test verifies state or communication
   • When a double deceives: it answers the way you believe the real dependency answers
   • Only replacing your own interfaces, and never value objects
   • Generated tests often replace everything: what do they still check then?

   Result: Every test double in given tests named, with a decision on whether it is the right one, whether it is needed at all, and what it hides

6. 6

   Session 6: Code coverage and the big picture

   What does code coverage measure, and what does it not?

   • Line, branch, and path coverage
   • Coverage measures execution and says nothing about verification: a test without an assertion reaches 100 %, and generated tests reach high values that say nothing
   • Coverage as a gap finder: a line that is not executed raises a question, and covering it is no goal in itself
   • Outlook: mutation testing measures whether tests verify
   • The checklist: oracle, F.I.R.S.T., lifetime, scope, doubles, coverage

   Result: A complete generated test suite for the shopping cart, assessed with the checklist: which tests you keep, which you change, and which you delete

Who it is for

• Developers who have coding agents write tests and want to assess whether those tests are any good
• Everyone who already writes tests but never learnt the terms and principles behind them systematically
• Tech leads and reviewers who judge tests in pull requests
• Developers who work in languages other than PHP: the concepts do not depend on the language

Prerequisites

• Experience in software development with an object-oriented programming language
• No prior knowledge of testing is required
• You do not need a development environment: you read and assess code, you do not write any

What you take away

• You determine test cases and edge cases for a requirement systematically
• You recognise when a test takes its expectation from the code instead of from the requirement
• You assess tests by the F.I.R.S.T. principles and name what a test is missing
• You classify tests by their scope and know what each level can find
• You tell test doubles apart and recognise when one hides more than it helps
• You read coverage reports correctly and are not fooled by 100 %
• You take away a checklist with which you assess every test, whether a human or an agent wrote it

Related training

This training is part of the learning paths Getting started with testing, Testing rules for the team.