Testing all 2 written

Proving your code does what you claim, with the library that ships in the box.

39 Testing with unittest, run for real

A test is a small program that runs your code and states what the answer should be. Every panel in this course is one, which is why you can trust the output on these pages. This lesson uses unittest, which ships with Python, and runs real suites — including one that fails on purpose, so you can read what a failure actually tells you.

A first suite, really executed

Three tests, run for real

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34import io
import unittest

# The code under test.
def split_name(full):
    """Return (first, last). Raises ValueError if there is no space."""
    if " " not in full:
        raise ValueError(f"expected two names, got {full!r}")
    first, _, last = full.partition(" ")
    return first, last

# The tests: a class per unit, a method per case, each named test_*.
class TestSplitName(unittest.TestCase):
    def test_splits_two_names(self):
        self.assertEqual(split_name("Ada Lovelace"), ("Ada", "Lovelace"))

    def test_keeps_the_remainder_together(self):
        self.assertEqual(split_name("Ada King Lovelace"), ("Ada", "King Lovelace"))

    def test_rejects_a_single_name(self):
        with self.assertRaises(ValueError):
            split_name("Ada")

# Run it and capture the report, rather than letting a runner exit the process.
suite = unittest.TestLoader().loadTestsFromTestCase(TestSplitName)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)

print(stream.getvalue())
print("tests run:", result.testsRun, " failures:", len(result.failures), " errors:", len(result.errors))

test("all three tests ran", result.testsRun, 3)
test("and all of them passed", (len(result.failures), len(result.errors)), (0, 0))
test("the suite reports success", result.wasSuccessful(), True)

What it printed, when the build ran it

test_keeps_the_remainder_together (__main__.TestSplitName.test_keeps_the_remainder_together) ... ok
test_rejects_a_single_name (__main__.TestSplitName.test_rejects_a_single_name) ... ok
test_splits_two_names (__main__.TestSplitName.test_splits_two_names) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

tests run: 3  failures: 0  errors: 0
ok   all three tests ran
ok   and all of them passed
ok   the suite reports success

Why the runner is driven by hand here. The usual entry point, unittest.main(), calls sys.exit when it finishes — which would end the panel before anything else could print. Using TextTestRunner with a stream captures the report as a string instead. In your own project you would put the tests in a file and run python -m unittest, which is shown at the end of this lesson.

A failing test, and what it tells you

This is the output you will spend the most time reading, so it is worth seeing deliberately. The assertion methods exist to produce exactly this: not "the test failed" but what was expected, what happened, and where.

A real failure, with the real message

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28import io
import unittest

def discount(price, percent):
    # The bug: dividing by 10 rather than 100.
    return price - (price * percent / 10)

class TestDiscount(unittest.TestCase):
    def test_ten_percent_off_200(self):
        self.assertEqual(discount(200, 10), 180)

    def test_zero_percent_changes_nothing(self):
        self.assertEqual(discount(200, 0), 200)

suite = unittest.TestLoader().loadTestsFromTestCase(TestDiscount)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)

print(stream.getvalue())
print("failures:", len(result.failures))

failure_text = result.failures[0][1]
print("the useful line:")
print("  ", [line for line in failure_text.splitlines() if "!=" in line][0].strip())

test("one of the two tests failed", len(result.failures), 1)
test("the passing one still passed", result.testsRun - len(result.failures), 1)
test("and the message shows both values", "180" in failure_text and "0.0" in failure_text, True)

What it printed, when the build ran it

test_ten_percent_off_200 (__main__.TestDiscount.test_ten_percent_off_200) ... FAIL
test_zero_percent_changes_nothing (__main__.TestDiscount.test_zero_percent_changes_nothing) ... ok

======================================================================
FAIL: test_ten_percent_off_200 (__main__.TestDiscount.test_ten_percent_off_200)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "example.py", line 10, in test_ten_percent_off_200
    self.assertEqual(discount(200, 10), 180)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0.0 != 180

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)

failures: 1
the useful line:
   AssertionError: 0.0 != 180
ok   one of the two tests failed
ok   the passing one still passed
ok   and the message shows both values

The assertions worth knowing

The assertion vocabulary

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43import io
import unittest

class TestAssertions(unittest.TestCase):
    def test_equality(self):
        self.assertEqual(2 + 2, 4)
        self.assertNotEqual(2 + 2, 5)

    def test_truthiness(self):
        self.assertTrue([1])
        self.assertFalse([])

    def test_identity_and_none(self):
        value = None
        self.assertIsNone(value)
        shared = [1]
        self.assertIs(shared, shared)

    def test_membership_and_types(self):
        self.assertIn("a", "abc")
        self.assertNotIn("z", "abc")
        self.assertIsInstance(1, int)

    def test_floats_need_a_tolerance(self):
        # assertEqual would fail here, for lesson 4's reason.
        self.assertAlmostEqual(0.1 + 0.2, 0.3)

    def test_collections_ignoring_order(self):
        self.assertCountEqual([1, 2, 2], [2, 1, 2])

    def test_exceptions_and_their_messages(self):
        with self.assertRaises(ZeroDivisionError):
            1 / 0
        with self.assertRaisesRegex(ValueError, "invalid literal"):
            int("nope")

suite = unittest.TestLoader().loadTestsFromTestCase(TestAssertions)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=1).run(suite)
print(stream.getvalue())

test("every assertion style passed", result.wasSuccessful(), True)
test("seven test methods ran", result.testsRun, 7)

What it printed, when the build ran it

.......
----------------------------------------------------------------------
Ran 7 tests in 0.000s

OK

ok   every assertion style passed
ok   seven test methods ran

Use assertAlmostEqual for floats. assertEqual(0.1 + 0.2, 0.3) fails for exactly the reason lesson 4 demonstrated, and the failure looks like a bug in your code rather than in the test. This is the same rule as close() in this course's own panels.

`setUp`: the same starting state for every test

setUp and tearDown, and the isolation they give

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60import io
import unittest

class Cart:
    def __init__(self):
        self.items = []

    def add(self, name, price):
        self.items.append((name, price))

    def total(self):
        return sum(price for _, price in self.items)

order_of_events = []

class TestCart(unittest.TestCase):
    def setUp(self):
        """Runs before EVERY test method."""
        order_of_events.append("setUp")
        self.cart = Cart()
        self.cart.add("book", 10)

    def tearDown(self):
        """Runs after every test method, even if it failed."""
        order_of_events.append("tearDown")

    def test_starts_with_the_fixture(self):
        order_of_events.append("test_a")
        self.assertEqual(self.cart.total(), 10)

    def test_additions_do_not_leak_between_tests(self):
        order_of_events.append("test_b")
        self.cart.add("pen", 2)
        self.assertEqual(self.cart.total(), 12)

suite = unittest.TestLoader().loadTestsFromTestCase(TestCart)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)
print(stream.getvalue())

print("order of events:")
for event in order_of_events:
    print("  ", event)

print()
print("Note which test ran first: 'additions' before 'starts', because unittest")
print("sorts test methods by NAME, not by the order they appear in the file.")

# The shape that matters: setUp/teardown wrap every test, whatever the order.
wrapping = [
    order_of_events[i:i + 3] for i in range(0, len(order_of_events), 3)
]

test("both tests passed, so neither saw the other's cart", result.wasSuccessful(), True)
test("setUp ran before each test", order_of_events.count("setUp"), 2)
test("tearDown ran after each", order_of_events.count("tearDown"), 2)
test("every test is wrapped by a setUp and a tearDown",
     [[block[0], block[2]] for block in wrapping], [["setUp", "tearDown"]] * 2)
test("and unittest ran them in alphabetical order, not file order",
     order_of_events[1], "test_b")

What it printed, when the build ran it

test_additions_do_not_leak_between_tests (__main__.TestCart.test_additions_do_not_leak_between_tests) ... ok
test_starts_with_the_fixture (__main__.TestCart.test_starts_with_the_fixture) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

order of events:
   setUp
   test_b
   tearDown
   setUp
   test_a
   tearDown

Note which test ran first: 'additions' before 'starts', because unittest
sorts test methods by NAME, not by the order they appear in the file.
ok   both tests passed, so neither saw the other's cart
ok   setUp ran before each test
ok   tearDown ran after each
ok   every test is wrapped by a setUp and a tearDown
ok   and unittest ran them in alphabetical order, not file order

Tests run in alphabetical order, not the order you wrote them. The panel above proves it: test_additions... ran before test_starts.... So never write a test that depends on another having run first — it will break the moment someone renames one. setUp exists precisely so each test can start from a known state without that coupling.

`subTest`: many cases, one test method

A loop inside a test stops at the first failure and hides the rest. subTest reports each iteration separately, so one run tells you about every failing case.

One failure reported, versus every failure reported

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33import io
import unittest

def is_even(n):
    # The bug: wrong for negative odd numbers.
    return n % 2 == 0 if n >= 0 else True

class TestIsEven(unittest.TestCase):
    def test_without_subtest(self):
        """Stops at the first failure -- you learn about one case."""
        for value, expected in [(-3, False), (-1, False), (2, True)]:
            self.assertEqual(is_even(value), expected)

    def test_with_subtest(self):
        """Reports every failing case in one run."""
        for value, expected in [(-3, False), (-1, False), (2, True)]:
            with self.subTest(value=value):
                self.assertEqual(is_even(value), expected)

suite = unittest.TestLoader().loadTestsFromTestCase(TestIsEven)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)
print(stream.getvalue()[:900])

print("total failures reported:", len(result.failures))
print()
print("The plain loop contributed one failure; the subTest version contributed")
print("one per failing value, and each names the value that failed.")

subtest_failures = [f for f in result.failures if "value=" in str(f[0])]
test("the subTest version reported each failing case separately", len(subtest_failures), 2)
test("and the failures name the offending values",
     sorted("-3" in str(f[0]) or "-1" in str(f[0]) for f in subtest_failures), [True, True])

What it printed, when the build ran it

test_with_subtest (__main__.TestIsEven.test_with_subtest)
Reports every failing case in one run. ... 
  test_with_subtest (__main__.TestIsEven.test_with_subtest) (value=-3)
Reports every failing case in one run. ... FAIL
  test_with_subtest (__main__.TestIsEven.test_with_subtest) (value=-1)
Reports every failing case in one run. ... FAIL
test_without_subtest (__main__.TestIsEven.test_without_subtest)
Stops at the first failure -- you learn about one case. ... FAIL

======================================================================
FAIL: test_with_subtest (__main__.TestIsEven.test_with_subtest) (value=-3)
Reports every failing case in one run.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "example.py", line 18, in test_with_subtest
    self.assertEqual(is
total failures reported: 3

The plain loop contributed one failure; the subTest version contributed
one per failing value, and each names the value that failed.
ok   the subTest version reported each failing case separately
ok   and the failures name the offending values

Test doubles with `unittest.mock`

Lesson 30 argued for composition because it lets a collaborator be swapped. unittest.mock is the standard library's tool for that swap, and it also records what happened.

Mock: a stand-in that records what it was asked to do

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41import io
import unittest
from unittest.mock import Mock, patch

class Notifier:
    def send(self, address, message):
        raise RuntimeError("this would really send an email")

def register(email, notifier):
    if "@" not in email:
        raise ValueError("not an email address")
    notifier.send(email, "welcome")
    return "registered"

class TestRegister(unittest.TestCase):
    def test_sends_a_welcome_message(self):
        fake = Mock()                      # accepts any call, records everything
        result = register("ada@example.com", fake)

        self.assertEqual(result, "registered")
        fake.send.assert_called_once_with("ada@example.com", "welcome")

    def test_does_not_notify_on_invalid_input(self):
        fake = Mock()
        with self.assertRaises(ValueError):
            register("not-an-email", fake)
        fake.send.assert_not_called()

    def test_a_mock_can_also_return_values(self):
        fake = Mock()
        fake.send.return_value = "queued"
        self.assertEqual(fake.send("a@b.c", "hi"), "queued")
        self.assertEqual(fake.send.call_count, 1)

suite = unittest.TestLoader().loadTestsFromTestCase(TestRegister)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)
print(stream.getvalue())

test("all three mock-based tests passed", result.wasSuccessful(), True)
test("no real email was ever sent, because Notifier was never used", result.testsRun, 3)

What it printed, when the build ran it

test_a_mock_can_also_return_values (__main__.TestRegister.test_a_mock_can_also_return_values) ... ok
test_does_not_notify_on_invalid_input (__main__.TestRegister.test_does_not_notify_on_invalid_input) ... ok
test_sends_a_welcome_message (__main__.TestRegister.test_sends_a_welcome_message) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

ok   all three mock-based tests passed
ok   no real email was ever sent, because Notifier was never used

patch replaces something for the duration of a test — useful when the dependency is not injectable, such as a module-level function or the current time.

patch, and the injection that makes it unnecessary

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38import io
import unittest
from unittest.mock import patch
from datetime import datetime, UTC

def greeting(now=None):
    """Uses the real clock unless told otherwise."""
    hour = (now or datetime.now(UTC)).hour
    return "good morning" if hour < 12 else "good afternoon"

# Injectable, so no patching needed -- this is the better design.
print("injected 9am: ", greeting(datetime(2026, 1, 1, 9, tzinfo=UTC)))
print("injected 3pm: ", greeting(datetime(2026, 1, 1, 15, tzinfo=UTC)))

# And when a dependency is NOT injectable, patch reaches in.
def uses_random():
    import random
    return random.randint(1, 6)

class TestPatching(unittest.TestCase):
    def test_patching_a_module_function(self):
        with patch("random.randint", return_value=4):
            self.assertEqual(uses_random(), 4)

    def test_the_patch_is_undone_afterwards(self):
        value = uses_random()
        self.assertIn(value, range(1, 7))

suite = unittest.TestLoader().loadTestsFromTestCase(TestPatching)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)
print()
print(stream.getvalue())

test("injecting the value needs no mocking at all",
     greeting(datetime(2026, 1, 1, 9, tzinfo=UTC)), "good morning")
test("patch replaced the function inside the block", result.wasSuccessful(), True)
test("and both tests ran", result.testsRun, 2)

What it printed, when the build ran it

injected 9am:  good morning
injected 3pm:  good afternoon

test_patching_a_module_function (__main__.TestPatching.test_patching_a_module_function) ... ok
test_the_patch_is_undone_afterwards (__main__.TestPatching.test_the_patch_is_undone_afterwards) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK

ok   injecting the value needs no mocking at all
ok   patch replaced the function inside the block
ok   and both tests ran

Prefer injection to patching. patch("random.randint") depends on the string path to the thing being patched, so it silently stops working when code is moved or renamed — and it couples the test to the implementation rather than the behaviour. A parameter with a default, as greeting(now=None) has, needs no mocking and cannot break that way.

Running tests in a real project

The commands you will actually type

shown, not run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22# Layout: tests live beside the package, in files named test_*.py
myproject/
    src/myproject/core.py
    tests/test_core.py

# Discover and run everything
python -m unittest discover

# Run one module, class, or method
python -m unittest tests.test_core
python -m unittest tests.test_core.TestTotals
python -m unittest tests.test_core.TestTotals.test_empty

# Useful flags
python -m unittest -v          # name every test as it runs
python -m unittest -f          # stop at the first failure
python -m unittest --locals    # show local variables in tracebacks

# Coverage, which is third-party but the standard choice
pip install coverage
coverage run -m unittest discover
coverage report -m

Shell commands rather than Python, and they need a project directory on disk plus (for coverage) a third-party package. The suites in the panels above are run for real by the build using TextTestRunner; only these command-line invocations are shown rather than executed.

On pytest

Most Python projects use pytest rather than unittest. It is third-party, so this course cannot run it — but the difference is worth knowing, because it is mostly less ceremony for the same ideas: a plain assert instead of self.assertEqual, plain functions instead of a TestCase subclass, and fixtures as arguments instead of setUp.

The same tests, in pytest

shown, not run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27# The same tests, in pytest style. NOT RUN HERE: pytest is third-party and
# this course installs nothing.

import pytest

def test_splits_two_names():
    # A plain assert. pytest rewrites it to report both values on failure.
    assert split_name("Ada Lovelace") == ("Ada", "Lovelace")

def test_rejects_a_single_name():
    with pytest.raises(ValueError, match="expected two names"):
        split_name("Ada")

# A fixture replaces setUp, and is requested by naming it as a parameter.
@pytest.fixture
def cart():
    c = Cart()
    c.add("book", 10)
    return c

def test_starts_with_the_fixture(cart):
    assert cart.total() == 10

# Parametrisation replaces the subTest loop, and each case is a separate test.
@pytest.mark.parametrize("value,expected", [(-3, False), (-1, False), (2, True)])
def test_is_even(value, expected):
    assert is_even(value) is expected

pytest is not in the standard library and this course installs nothing, so this is the real pytest API written out rather than executed. Every concept it shows — assertions, expected exceptions, fixtures, parametrised cases — is run for real in the unittest panels above.

Everything in this lesson transfers. setUp becomes a fixture, subTest becomes parametrize, assertRaises becomes pytest.raises, and unittest.mock is used by pytest projects unchanged. Learn the ideas here and the syntax is a small translation.

What to take from this lesson

A TestCase subclass, methods named test_*, run with python -m unittest.
Read the failure message: it names what was expected and what happened.
assertAlmostEqual for floats, assertRaises as a context manager for exceptions.
setUp gives every test the same fresh starting state.
subTest so a loop reports every failing case, not just the first.
Prefer injecting a collaborator to patching a name.

40 What to test, and the tests that earn their keep

Knowing how to write a test is the easy half. The hard half is knowing which tests are worth having — because a suite of the wrong tests is worse than a small suite of the right ones: it takes time to run, breaks whenever you refactor, and still lets bugs through.

Test behaviour, not implementation

A test that mirrors the code's internal steps passes for the wrong reason and breaks when nothing is actually broken. A test that states what the function is for survives a rewrite.

One set of tests, two implementations

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40import io, unittest

# Version one: a loop.
def total_v1(prices, tax_rate):
    running = 0
    for price in prices:
        running += price
    return round(running * (1 + tax_rate), 2)

# Version two: the same behaviour, rewritten. No caller should notice.
def total_v2(prices, tax_rate):
    return round(sum(prices) * (1 + tax_rate), 2)

class BehaviourTests(unittest.TestCase):
    """States what the function is for. Survives the rewrite."""

    def test_applies_tax_to_the_sum(self):
        for implementation in (total_v1, total_v2):
            with self.subTest(fn=implementation.__name__):
                self.assertEqual(implementation([10, 20], 0.2), 36.0)

    def test_an_empty_basket_is_free(self):
        for implementation in (total_v1, total_v2):
            with self.subTest(fn=implementation.__name__):
                self.assertEqual(implementation([], 0.2), 0.0)

    def test_rounds_to_two_places(self):
        for implementation in (total_v1, total_v2):
            with self.subTest(fn=implementation.__name__):
                self.assertEqual(implementation([0.333], 0.0), 0.33)

suite = unittest.TestLoader().loadTestsFromTestCase(BehaviourTests)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=1).run(suite)
print(stream.getvalue())
print("both implementations satisfy the same tests:", result.wasSuccessful())

test("the two implementations agree", total_v1([10, 20], 0.2), total_v2([10, 20], 0.2))
test("the behaviour tests passed for both", result.wasSuccessful(), True)
test("nine sub-cases in three methods", result.testsRun, 3)

What it printed, when the build ran it

...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

both implementations satisfy the same tests: True
ok   the two implementations agree
ok   the behaviour tests passed for both
ok   nine sub-cases in three methods

The tell for an implementation test. If you had to read the function body to write the test, or the test would break when you rename a private helper, it is testing structure. Ask instead: what would a caller be upset about if it changed? Test that.

A test that cannot fail

The most common wasted test restates the code. It passes because it duplicates the same mistake, and it will keep passing while the function is wrong.

Two tests, one buggy function, one green tick

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35import io, unittest

# The function has a bug: percent is divided by 10, not 100.
def discount(price, percent):
    return price - (price * percent / 10)

class UselessTest(unittest.TestCase):
    def test_discount(self):
        """Restates the implementation. Cannot detect the bug."""
        price, percent = 200, 10
        expected = price - (price * percent / 10)      # the same wrong formula
        self.assertEqual(discount(price, percent), expected)

class UsefulTest(unittest.TestCase):
    def test_ten_percent_off_two_hundred_is_one_eighty(self):
        """States the requirement independently. Catches the bug."""
        self.assertEqual(discount(200, 10), 180)

for name, case in [("restates the code", UselessTest), ("states the requirement", UsefulTest)]:
    suite = unittest.TestLoader().loadTestsFromTestCase(case)
    stream = io.StringIO()
    outcome = unittest.TextTestRunner(stream=stream, verbosity=0).run(suite)
    print(f"{name:24} passed={outcome.wasSuccessful()}")

print()
print("Same buggy function. The first test is green and worthless.")

useless = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(
    unittest.TestLoader().loadTestsFromTestCase(UselessTest))
useful = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(
    unittest.TestLoader().loadTestsFromTestCase(UsefulTest))

test("the test that restates the code passes", useless.wasSuccessful(), True)
test("the test that states the requirement fails", useful.wasSuccessful(), False)
test("and the function really is wrong", discount(200, 10), 0.0)

What it printed, when the build ran it

restates the code        passed=True
states the requirement   passed=False

Same buggy function. The first test is green and worthless.
ok   the test that restates the code passes
ok   the test that states the requirement fails
ok   and the function really is wrong

Write the expected value by hand. Ten percent off two hundred is one hundred and eighty — you know that without reading any code, which is exactly why it is a useful assertion. If you find yourself computing the expected value with the same expression the function uses, stop.

Coverage is not correctness

100% of the lines, none of the boundaries

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26import io, unittest

def classify(n):
    """Every line of this runs in the test below. It is still wrong."""
    if n > 0:
        return "positive"
    return "negative"          # wrong for zero

class FullCoverage(unittest.TestCase):
    def test_positive(self):
        self.assertEqual(classify(5), "positive")

    def test_negative(self):
        self.assertEqual(classify(-5), "negative")

suite = unittest.TestLoader().loadTestsFromTestCase(FullCoverage)
result = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)

print("every line executed, every test green:", result.wasSuccessful())
print()
print("and yet:")
print("   classify(0) ->", classify(0), "<- nobody would call zero negative")

test("the suite passes", result.wasSuccessful(), True)
test("every branch was executed by those two tests", result.testsRun, 2)
test("and the boundary is still wrong", classify(0), "negative")

What it printed, when the build ran it

every line executed, every test green: True

and yet:
   classify(0) -> negative <- nobody would call zero negative
ok   the suite passes
ok   every branch was executed by those two tests
ok   and the boundary is still wrong

Coverage tells you what was executed, not what was checked. It is useful for finding code no test touches at all — a genuine gap — and useless as a target. The interesting inputs are almost never the ones that add coverage: they are zero, the empty collection, the single-element collection, the boundary value, the duplicate, and the one item more than the limit.

The inputs worth testing

Empty, one, even, duplicates, negatives, and no mutation

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48import io, unittest

def median(values):
    """The middle value, or the mean of the two middle values."""
    if not values:
        raise ValueError("median of an empty sequence")
    ordered = sorted(values)
    middle = len(ordered) // 2
    if len(ordered) % 2 == 1:
        return ordered[middle]
    return (ordered[middle - 1] + ordered[middle]) / 2

class MedianTests(unittest.TestCase):
    def test_typical_case(self):
        self.assertEqual(median([3, 1, 2]), 2)

    def test_empty_is_an_error(self):
        with self.assertRaises(ValueError):
            median([])

    def test_single_element(self):
        self.assertEqual(median([7]), 7)

    def test_even_length_averages_the_middle_two(self):
        self.assertEqual(median([1, 2, 3, 4]), 2.5)

    def test_duplicates_do_not_confuse_it(self):
        self.assertEqual(median([2, 2, 2]), 2)

    def test_input_is_not_modified(self):
        original = [3, 1, 2]
        median(original)
        self.assertEqual(original, [3, 1, 2])

    def test_negatives_and_zero(self):
        self.assertEqual(median([-1, 0, 1]), 0)

suite = unittest.TestLoader().loadTestsFromTestCase(MedianTests)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)
print(stream.getvalue())

unsorted_input = [3, 1, 2]
median(unsorted_input)

test("all seven cases pass", result.wasSuccessful(), True)
test("including the ones nobody thinks of first", result.testsRun, 7)
test("and the function really does leave its input alone", unsorted_input, [3, 1, 2])

What it printed, when the build ran it

test_duplicates_do_not_confuse_it (__main__.MedianTests.test_duplicates_do_not_confuse_it) ... ok
test_empty_is_an_error (__main__.MedianTests.test_empty_is_an_error) ... ok
test_even_length_averages_the_middle_two (__main__.MedianTests.test_even_length_averages_the_middle_two) ... ok
test_input_is_not_modified (__main__.MedianTests.test_input_is_not_modified) ... ok
test_negatives_and_zero (__main__.MedianTests.test_negatives_and_zero) ... ok
test_single_element (__main__.MedianTests.test_single_element) ... ok
test_typical_case (__main__.MedianTests.test_typical_case) ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.000s

OK

ok   all seven cases pass
ok   including the ones nobody thinks of first
ok   and the function really does leave its input alone

That last test — "input is not modified" — is the kind people forget to write and then regret. A function that sorts its argument in place instead of copying breaks its caller, and nothing about its name warns you.

The test that catches an unwanted side effect

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32import io, unittest

# A subtly different implementation: sorts in place.
def median_mutating(values):
    values.sort()                       # the bug: modifies the caller's list
    middle = len(values) // 2
    if len(values) % 2 == 1:
        return values[middle]
    return (values[middle - 1] + values[middle]) / 2

class MutationTest(unittest.TestCase):
    def test_returns_the_right_answer(self):
        self.assertEqual(median_mutating([3, 1, 2]), 2)      # passes

    def test_does_not_modify_its_input(self):
        original = [3, 1, 2]
        median_mutating(original)
        self.assertEqual(original, [3, 1, 2])                # fails

suite = unittest.TestLoader().loadTestsFromTestCase(MutationTest)
result = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)

print("the answer is correct, so one test passes")
print("failures:", len(result.failures))
print()
caller_list = [3, 1, 2]
median_mutating(caller_list)
print("the caller's list afterwards:", caller_list, "<- reordered without warning")

test("the value returned is right", median_mutating([3, 1, 2]), 2)
test("but exactly one test caught the side effect", len(result.failures), 1)
test("and the caller's list really was reordered", caller_list, [1, 2, 3])

What it printed, when the build ran it

the answer is correct, so one test passes
failures: 1

the caller's list afterwards: [1, 2, 3] <- reordered without warning
ok   the value returned is right
ok   but exactly one test caught the side effect
ok   and the caller's list really was reordered

Write the failing test first, for every bug

This is the single habit that makes a suite prevent recurrence rather than describe the present. When a bug is reported, reproduce it as a test, watch it fail, then fix the code.

Reproduce, then fix

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41import io, unittest

# The bug as reported: "an empty basket crashes".
def average_v1(values):
    return sum(values) / len(values)

# Step 1: write the test that reproduces it, and watch it fail.
class RegressionTest(unittest.TestCase):
    def test_empty_basket_returns_zero(self):
        self.assertEqual(average_v1([]), 0)

before = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(
    unittest.TestLoader().loadTestsFromTestCase(RegressionTest))

print("step 1 -- the test fails, which proves it reproduces the bug")
print("   errors:", len(before.errors), " (a ZeroDivisionError, not a failure)")

# Step 2: fix the code.
def average_v2(values):
    if not values:
        return 0
    return sum(values) / len(values)

class RegressionTestFixed(unittest.TestCase):
    def test_empty_basket_returns_zero(self):
        self.assertEqual(average_v2([]), 0)

    def test_the_normal_case_still_works(self):
        self.assertEqual(average_v2([2, 4]), 3)

after = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(
    unittest.TestLoader().loadTestsFromTestCase(RegressionTestFixed))

print()
print("step 2 -- the fix makes it pass, and the old behaviour is protected")
print("   passed:", after.wasSuccessful())

test("the test reproduced the bug before the fix", len(before.errors), 1)
test("the fix handles the empty case", average_v2([]), 0)
test("without breaking the normal one", average_v2([2, 4]), 3)
test("and the regression suite passes", after.wasSuccessful(), True)

What it printed, when the build ran it

step 1 -- the test fails, which proves it reproduces the bug
   errors: 1  (a ZeroDivisionError, not a failure)

step 2 -- the fix makes it pass, and the old behaviour is protected
   passed: True
ok   the test reproduced the bug before the fix
ok   the fix handles the empty case
ok   without breaking the normal one
ok   and the regression suite passes

Why the order matters. A test written after the fix proves only that the code does what it currently does. A test written first proves it reproduces the reported problem — so if the fix is ever reverted or refactored away, that test fails again and says why. It is also the cheapest possible specification of the bug.

Name the test after the behaviour

Test names as a specification

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33import io, unittest

def parse_age(text):
    age = int(text)
    if age < 0:
        raise ValueError("age cannot be negative")
    return age

# Names that describe what is guaranteed. The failure output reads as a sentence.
class WellNamed(unittest.TestCase):
    def test_parses_a_numeric_string(self):
        self.assertEqual(parse_age("30"), 30)

    def test_rejects_a_negative_age(self):
        with self.assertRaises(ValueError):
            parse_age("-1")

    def test_rejects_non_numeric_text(self):
        with self.assertRaises(ValueError):
            parse_age("thirty")

suite = unittest.TestLoader().loadTestsFromTestCase(WellNamed)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)
print(stream.getvalue())

print("The verbose output is a readable specification of parse_age.")
print("Compare with test_1, test_2, test_parse -- which tell a reader nothing")
print("when one of them fails at 3am.")

test("the suite passes", result.wasSuccessful(), True)
test("three behaviours are documented by three test names", result.testsRun, 3)
test("and the report names each one", "test_rejects_a_negative_age" in stream.getvalue(), True)

What it printed, when the build ran it

test_parses_a_numeric_string (__main__.WellNamed.test_parses_a_numeric_string) ... ok
test_rejects_a_negative_age (__main__.WellNamed.test_rejects_a_negative_age) ... ok
test_rejects_non_numeric_text (__main__.WellNamed.test_rejects_non_numeric_text) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

The verbose output is a readable specification of parse_age.
Compare with test_1, test_2, test_parse -- which tell a reader nothing
when one of them fails at 3am.
ok   the suite passes
ok   three behaviours are documented by three test names
ok   and the report names each one

Fast and independent, or it will not get run

A slow suite is a suite nobody runs. If tests take ten minutes, people stop running them before committing, and the suite stops catching anything. Keep the fast tests — no network, no real database, no sleep — as the ones that run on every change, and put the slow integration tests behind a separate command.

Making the delay injectable is what makes the test fast

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48import io, time, unittest

# A test that sleeps to "wait for" something. Slow and unreliable.
def poll_until_ready(check, attempts=3, delay=0.0):
    for _ in range(attempts):
        if check():
            return True
        time.sleep(delay)
    return False

class SlowStyle(unittest.TestCase):
    def test_with_real_sleeping(self):
        calls = []
        def check():
            calls.append(1)
            return len(calls) >= 3
        # A real delay, multiplied across a whole suite.
        self.assertTrue(poll_until_ready(check, attempts=3, delay=0.05))

class FastStyle(unittest.TestCase):
    def test_with_no_delay_at_all(self):
        calls = []
        def check():
            calls.append(1)
            return len(calls) >= 3
        # The delay is a parameter, so a test can set it to zero.
        self.assertTrue(poll_until_ready(check, attempts=3, delay=0))

timings = {}
for name, case in [("with real sleeps", SlowStyle), ("with delay=0", FastStyle)]:
    start = time.perf_counter()
    outcome = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(
        unittest.TestLoader().loadTestsFromTestCase(case))
    elapsed = time.perf_counter() - start
    timings[name] = (outcome.wasSuccessful(), elapsed)
    print(f"{name:18} passed={outcome.wasSuccessful()}  took {elapsed:.3f}s")

print()
print("Making the delay a parameter is what allows the fast version.")
print("Hard-coding time.sleep inside the function would not.")

test("both styles pass", [ok for ok, _ in timings.values()], [True, True])
test("and the injectable-delay version is the faster one",
     timings["with delay=0"][1] < timings["with real sleeps"][1], True)
test("the polling helper works with no delay",
     poll_until_ready(lambda: True, delay=0), True)
test("and reports failure when the condition never holds",
     poll_until_ready(lambda: False, attempts=2, delay=0), False)

What it printed, when the build ran it

with real sleeps   passed=True  took 0.104s
with delay=0       passed=True  took 0.000s

Making the delay a parameter is what allows the fast version.
Hard-coding time.sleep inside the function would not.
ok   both styles pass
ok   and the injectable-delay version is the faster one
ok   the polling helper works with no delay
ok   and reports failure when the condition never holds

Tests are read more than they are written

Arrange, act, assert

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40import io, unittest

class Cart:
    def __init__(self):
        self.lines = []

    def add(self, name, price, quantity=1):
        self.lines.append({"name": name, "price": price, "quantity": quantity})

    def total(self):
        return sum(line["price"] * line["quantity"] for line in self.lines)

# Arrange, act, assert -- visibly, in that order.
class CartTests(unittest.TestCase):
    def test_total_multiplies_price_by_quantity(self):
        # arrange
        cart = Cart()
        cart.add("book", price=10, quantity=2)
        cart.add("pen", price=2, quantity=3)

        # act
        result = cart.total()

        # assert
        self.assertEqual(result, 26)

    def test_an_empty_cart_totals_zero(self):
        cart = Cart()
        self.assertEqual(cart.total(), 0)

suite = unittest.TestLoader().loadTestsFromTestCase(CartTests)
result = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
print("passed:", result.wasSuccessful())
print()
print("One behaviour per test, one assertion where possible, and the three")
print("phases in order. A test asserting six things fails on the first and")
print("tells you nothing about the other five.")

test("the arithmetic is right", 10 * 2 + 2 * 3, 26)
test("the suite passes", result.wasSuccessful(), True)

What it printed, when the build ran it

passed: True

One behaviour per test, one assertion where possible, and the three
phases in order. A test asserting six things fails on the first and
tells you nothing about the other five.
ok   the arithmetic is right
ok   the suite passes

What to take from this lesson

Test behaviour, not implementation — then a rewrite does not break the suite.
Write the expected value by hand. Computing it with the function's own formula tests nothing.
Coverage shows what ran, not what was checked. The valuable inputs are empty, one, the boundary, the duplicate, and one past the limit.
Test that a function does not modify its arguments.
For every bug: write the failing test first, watch it fail, then fix.
Name tests after the behaviour they guarantee, and keep them fast enough that people still run them.

39 Testing with unittest, run for real

A first suite, really executed

A failing test, and what it tells you

The assertions worth knowing

setUp: the same starting state for every test

subTest: many cases, one test method

Test doubles with unittest.mock

Running tests in a real project

On pytest

What to take from this lesson

40 What to test, and the tests that earn their keep

Test behaviour, not implementation

A test that cannot fail

Coverage is not correctness

The inputs worth testing

Write the failing test first, for every bug

Name the test after the behaviour

Fast and independent, or it will not get run

Tests are read more than they are written

What to take from this lesson

New writing, when it lands.

`setUp`: the same starting state for every test

`subTest`: many cases, one test method

Test doubles with `unittest.mock`