Shipping it all 2 written

Turning a folder of scripts into something another person can install and run, and one project built end to end.

49 Packaging: from a folder to something installable

A folder of scripts works on your machine, from one directory, for you. Making it work anywhere for anyone is a small amount of configuration and one idea: the code becomes a package with a declared name, and everything else refers to it by that name.

The layout

notes-cli/
├── pyproject.toml        the whole build configuration
├── README.md             what it is, how to use it
├── LICENSE               without this, nobody may legally use it
├── .gitignore
├── src/
│   └── notes/            the importable package
│       ├── __init__.py
│       ├── __main__.py   makes "python -m notes" work
│       ├── cli.py
│       ├── store.py
│       └── py.typed      empty; says "my type hints are real"
└── tests/
    ├── __init__.py
    └── test_store.py

Two things about that shape are worth defending, because both are commonly skipped.

Why the code lives in `src/`

If the package sits at the top level, then running anything from the project root puts your working copy on the import path automatically — so your tests import the folder you are editing, not the thing you installed. Every packaging mistake becomes invisible until someone else tries it. Putting it in src/ makes that import fail, which is the entire point.

The src layout, and the failure it buys you

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27import os, subprocess, sys
from pathlib import Path

root = Path("notes-cli")
(root / "src" / "notes").mkdir(parents=True)
(root / "src" / "notes" / "__init__.py").write_text(
    '__version__ = "1.0.0"\nfrom .core import shout\n', encoding="utf-8")
(root / "src" / "notes" / "core.py").write_text(
    "def shout(text):\n    return text.upper() + '!'\n", encoding="utf-8")

# From the project root, with no install, the package is not importable.
from_root = subprocess.run([sys.executable, "-c", "import notes"],
                           cwd=root, capture_output=True, text=True)
print("$ cd notes-cli && python3 -c 'import notes'")
print("  exit", from_root.returncode, "->", from_root.stderr.strip().splitlines()[-1])

# From inside src/, it is. This is what an install arranges, permanently.
from_src = subprocess.run(
    [sys.executable, "-c", "import notes; print(notes.__version__, notes.shout('hi'))"],
    cwd=root / "src", capture_output=True, text=True)
print()
print("$ cd notes-cli/src && python3 -c 'import notes; ...'")
print("  exit", from_src.returncode, "->", from_src.stdout.strip())

test("the root directory cannot import the package by accident", from_root.returncode, 1)
test("and the reason is stated plainly", "ModuleNotFoundError" in from_root.stderr, True)
test("while the package itself is fine", from_src.stdout.strip(), "1.0.0 HI!")

What it printed, when the build ran it

$ cd notes-cli && python3 -c 'import notes'
  exit 1 -> ModuleNotFoundError: No module named 'notes'

$ cd notes-cli/src && python3 -c 'import notes; ...'
  exit 0 -> 1.0.0 HI!
ok   the root directory cannot import the package by accident
ok   and the reason is stated plainly
ok   while the package itself is fine

That failure is a feature. With src/, the only way your tests can import the package is if it is really installed — so "works on my machine" and "works installed" stop being different states. It costs one directory.

`pyproject.toml`, field by field

One file, in TOML, replacing setup.py, setup.cfg and MANIFEST.in. It is data, not code — which is why it can be read back and checked.

Every field, parsed back out

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48import textwrap, tomllib
from pathlib import Path

Path("pyproject.toml").write_text(textwrap.dedent('''
    [build-system]
    # Who turns this directory into a wheel. Any PEP 517 backend will do:
    # hatchling, setuptools, flit-core, pdm-backend.
    requires = ["hatchling>=1.0"]
    build-backend = "hatchling.build"

    [project]
    name = "notes-cli"              # the install name: pip install notes-cli
    version = "1.0.0"
    description = "A tiny note keeper"
    readme = "README.md"
    requires-python = ">=3.11"
    license = "MIT"
    authors = [{ name = "Your Name", email = "you@example.com" }]
    keywords = ["notes", "cli"]
    dependencies = []               # runtime dependencies, deliberately none

    [project.optional-dependencies]
    dev = ["pytest>=8", "mypy>=1.8"]    # pip install -e ".[dev]"

    [project.scripts]
    notes = "notes.cli:main"        # becomes a "notes" command on the PATH

    [project.urls]
    Homepage = "https://example.com/notes"
    Source = "https://example.com/notes/src"
''').lstrip(), encoding="utf-8")

config = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))
project = config["project"]

print("install name:    ", project["name"])
print("import name:     ", "notes", "(they do not have to match, and often do not)")
print("version:         ", project["version"])
print("minimum Python:  ", project["requires-python"])
print("runtime deps:    ", project["dependencies"], "<- keep this list short")
print("dev extras:      ", project["optional-dependencies"]["dev"])
print("console script:  ", project["scripts"])
print("build backend:   ", config["build-system"]["build-backend"])

test("name and version are the only required fields", {"name", "version"} <= set(project), True)
test("the console script points at a module:function", project["scripts"]["notes"], "notes.cli:main")
test("there are no runtime dependencies", project["dependencies"], [])
test("and the file really is valid TOML, since it just parsed", isinstance(config, dict), True)

What it printed, when the build ran it

install name:     notes-cli
import name:      notes (they do not have to match, and often do not)
version:          1.0.0
minimum Python:   >=3.11
runtime deps:     [] <- keep this list short
dev extras:       ['pytest>=8', 'mypy>=1.8']
console script:   {'notes': 'notes.cli:main'}
build backend:    hatchling.build
ok   name and version are the only required fields
ok   the console script points at a module:function
ok   there are no runtime dependencies
ok   and the file really is valid TOML, since it just parsed

Pin your dependencies loosely, and your dev tools tightly. A library that says requests==2.31.0 cannot be installed alongside anything else that needs requests. Use >= with a lower bound you have actually tested in dependencies, and save exact pins for a lock file or for the dev extra.

`main.py`, and being runnable

A package with a __main__.py can be run with python -m. That works with no install, no PATH entry and no shell configuration, which makes it the first thing to get working.

python -m, and exit codes

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35import subprocess, sys, textwrap
from pathlib import Path

pkg = Path("notes")
pkg.mkdir()
(pkg / "__init__.py").write_text('__version__ = "1.0.0"\n', encoding="utf-8")
(pkg / "core.py").write_text("def shout(text):\n    return text.upper() + '!'\n", encoding="utf-8")
(pkg / "__main__.py").write_text(textwrap.dedent('''
    import sys
    from .core import shout

    def main(argv=None):
        argv = sys.argv[1:] if argv is None else argv
        if not argv:
            print("usage: python -m notes WORD", file=sys.stderr)
            return 2                      # a non-zero exit code, as a shell expects
        print(shout(argv[0]))
        return 0

    if __name__ == "__main__":
        sys.exit(main())
''').lstrip(), encoding="utf-8")

good = subprocess.run([sys.executable, "-m", "notes", "hello"], capture_output=True, text=True)
print("$ python3 -m notes hello")
print("  ", good.stdout.strip(), f"(exit {good.returncode})")

bad = subprocess.run([sys.executable, "-m", "notes"], capture_output=True, text=True)
print("$ python3 -m notes")
print("  ", bad.stderr.strip(), f"(exit {bad.returncode})")

test("it ran and printed the result", good.stdout.strip(), "HELLO!")
test("success is exit 0", good.returncode, 0)
test("and misuse is a non-zero exit, not a traceback", bad.returncode, 2)
test("with the message on stderr where it belongs", "usage:" in bad.stderr, True)

What it printed, when the build ran it

$ python3 -m notes hello
   HELLO! (exit 0)
$ python3 -m notes
   usage: python -m notes WORD (exit 2)
ok   it ran and printed the result
ok   success is exit 0
ok   and misuse is a non-zero exit, not a traceback
ok   with the message on stderr where it belongs

The [project.scripts] entry above does the nicer version of the same thing: on install, the packaging tool generates a small executable called notes that imports notes.cli and calls main. Same function, shorter command.

Installing it, which is the part this page cannot run

The editable install

shown, not run

1
2
3
4
5
6
7
8
9
10
11
12
13# In a virtual environment -- see lesson 21.
python3 -m venv .venv
source .venv/bin/activate

# An editable install: the package is importable from anywhere, and your edits
# take effect immediately. This is how you work on a project day to day.
pip install -e ".[dev]"

python -c "import notes; print(notes.__version__)"   # works from any directory
notes add "buy milk"                                  # the console script exists

# What editable actually did: put a path hook in site-packages pointing at src/.
pip show -f notes-cli

pip is shown rather than run: this course installs nothing, so that every page verifies itself on a machine with only a python3. The two things pip would prove -- that the package becomes importable from anywhere, and that the console script appears -- are demonstrated above with python -m and PYTHONPATH, which need no installer.

What an install leaves behind

An installed distribution is a directory of files plus a .dist-info folder holding its metadata. importlib.metadata reads that at runtime — which is how a program reports its own version without hard-coding it twice.

importlib.metadata, and where the version comes from

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35import importlib.metadata as metadata
import sys, textwrap
from pathlib import Path

# Written by hand here, to show what an installer would have produced. The
# directory name and the METADATA file are the whole interface.
site = Path("site-packages")
dist = site / "notes_cli-1.0.0.dist-info"
dist.mkdir(parents=True)
(dist / "METADATA").write_text(textwrap.dedent('''
    Metadata-Version: 2.1
    Name: notes-cli
    Version: 1.0.0
    Summary: A tiny note keeper
    Requires-Python: >=3.11
''').lstrip(), encoding="utf-8")
(dist / "RECORD").write_text("", encoding="utf-8")

sys.path.insert(0, str(site))

print("metadata.version('notes-cli'):", metadata.version("notes-cli"))
print("summary:                      ", metadata.metadata("notes-cli")["Summary"])
print("requires python:              ", metadata.metadata("notes-cli")["Requires-Python"])

try:
    metadata.version("something-else")
    missing = None
except metadata.PackageNotFoundError as exc:
    missing = str(exc)
print()
print("asking about something not installed:", missing)

test("the version came from the installed metadata, not the source", metadata.version("notes-cli"), "1.0.0")
test("other fields are readable too", metadata.metadata("notes-cli")["Summary"], "A tiny note keeper")
test("and a missing distribution raises rather than guessing", "No package metadata" in missing, True)

What it printed, when the build ran it

metadata.version('notes-cli'): 1.0.0
summary:                       A tiny note keeper
requires python:               >=3.11

asking about something not installed: No package metadata was found for something-else
ok   the version came from the installed metadata, not the source
ok   other fields are readable too
ok   and a missing distribution raises rather than guessing

One source of truth for the version. Either put it in pyproject.toml and read it back with importlib.metadata.version(__package__), or put __version__ in __init__.py and tell the backend to read it from there ([project] dynamic = ["version"]). Never both: they will disagree, and you will find out from a bug report.

The two artefacts

	sdist (`.tar.gz`)	wheel (`.whl`)
Contains	the source tree, plus `pyproject.toml`	the files exactly as they should be installed
Installing it	runs the build backend on the user's machine	unzips
Needs a compiler	possibly	no
Why publish it	so the source is available and buildable anywhere	so installation is fast and cannot fail

Publish both. A wheel for everyone, and an sdist because a platform you have never heard of will need to build its own.

Building and publishing

shown, not run

1
2
3
4
5
6
7
8
9
10pip install build twine          # both third-party

python -m build                   # writes dist/notes_cli-1.0.0-py3-none-any.whl
                                  #    and dist/notes_cli-1.0.0.tar.gz
twine check dist/*                # catches a malformed README before PyPI does
twine upload --repository testpypi dist/*    # rehearse on TestPyPI first
twine upload dist/*               # and then the real thing

# py3-none-any in that filename means: any Python 3, no ABI, any platform.
# Pure Python, so one wheel serves everyone.

Shell commands using third-party tools (build, twine), which this course does not install. The one thing on this page that does produce a real runnable artefact is the zipapp below, which is standard library.

Shipping without a package index

zipapp is in the standard library and makes a single executable file out of a package. No install, no index, no virtual environment — the recipient needs a python3 and nothing else. It is the right answer more often than it gets used.

A single runnable file, built and executed

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38import subprocess, sys, textwrap, zipapp, zipfile
from pathlib import Path

source = Path("build-root")
pkg = source / "notes"
pkg.mkdir(parents=True)
(pkg / "__init__.py").write_text('__version__ = "1.0.0"\n', encoding="utf-8")
(pkg / "core.py").write_text("def shout(text):\n    return text.upper() + '!'\n", encoding="utf-8")
(pkg / "cli.py").write_text(textwrap.dedent('''
    import sys
    from .core import shout

    def main():
        args = sys.argv[1:]
        print(shout(args[0]) if args else "usage: notes.pyz WORD")
''').lstrip(), encoding="utf-8")

# One call. main= names the entry point; compressed= shrinks it.
zipapp.create_archive(source, "notes.pyz", main="notes.cli:main", compressed=True)

archive = Path("notes.pyz")
with zipfile.ZipFile(archive) as z:
    contents = sorted(n for n in z.namelist() if n.endswith(".py"))

print("built notes.pyz:", archive.is_file(), f"({archive.stat().st_size} bytes)")
print("what is inside: ", contents)
print()

ran = subprocess.run([sys.executable, "notes.pyz", "shipped"], capture_output=True, text=True)
print("$ python3 notes.pyz shipped")
print("  ", ran.stdout.strip(), f"(exit {ran.returncode})")

test("the archive exists and is not empty", archive.stat().st_size > 0, True)
test("zipapp generated the __main__.py that calls the entry point",
     "__main__.py" in contents, True)
test("the package came along", "notes/core.py" in contents, True)
test("and the single file really runs", ran.stdout.strip(), "SHIPPED!")
test("with a clean exit", ran.returncode, 0)

What it printed, when the build ran it

built notes.pyz: True (753 bytes)
what is inside:  ['__main__.py', 'notes/__init__.py', 'notes/cli.py', 'notes/core.py']

$ python3 notes.pyz shipped
   SHIPPED! (exit 0)
ok   the archive exists and is not empty
ok   zipapp generated the __main__.py that calls the entry point
ok   the package came along
ok   and the single file really runs
ok   with a clean exit

The rest of the folder

Four files that are not code and are not optional.

README.md — what it does, how to install it, one example that works. Referenced by readme = and shown on PyPI.
LICENSE — without one, nobody may legally use your code. MIT or Apache-2.0 unless you have a reason.
.gitignore — at minimum __pycache__/, *.pyc, .venv/, dist/, build/, *.egg-info/.
src/notes/py.typed — an empty file that tells type checkers your annotations are intended to be used. Without it they ignore your hints entirely.

The order to do this in

Move the code into src/yourpackage/ and add __init__.py.
Write pyproject.toml with name and version. Nothing else is required.
Add __main__.py and check python -m yourpackage works.
Make a virtual environment and pip install -e ".[dev]".
Run the tests against the installed package, from a different directory.
Add README, LICENSE, py.typed.
python -m build, then twine check dist/*.
Upload to TestPyPI, install from there into a clean environment, and run it.
Then the real index.

Step 8 is the one everybody skips and the one that catches the mistakes — a missing subpackage, a data file that was never included, an import that only worked because of your working directory.

pyproject.toml is the only configuration file you need, and it is data you can parse.
Put the package in src/ so an accidental import of your working copy fails loudly.
__main__.py gives you python -m for free; [project.scripts] turns that into a command.
Return exit codes from main and print errors to stderr.
Ship a wheel and an sdist. Read your own version with importlib.metadata, declared in one place.
zipapp makes one runnable file with no installer, using only the standard library.
Install from TestPyPI into a clean environment before you believe any of it.

50 One real project, end to end

Everything in this course, used once, on one program. A note keeper with a data layer, a command line, a test suite and a shippable artefact — built out of the standard library and nothing else, and run for real at the bottom of this page.

What it does, before any code

Deciding this first is not ceremony. Every choice below follows from it, and the ones that are left out matter as much as the ones that are in.

Decision	Choice	Because
Interface	a command line	Testable without a browser, and honest about scope.
Storage	`sqlite3`	Standard library, one file, real SQL, real transactions.
Errors	one exception type of our own	The CLI can catch exactly our failures and nothing else.
Config	a `--db` flag	Nothing global. Tests pass `:memory:`.
Not doing	users, sync, a server	Each would double the program. Ship the small one.

notes-cli/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│   └── notes/
│       ├── __init__.py       version, and the public names
│       ├── __main__.py       python -m notes
│       ├── errors.py         one exception type
│       ├── store.py          all the SQL, and nothing else
│       └── cli.py            argument parsing and output, and nothing else
└── tests/
    ├── __init__.py
    ├── test_store.py
    └── test_cli.py

The split between store.py and cli.py is the one structural decision worth defending: the store knows no argument parsing and prints nothing; the CLI writes no SQL. That is what makes the store testable without a subprocess, and what would let a web interface reuse it unchanged.

The data layer

A context manager for the connection (lesson 24), parameterised queries everywhere, and failures raised as our own exception type.

store.py, with its own exercise

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73import sqlite3
from contextlib import contextmanager

class NoteError(Exception):
    """Anything the user did wrong. The CLI turns this into an exit code."""

SCHEMA = """
CREATE TABLE IF NOT EXISTS notes (
    id    INTEGER PRIMARY KEY,
    body  TEXT    NOT NULL,
    done  INTEGER NOT NULL DEFAULT 0
)"""

@contextmanager
def connect(path=":memory:"):
    conn = sqlite3.connect(path)
    conn.row_factory = sqlite3.Row          # rows behave like dicts
    try:
        conn.executescript(SCHEMA)
        yield conn
        conn.commit()                       # only on a clean exit
    finally:
        conn.close()                        # always

def add(conn, body):
    if not body.strip():
        raise NoteError("a note needs a body")
    return conn.execute("INSERT INTO notes (body) VALUES (?)", (body,)).lastrowid

def complete(conn, note_id):
    changed = conn.execute("UPDATE notes SET done = 1 WHERE id = ?", (note_id,)).rowcount
    if changed == 0:
        raise NoteError(f"no note with id {note_id}")

def listing(conn, done=None):
    sql, args = "SELECT id, body, done FROM notes", ()
    if done is not None:
        sql, args = sql + " WHERE done = ?", (int(done),)
    return [dict(row) for row in conn.execute(sql + " ORDER BY id", args)]

# Exercise it.
with connect() as conn:
    first = add(conn, "buy milk")
    second = add(conn, "write lesson 50")
    complete(conn, first)

    for row in listing(conn):
        print(f"{row['id']:>3} [{'x' if row['done'] else ' '}] {row['body']}")

    done_ids = [r["id"] for r in listing(conn, done=True)]
    open_ids = [r["id"] for r in listing(conn, done=False)]

    try:
        add(conn, "   ")
        blank = None
    except NoteError as exc:
        blank = str(exc)

    try:
        complete(conn, 99)
        missing = None
    except NoteError as exc:
        missing = str(exc)

print()
print("rejecting a blank note: ", blank)
print("completing a missing id:", missing)

test("ids are assigned by the database", (first, second), (1, 2))
test("completing one moves it into the done list", done_ids, [1])
test("and out of the open list", open_ids, [2])
test("a blank body is refused", blank, "a note needs a body")
test("so is an id that does not exist", missing, "no note with id 99")

What it printed, when the build ran it

  1 [x] buy milk
  2 [ ] write lesson 50

rejecting a blank note:  a note needs a body
completing a missing id: no note with id 99
ok   ids are assigned by the database
ok   completing one moves it into the done list
ok   and out of the open list
ok   a blank body is refused
ok   so is an id that does not exist

The commit is inside the try, the close is in the finally. That ordering is the whole value of the context manager: an exception raised inside the with block skips the commit and still closes the connection. You get atomicity and cleanup from four lines, and no caller can forget either.

Why every query has a `?` in it

This is the one place where getting it wrong is a security failure rather than a bug. Both halves below are run.

One value, two ways to use it

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31import sqlite3

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE users (name TEXT)")
conn.executemany("INSERT INTO users VALUES (?)", [("ada",), ("alan",)])

def table_exists():
    return conn.execute(
        "SELECT count(*) FROM sqlite_master WHERE type='table' AND name='users'"
    ).fetchone()[0]

hostile = "x'; DROP TABLE users; --"

print("table exists:", table_exists())
print()

# Parameterised: the value is data. It is compared against, not executed.
matched = conn.execute("SELECT count(*) FROM users WHERE name = ?", (hostile,)).fetchone()[0]
after_parameterised = table_exists()
print("parameterised, looking for that name:")
print("   rows matched:", matched, "  table still exists:", after_parameterised)

# Concatenated: the value becomes part of the statement.
conn.executescript("SELECT * FROM users WHERE name = '" + hostile + "'")
print()
print("the same value concatenated into the SQL:")
print("   table still exists:", table_exists(), " <- it is gone")

test("the parameterised query found nothing, because no user is called that", matched, 0)
test("and left the table alone", after_parameterised, 1)
test("the concatenated one destroyed the table", table_exists(), 0)

What it printed, when the build ran it

table exists: 1

parameterised, looking for that name:
   rows matched: 0   table still exists: 1

the same value concatenated into the SQL:
   table still exists: 0  <- it is gone
ok   the parameterised query found nothing, because no user is called that
ok   and left the table alone
ok   the concatenated one destroyed the table

There is no safe way to build SQL with string formatting. Not %, not .format(), not an f-string, not "but I escaped the quotes". Pass values as parameters and let the driver do it. The only part of a query you may ever build by concatenation is structure you chose yourself — a column name from a fixed allow-list, never anything from a user.

The command line

argparse for parsing, and main returning an exit code rather than calling sys.exit — so the tests can call it directly.

cli.py, driven in-process

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85import argparse, io, json, sqlite3, sys
from contextlib import contextmanager, redirect_stdout, redirect_stderr

# ---- the store, condensed (same code as above) --------------------------------
class NoteError(Exception): pass

@contextmanager
def connect(path=":memory:"):
    conn = sqlite3.connect(path)
    conn.row_factory = sqlite3.Row
    try:
        conn.executescript("CREATE TABLE IF NOT EXISTS notes ("
                           "id INTEGER PRIMARY KEY, body TEXT NOT NULL, "
                           "done INTEGER NOT NULL DEFAULT 0)")
        yield conn
        conn.commit()
    finally:
        conn.close()

def add(conn, body):
    if not body.strip():
        raise NoteError("a note needs a body")
    return conn.execute("INSERT INTO notes (body) VALUES (?)", (body,)).lastrowid

def listing(conn, done=None):
    sql, args = "SELECT id, body, done FROM notes", ()
    if done is not None:
        sql, args = sql + " WHERE done = ?", (int(done),)
    return [dict(r) for r in conn.execute(sql + " ORDER BY id", args)]

# ---- cli.py ------------------------------------------------------------------
def build_parser():
    parser = argparse.ArgumentParser(prog="notes", description="a tiny note keeper")
    parser.add_argument("--db", default=":memory:", help="database file")
    sub = parser.add_subparsers(dest="command", required=True)

    added = sub.add_parser("add", help="add a note")
    added.add_argument("body")

    shown = sub.add_parser("list", help="show notes")
    shown.add_argument("--json", action="store_true", help="machine-readable output")
    return parser

def render(rows):
    return "\n".join(f"{r['id']:>3} [{'x' if r['done'] else ' '}] {r['body']}" for r in rows)

def main(argv=None, connect=connect):
    args = build_parser().parse_args(argv)
    try:
        with connect(args.db) as conn:
            if args.command == "add":
                print("added", add(conn, args.body))
            else:
                add(conn, "seeded, since :memory: starts empty")
                rows = listing(conn)
                print(json.dumps(rows) if args.json else render(rows))
    except NoteError as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 1
    return 0

# ---- calling main directly, which is why it returns instead of exiting -------
def run(argv):
    out, err = io.StringIO(), io.StringIO()
    with redirect_stdout(out), redirect_stderr(err):
        code = main(argv)
    return code, out.getvalue().strip(), err.getvalue().strip()

for argv in (["add", "buy milk"], ["list"], ["list", "--json"], ["add", "  "]):
    code, out, err = run(argv)
    print(f"$ notes {' '.join(argv)}")
    print("  ", out or err, f"(exit {code})")

parsed = build_parser().parse_args(["--db", "notes.db", "add", "hello"])
print()
print("parsed arguments:", vars(parsed))

ok_code, _, _ = run(["add", "something"])
bad_code, _, bad_err = run(["add", ""])

test("a successful command exits 0", ok_code, 0)
test("a rejected one exits 1", bad_code, 1)
test("with the reason on stderr", bad_err, "error: a note needs a body")
test("argparse produced the flags and the subcommand",
     vars(parsed), {"db": "notes.db", "command": "add", "body": "hello"})

What it printed, when the build ran it

$ notes add buy milk
   added 1 (exit 0)
$ notes list
   1 [ ] seeded, since :memory: starts empty (exit 0)
$ notes list --json
   [{"id": 1, "body": "seeded, since :memory: starts empty", "done": 0}] (exit 0)
$ notes add   
   error: a note needs a body (exit 1)

parsed arguments: {'db': 'notes.db', 'command': 'add', 'body': 'hello'}
ok   a successful command exits 0
ok   a rejected one exits 1
ok   with the reason on stderr
ok   argparse produced the flags and the subcommand

main(argv=None) returning an int is the shape to copy. sys.exit belongs in the if __name__ == "__main__": line and nowhere else. A main that exits cannot be called from a test without catching SystemExit, and a main that reads sys.argv itself cannot be called with different arguments twice.

A bug, and the test that stops it coming back

The shape from lesson 40, on a real defect: list --json is meant to be machine-readable, and the version below is not.

The regression test comes first

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35import io, json, unittest

def render_json_broken(rows):
    # Looks like JSON. Is not JSON: single quotes, and True instead of true.
    return str(rows)

def render_json(rows):
    return json.dumps(rows)

class JsonOutputTests(unittest.TestCase):
    """Written against the bug, and kept afterwards."""

    rows = [{"id": 1, "body": "buy milk", "done": 1}]

    def test_output_parses_as_json(self):
        self.assertEqual(json.loads(render_json(self.rows)), self.rows)

    def test_the_old_version_did_not(self):
        with self.assertRaises(json.JSONDecodeError):
            json.loads(render_json_broken(self.rows))

    def test_an_empty_list_is_still_valid_json(self):
        self.assertEqual(json.loads(render_json([])), [])

suite = unittest.TestLoader().loadTestsFromTestCase(JsonOutputTests)
stream = io.StringIO()
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)
print(stream.getvalue())

print("what the broken version produced:", render_json_broken(JsonOutputTests.rows))
print("what the fixed version produces: ", render_json(JsonOutputTests.rows))

test("the suite passed", result.wasSuccessful(), True)
test("three tests ran", result.testsRun, 3)
test("the fixed output round-trips", json.loads(render_json([{"a": 1}])), [{"a": 1}])

What it printed, when the build ran it

test_an_empty_list_is_still_valid_json (__main__.JsonOutputTests.test_an_empty_list_is_still_valid_json) ... ok
test_output_parses_as_json (__main__.JsonOutputTests.test_output_parses_as_json) ... ok
test_the_old_version_did_not (__main__.JsonOutputTests.test_the_old_version_did_not) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

what the broken version produced: [{'id': 1, 'body': 'buy milk', 'done': 1}]
what the fixed version produces:  [{"id": 1, "body": "buy milk", "done": 1}]
ok   the suite passed
ok   three tests ran
ok   the fixed output round-trips

All of it, on disk, run as a program

Everything above has been exercised in a panel. This panel writes it out as the real directory tree, then runs three separate programs against it: the test suite, the command line, and a zipapp build of the finished thing.

The project, written out and run

ran at build time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258import os, subprocess, sys, textwrap
from pathlib import Path

root = Path("notes-cli")
(root / "src" / "notes").mkdir(parents=True)
(root / "tests").mkdir()

files = {
    "src/notes/__init__.py": '''
        __version__ = "1.0.0"
        from .errors import NoteError
        from .store import add, complete, connect, listing

        __all__ = ["NoteError", "add", "complete", "connect", "listing", "__version__"]
    ''',
    "src/notes/errors.py": '''
        class NoteError(Exception):
            """Something the user asked for that cannot be done."""
    ''',
    "src/notes/store.py": '''
        import sqlite3
        from contextlib import contextmanager

        from .errors import NoteError

        SCHEMA = """
        CREATE TABLE IF NOT EXISTS notes (
            id    INTEGER PRIMARY KEY,
            body  TEXT    NOT NULL,
            done  INTEGER NOT NULL DEFAULT 0
        )"""

        @contextmanager
        def connect(path=":memory:"):
            conn = sqlite3.connect(path)
            conn.row_factory = sqlite3.Row
            try:
                conn.executescript(SCHEMA)
                yield conn
                conn.commit()
            finally:
                conn.close()

        def add(conn, body):
            if not body.strip():
                raise NoteError("a note needs a body")
            return conn.execute("INSERT INTO notes (body) VALUES (?)", (body,)).lastrowid

        def complete(conn, note_id):
            changed = conn.execute("UPDATE notes SET done = 1 WHERE id = ?", (note_id,)).rowcount
            if changed == 0:
                raise NoteError(f"no note with id {note_id}")

        def listing(conn, done=None):
            sql, args = "SELECT id, body, done FROM notes", ()
            if done is not None:
                sql, args = sql + " WHERE done = ?", (int(done),)
            return [dict(row) for row in conn.execute(sql + " ORDER BY id", args)]
    ''',
    "src/notes/cli.py": '''
        import argparse
        import json
        import sys

        from . import __version__, store
        from .errors import NoteError

        def build_parser():
            parser = argparse.ArgumentParser(prog="notes", description="a tiny note keeper")
            parser.add_argument("--version", action="version", version=__version__)
            parser.add_argument("--db", default="notes.db")
            sub = parser.add_subparsers(dest="command", required=True)
            added = sub.add_parser("add"); added.add_argument("body")
            finished = sub.add_parser("done"); finished.add_argument("id", type=int)
            shown = sub.add_parser("list"); shown.add_argument("--json", action="store_true")
            return parser

        def render(rows):
            if not rows:
                return "no notes yet"
            return "\\n".join(
                f"{r['id']:>3} [{'x' if r['done'] else ' '}] {r['body']}" for r in rows)

        def main(argv=None):
            args = build_parser().parse_args(argv)
            try:
                with store.connect(args.db) as conn:
                    if args.command == "add":
                        print("added", store.add(conn, args.body))
                    elif args.command == "done":
                        store.complete(conn, args.id)
                        print("done", args.id)
                    else:
                        rows = store.listing(conn)
                        print(json.dumps(rows) if args.json else render(rows))
            except NoteError as exc:
                print(f"error: {exc}", file=sys.stderr)
                return 1
            return 0
    ''',
    "src/notes/__main__.py": '''
        import sys

        from .cli import main

        if __name__ == "__main__":
            sys.exit(main())
    ''',
    "tests/__init__.py": "",
    "tests/test_store.py": '''
        import unittest

        from notes import NoteError, add, complete, connect, listing

        class StoreTests(unittest.TestCase):
            def setUp(self):
                self.ctx = connect()
                self.conn = self.ctx.__enter__()

            def tearDown(self):
                self.ctx.__exit__(None, None, None)

            def test_add_returns_the_new_id(self):
                self.assertEqual(add(self.conn, "milk"), 1)

            def test_a_blank_body_is_rejected(self):
                with self.assertRaises(NoteError):
                    add(self.conn, "   ")

            def test_completing_a_missing_note_raises(self):
                with self.assertRaises(NoteError):
                    complete(self.conn, 99)

            def test_listing_filters_on_done(self):
                add(self.conn, "a")
                add(self.conn, "b")
                complete(self.conn, 1)
                self.assertEqual([r["id"] for r in listing(self.conn, done=True)], [1])
                self.assertEqual([r["id"] for r in listing(self.conn, done=False)], [2])

            def test_a_hostile_body_is_stored_as_text(self):
                body = "x'; DROP TABLE notes; --"
                add(self.conn, body)
                self.assertEqual(listing(self.conn)[0]["body"], body)
    ''',
    "tests/test_cli.py": '''
        import io
        import unittest
        from contextlib import redirect_stderr, redirect_stdout

        from notes.cli import build_parser, main, render

        class CliTests(unittest.TestCase):
            def run_cli(self, argv):
                # Global options come before the subcommand, so --db goes first.
                out, err = io.StringIO(), io.StringIO()
                with redirect_stdout(out), redirect_stderr(err):
                    code = main(["--db", ":memory:", *argv])
                return code, out.getvalue().strip(), err.getvalue().strip()

            def test_adding_succeeds(self):
                code, out, _ = self.run_cli(["add", "milk"])
                self.assertEqual((code, out), (0, "added 1"))

            def test_a_blank_note_is_an_error_exit(self):
                code, _, err = self.run_cli(["add", " "])
                self.assertEqual(code, 1)
                self.assertIn("a note needs a body", err)

            def test_an_empty_list_says_so(self):
                self.assertEqual(render([]), "no notes yet")

            def test_a_missing_subcommand_is_rejected(self):
                with self.assertRaises(SystemExit):
                    build_parser().parse_args([])
    ''',
    "pyproject.toml": '''
        [build-system]
        requires = ["hatchling"]
        build-backend = "hatchling.build"

        [project]
        name = "notes-cli"
        version = "1.0.0"
        description = "A tiny note keeper"
        requires-python = ">=3.11"
        dependencies = []

        [project.scripts]
        notes = "notes.cli:main"
    ''',
    "README.md": '''
        # notes-cli

        A tiny note keeper. Standard library only.

            pip install notes-cli
            notes add "buy milk"
            notes list
    ''',
}

for name, body in files.items():
    (root / name).write_text(textwrap.dedent(body).lstrip(), encoding="utf-8")

written = sorted(str(p.relative_to(root)) for p in root.rglob("*") if p.is_file())
print("files written:")
for name in written:
    print("  ", name)

env = {**os.environ, "PYTHONPATH": str((root / "src").resolve())}

print()
print("=== python3 -m unittest discover -s tests -t . ===")
tests = subprocess.run(
    [sys.executable, "-m", "unittest", "discover", "-s", "tests", "-t", ".", "-v"],
    cwd=root, capture_output=True, text=True, env=env)
print(tests.stderr.strip())

print()
print("=== the command line, one invocation at a time ===")
db = str((root / "notes.db").resolve())
transcript = []
for argv in (["--version"], ["list"], ["add", "buy milk"], ["add", "write lesson 50"],
             ["done", "1"], ["list"], ["list", "--json"], ["done", "99"], ["add", "   "]):
    ran = subprocess.run([sys.executable, "-m", "notes", "--db", db, *argv],
                         cwd=root, capture_output=True, text=True, env=env)
    shown = (ran.stdout or ran.stderr).rstrip()
    transcript.append((tuple(argv), ran.returncode))
    print(f"$ notes {' '.join(argv)}")
    for line in shown.splitlines():
        print("   ", line)
    print(f"    (exit {ran.returncode})")

print()
print("=== zipapp, and running the artefact ===")
import zipapp
zipapp.create_archive(root / "src", "notes.pyz", main="notes.cli:main", compressed=True)
shipped = subprocess.run([sys.executable, "notes.pyz", "--db", db, "list"],
                         capture_output=True, text=True)
print("notes.pyz built:", Path("notes.pyz").stat().st_size, "bytes")
print("$ python3 notes.pyz --db notes.db list")
for line in shipped.stdout.rstrip().splitlines():
    print("   ", line)
print(f"    (exit {shipped.returncode})")

codes = dict(transcript)

test("the whole test suite passed", tests.returncode, 0)
test("all nine tests ran", "Ran 9 tests" in tests.stderr, True)
test("and none of them failed", "FAILED" in tests.stderr, False)
test("an empty database lists cleanly", codes[("list",)], 0)
test("adding works", codes[("add", "buy milk")], 0)
test("completing a note that does not exist is an error", codes[("done", "99")], 1)
test("so is a blank note", codes[("add", "   ")], 1)
test("--version exits 0", codes[("--version",)], 0)
test("the zipapp reads the same database the CLI wrote", shipped.returncode, 0)
test("and shows both notes, one of them done", shipped.stdout.count("[x]"), 1)

What it printed, when the build ran it

files written:
   README.md
   pyproject.toml
   src/notes/__init__.py
   src/notes/__main__.py
   src/notes/cli.py
   src/notes/errors.py
   src/notes/store.py
   tests/__init__.py
   tests/test_cli.py
   tests/test_store.py

=== python3 -m unittest discover -s tests -t . ===
test_a_blank_note_is_an_error_exit (tests.test_cli.CliTests.test_a_blank_note_is_an_error_exit) ... ok
test_a_missing_subcommand_is_rejected (tests.test_cli.CliTests.test_a_missing_subcommand_is_rejected) ... usage: notes [-h] [--version] [--db DB] {add,done,list} ...
notes: error: the following arguments are required: command
ok
test_adding_succeeds (tests.test_cli.CliTests.test_adding_succeeds) ... ok
test_an_empty_list_says_so (tests.test_cli.CliTests.test_an_empty_list_says_so) ... ok
test_a_blank_body_is_rejected (tests.test_store.StoreTests.test_a_blank_body_is_rejected) ... ok
test_a_hostile_body_is_stored_as_text (tests.test_store.StoreTests.test_a_hostile_body_is_stored_as_text) ... ok
test_add_returns_the_new_id (tests.test_store.StoreTests.test_add_returns_the_new_id) ... ok
test_completing_a_missing_note_raises (tests.test_store.StoreTests.test_completing_a_missing_note_raises) ... ok
test_listing_filters_on_done (tests.test_store.StoreTests.test_listing_filters_on_done) ... ok

----------------------------------------------------------------------
Ran 9 tests in 0.007s

OK

=== the command line, one invocation at a time ===
$ notes --version
    1.0.0
    (exit 0)
$ notes list
    no notes yet
    (exit 0)
$ notes add buy milk
    added 1
    (exit 0)
$ notes add write lesson 50
    added 2
    (exit 0)
$ notes done 1
    done 1
    (exit 0)
$ notes list
      1 [x] buy milk
      2 [ ] write lesson 50
    (exit 0)
$ notes list --json
    [{"id": 1, "body": "buy milk", "done": 1}, {"id": 2, "body": "write lesson 50", "done": 0}]
    (exit 0)
$ notes done 99
    error: no note with id 99
    (exit 1)
$ notes add    
    error: a note needs a body
    (exit 1)

=== zipapp, and running the artefact ===
notes.pyz built: 2195 bytes
$ python3 notes.pyz --db notes.db list
      1 [x] buy milk
      2 [ ] write lesson 50
    (exit 0)
ok   the whole test suite passed
ok   all nine tests ran
ok   and none of them failed
ok   an empty database lists cleanly
ok   adding works
ok   completing a note that does not exist is an error
ok   so is a blank note
ok   --version exits 0
ok   the zipapp reads the same database the CLI wrote
ok   and shows both notes, one of them done

Nine tests, nine command-line invocations with their real exit codes, and a single-file build that reads the database the CLI just wrote. None of that output was typed by hand.

What was used, and where it came from

In this project	Introduced in
Functions, default and keyword arguments	Part 4
Packages, `__init__.py`, relative imports	Part 5
An exception type of our own, caught at one boundary	Part 6
`@contextmanager`, and cleanup that cannot be skipped	Part 6
Dict rows, comprehensions, f-string formatting	Parts 2 and 3
`unittest`, `setUp`, `assertRaises`, testing behaviour	Part 12
`pyproject.toml`, `python -m`, `zipapp`	Part 15

What a real version would add next

Migrations. CREATE TABLE IF NOT EXISTS works exactly once. The second schema change needs a version number in the database and a list of upgrade steps.
Type checking. Annotate the store, add a py.typed, and run mypy --strict in CI. Lesson 38 is the reason this is worth doing early.
Logging, not printing. print is the interface; logging is the diagnostics. Mixing them means you cannot turn one off.
A continuous integration file. Run the tests on every push, on the oldest Python you claim to support.
Somebody else's machine. Install from TestPyPI into an empty virtual environment and run it. This finds the mistakes nothing else does.

The checks worth automating

shown, not run

1
2
3
4
5
6
7# What the CI step for this project looks like. Not run here: there is no
# runner, and the tools after the first line are third-party.

python3 -m unittest discover -s tests -t .     # this one needs nothing installed
python3 -m mypy --strict src/notes
python3 -m ruff check src tests
python3 -m build && twine check dist/*

Shell commands, three of which use third-party tools this course does not install. The first line is exactly what the panel above ran, and its output is on this page.

The end of the course

Fifty lessons ago the first panel printed a string. This one builds a package, proves it with a test suite, runs it from a command line and ships it as a single file — and every line of output on both pages was produced by the same interpreter at the same moment the page was written.

That is the habit worth keeping, more than any particular feature covered here. The language will change; three of the things on these pages contradict what was true two releases ago, and the bytecode lesson will be partly wrong by 3.16. What does not go stale is running the code and reading what it actually says. Every claim in this course was checked that way, several were wrong the first time, and the interpreter said so.

Decide what the program is, and what it is not, before writing any of it.
Separate the layer that knows the data from the layer that knows the user. Test the first without a subprocess.
Parameterise every query. There is no safe formatting.
main(argv=None) -> int, errors on stderr, sys.exit only under __main__.
Write the failing test before the fix, and keep it.
Make it installable and run it somewhere that is not your working directory.
Then keep measuring, and keep reading the output. That is the whole method.

49 Packaging: from a folder to something installable

The layout

Why the code lives in src/

pyproject.toml, field by field

__main__.py, and being runnable

Installing it, which is the part this page cannot run

What an install leaves behind

The two artefacts

Shipping without a package index

The rest of the folder

The order to do this in

50 One real project, end to end

What it does, before any code

The data layer

Why every query has a ? in it

The command line

A bug, and the test that stops it coming back

All of it, on disk, run as a program

What was used, and where it came from

What a real version would add next

The end of the course

New writing, when it lands.

Why the code lives in `src/`

`pyproject.toml`, field by field

`main.py`, and being runnable

Why every query has a `?` in it