[wic][PATCH v3 07/10] tests/docs: add the test-authoring guide
Trevor Woerner <[email protected]> Wed, 1 Jul 2026 03:40:27 -0400
| Newsgroups | org.yoctoproject.lists.yocto-patches |
|---|---|
| Message-ID | <[email protected]> |
A suite is only as good as the tests added to it over time, so it needs
a guide that says where a new test goes, what one looks like, and how
to choose its inputs and assertions. This commit adds
tests/docs/authoring.md.
The guide covers:
- where a test goes: one test_<area>.py per wic module under
tests/unit/, extending an existing file or starting a new one;
- the shape of a test: a small worked example, a preference for
parametrize over in-test loops so each input/output pair is its own
named case, and use of the tmp_path fixture for scratch files;
- asserting the correct behaviour rather than whatever wic currently
does, so a test never bakes in a bug; when a test exposes a defect,
the source fix lands in the same change so the test passes;
- probing the boundaries: a checklist of input classes worth covering
for numbers, strings, paths, types/shape, and state, each asserting
a specific value or exception rather than merely "it did not
crash";
- keeping the assertion strong: never weaken an assertion to get
green;
- how to run just the test you wrote, with and without coverage.
AI-Generated: codex/claude-opus 4.7 (xhigh)
Signed-off-by: Trevor Woerner <[email protected]>
---
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks
the work into a reviewable series, and this patch is one step of it.
---
tests/docs/authoring.md | 124 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 124 insertions(+)
create mode 100644 tests/docs/authoring.md
diff --git a/tests/docs/authoring.md b/tests/docs/authoring.md
new file mode 100644
index 000000000000..c60cceb2d32d
--- /dev/null
+++ b/tests/docs/authoring.md
@@ -0,0 +1,124 @@
+# Authoring a unit test
+
+Unit tests live under `tests/unit/`, one module per area of wic, named
+`test_<area>.py`. They import the wic module under test directly and
+run in-process, so they need no host tools and no build environment.
+
+## Contents
+
+- [Where a test goes](#where-a-test-goes)
+- [Shape of a test](#shape-of-a-test)
+- [Assert the correct behaviour](#assert-the-correct-behaviour)
+- [Probe the boundaries](#probe-the-boundaries)
+- [Keep the assertion strong](#keep-the-assertion-strong)
+- [Running what you wrote](#running-what-you-wrote)
+
+## Where a test goes
+
+Group tests by the wic module they exercise, one `test_<area>.py` per
+module. Start a new file when you begin covering a module that has no
+file yet; otherwise extend the existing one.
+
+## Shape of a test
+
+```python
+import pytest
+
+from wic.ksparser import sizetype
+
+
+class TestSizetype:
+ # sizetype(default, size_in_bytes=False) returns a parser f(arg);
+ # with default "M", a bare number is read as mebibytes-in-KiB.
+ def test_plain_value_uses_default_unit(self):
+ parse = sizetype("M")
+ assert parse("100") == 100 * 1024
+
+ @pytest.mark.parametrize("arg,expected", [
+ ("1K", 1),
+ ("1G", 1 * 1024 * 1024),
+ ("0", 0),
+ ])
+ def test_suffixes(self, arg, expected):
+ assert sizetype("M")(arg) == expected
+```
+
+Prefer `@pytest.mark.parametrize` to express each input/output pair as
+its own case rather than looping inside one test, so a single bad
+value shows up as one named failure.
+
+Use `pytest`'s `tmp_path` fixture for any scratch files; a passing
+test's scratch directory is removed automatically (see the retention
+policy in `pyproject.toml`), and a failing one is kept for inspection.
+
+## Assert the correct behaviour
+
+Every assertion states the behaviour wic is *expected* to provide,
+never the behaviour it currently happens to have. A test that locks in
+a wrong result is worse than no test: it gives false confidence and it
+turns red exactly when someone repairs the code.
+
+When a test exposes a defect, the fix to the wic source lands in the
+same change as the test, so the test asserts the correct behaviour and
+passes. Do not commit a test that asserts a known-wrong result.
+
+Wrong -- bakes in the bug:
+
+```python
+# value lost its comma; asserting the broken output
+assert result["file"] == "s3://bucket/a"
+```
+
+Right -- asserts the correct output (and the fix lands with it):
+
+```python
+assert result["file"] == "s3://bucket/a,b.img"
+```
+
+## Probe the boundaries
+
+For a parameter, probe the boundary and just past it rather than only
+the comfortable middle of its range. The classes worth covering:
+
+- numbers: empty, zero, negative, the maximum and one beyond it,
+ non-power-of-two and non-multiple-of-1024/1000 values, primes,
+ fractional values where an integer is assumed, `0` versus `0.0`
+ versus `"0"`, off-by-one at a block or sector boundary, and very
+ large magnitudes near `2**31` and `2**63`
+- strings: the empty string, whitespace only, embedded spaces, tabs,
+ newlines and CRLF endings, non-ASCII characters, shell
+ metacharacters in a value that becomes part of a command line,
+ leading-dash values that look like options, over-delimited or
+ malformed forms (stray commas, doubled or missing separators), and
+ very long values
+- paths: traversal (`../`), doubled slashes, a trailing slash or its
+ absence, `.` and `..` components, broken or looping symlinks, and a
+ non-existent path
+- types and shape: each wrong type the parameter might receive (a
+ string where an integer is expected, a list where a string is
+ expected, `None`, and so on), the wrong argument count, and
+ duplicate keys or entries
+- state: a second call that must not see stale cached data, a
+ `cache=False` path that must evict, calling an operation twice, and
+ finalising a half-constructed object
+
+Each case asserts a specific outcome -- an exact value, or a specific
+exception -- rather than merely "it did not crash." A test that only
+checks for the absence of an exception will pass against badly wrong
+output.
+
+## Keep the assertion strong
+
+If a test is hard to make pass, the answer is in the code or in
+understanding the correct behaviour, never in weakening the assertion
+to something vague enough to pass. Loosening an assertion to get green
+is how a suite quietly stops catching regressions.
+
+## Running what you wrote
+
+```bash
+tests/run-tests.sh tests/unit/test_<area>.py -v
+tests/run-tests.sh --coverage tests/unit/test_<area>.py
+```
+
+A new test should leave the suite green, with zero failures.
--
2.50.0.173.g8b6f19ccfc3a