Re: [yocto-patches] [wic][PATCH v3 05/10] tests: add ruff linting to run-tests.sh

Paul Barker <[email protected]> Mon, 06 Jul 2026 09:40:52 +0100
Newsgroups	org.yoctoproject.lists.yocto-patches
Message-ID	<[email protected]>
On Wed, 2026-07-01 at 03:40 -0400, Trevor Woerner via
lists.yoctoproject.org wrote:
> A test suite is only trustworthy if its own code is clean, so this
> commit brings ruff into the runner and holds the test tree to a clean
> bar. It also makes it easy to preview what ruff thinks of the wic
> source, without yet enforcing it.
> 
> pyproject.toml gains ruff in the tests extra and a [tool.ruff] section.
> The configuration is deliberately minimal for now: the test suite is
> the only tree under an enforced clean bar; the wic source under src/ is
> not yet ruff-clean and is reported, not gated.
> 
> run-tests.sh gains two lint modes, each used on its own:
> 
>   - --lint-tests runs ruff over tests/ and exits. The test suite must
>     report nothing; a finding here is a bug in our own test code and is
>     expected to be fixed.
> 
>   - --lint-src runs ruff over src/ and exits. The source is not yet
>     ruff-clean, so this is a preview: the runner prints ruff's findings
>     and exits with its status, but nothing in the suite asserts on
>     them. Keeping the two trees on separate flags means cleaning up the
>     source later does not disturb the test-tree gate.
> 
> A lint mode cannot be combined with coverage, with the other lint mode,
> or with pytest arguments; the runner rejects such combinations loudly
> rather than silently dropping the extras. If ruff is not installed it
> fails with the install command.
> 
> tests/docs/linting.md documents the two modes and why src/ is held back
> for now. That file replaces the tests/docs/.gitkeep placeholder, which
> is no longer needed now that the directory has real content.
> 
> .gitignore learns to ignore ruff's .ruff_cache/ directory.
> 
> AI-Generated: codex/claude-opus 4.7 (xhigh)
> Signed-off-by: Trevor Woerner <[email protected]>
> ---
> changes in v3:
> - no change in this revision.
> changes in v2:
> - v1 submitted the entire test suite as a single commit; v2 breaks
>   the work into a reviewable series, and this patch is one step of it.
> ---
>  .gitignore            |  3 +++
>  pyproject.toml        |  8 ++++++++
>  tests/docs/.gitkeep   |  0
>  tests/docs/linting.md | 39 +++++++++++++++++++++++++++++++++++
>  tests/run-tests.sh    | 47 +++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 97 insertions(+)
>  delete mode 100644 tests/docs/.gitkeep
>  create mode 100644 tests/docs/linting.md
> 
> diff --git a/.gitignore b/.gitignore
> index 534c49538091..3c3cfb328fb0 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -6,3 +6,6 @@
>  # coverage data and reports
>  /.coverage
>  /htmlcov/
> +
> +# ruff cache
> +/.ruff_cache/
> diff --git a/pyproject.toml b/pyproject.toml
> index ece2757bb686..656adcd4930a 100644
> --- a/pyproject.toml
> +++ b/pyproject.toml
> @@ -26,6 +26,7 @@ tests = [
>      "pytest >= 7.0",
>      "coverage >= 7.0",
>      "pytest-cov >= 4.0",
> +    "ruff >= 0.5",

The latest ruff version is 0.15.20, it's had lots of breaking changes
since v0.5 in 2024. Be careful with Claude filling in dependency
versions, it often picks outdated or inappropriate versions to depend
on, always check them by hand.

>  ]
>  
>  [project.scripts]
> @@ -50,3 +51,10 @@ path = "src/wic/cli.py"
>  # leftover files under the pytest base temp directory.
>  tmp_path_retention_policy = "failed"
>  tmp_path_retention_count  = 1
> +
> +[tool.ruff]
> +# For now only the test suite is actually linted (run-tests.sh
> +# --lint-tests passes the tests/ path); the wic source under src/ is
> +# not yet ruff-clean and is left out until its findings are fixed (see
> +# tests/docs/linting.md). --lint-src can still be run to preview the
> +# source findings, but it is reported, not enforced.
> diff --git a/tests/docs/.gitkeep b/tests/docs/.gitkeep
> deleted file mode 100644
> index e69de29bb2d1..000000000000
> diff --git a/tests/docs/linting.md b/tests/docs/linting.md
> new file mode 100644
> index 000000000000..71b4de21c100
> --- /dev/null
> +++ b/tests/docs/linting.md
> @@ -0,0 +1,39 @@
> +# Linting
> +
> +## Contents
> +
> +- [Running the linter](#running-the-linter)
> +- [tests/ must be clean](#tests-must-be-clean)
> +- [src/ is not linted yet](#src-is-not-linted-yet)
> +
> +The test suite is linted with [ruff](https://docs.astral.sh/ruff/). It
> +is configured in `pyproject.toml` (`[tool.ruff]`).
> +
> +## Running the linter
> +
> +The runner exposes ruff through two separate modes, each used on its
> +own:
> +
> +```bash
> +tests/run-tests.sh --lint-tests   # ruff over tests/
> +tests/run-tests.sh --lint-src     # ruff over src/ (preview only)
> +```
> +
> +A lint mode cannot be combined with coverage, with the other lint
> +mode, or with pytest arguments; the runner rejects such combinations.
> +
> +## tests/ must be clean
> +
> +Our own test code is held to a clean bar: `tests/run-tests.sh
> +--lint-tests` reports nothing. If you add a test that trips a rule, fix
> +the test before the change lands.
> +
> +## src/ is not linted yet
> +
> +`--lint-src` runs ruff over the wic source, but the source is **not**
> +yet ruff-clean, so its findings are a preview report rather than a
> +gate: the runner prints them and exits with ruff's status, but nothing
> +in the suite asserts on them. Treating `src/` findings as a hard
> +failure now would block every run on fixes that have not landed. Once
> +the source is cleaned up, `src/` can be promoted to the same clean bar
> +as `tests/`.

I am a bit concerned about a test suite needing its own docs directory.
Claude likes to write output I guess. But this level of detail is
unnecessary and is a pain to keep up-to-date with any changes in the
test suite.

You probably just need a top level TESTING.md file or something, with a
stripped down set of docs on running and authoring tests, any
assumptions that aren't documented in the test code, etc.

If you want to enforce that tests is kept clean of linter errors,
perhaps use pre-commit (https://pre-commit.com) to run the linter before
committing. This pairs well with using Claude for development as it
forces the AI agent to fix the errors before making a commit. If it's
automated, it also doesn't need documenting so much, just point at how
to enable pre-commit.

Best regards,

-- 
Paul Barker
signature.asc (application/pgp-signature, 252 B)