Re: [docs] [PATCH v3 5/5] tools: Add check-confusables pre-commit hook

--=-tpC2fA7WQohHUCgyNxyY
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, 2026-07-28 at 12:07 +0300, Niko Mauno via lists.yoctoproject.org
wrote:
> From: Niko Mauno <[email protected]>
>=20
> Add a check-confusables script, in the same fashion as
> check-glossaries, that scans the documentation .rst sources for
> non-ASCII "confusable" characters (curly quotes, en/em dashes,
> non-breaking and zero-width spaces, etc.) and reports each occurrence
> with its location and suggested ASCII replacement, exiting non-zero if
> any are found. This guards against the class of breakage fixed in the
> preceding commit, e.g. curly quotes causing recipe ParseErrors.
>=20
> Legitimate non-ASCII such as box-drawing characters used in directory
> trees, accented letters in contributor names and CJK characters are
> intentionally left untouched. No-break spaces are likewise tolerated
> on lines containing box-drawing characters, since the tree command
> emits them as indentation in directory listings.
>=20
> Wire it up both as a local pre-commit hook, which checks the changed
> files, and in the Makefile "checks" target, which scans the whole
> tree, alongside check-glossaries.
>=20
> Suggested-by: Quentin Schulz <[email protected]>
> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
> Signed-off-by: Niko Mauno <[email protected]>

Hi Niko,

Are there any existing tools we can use instead of implementing this
ourselves? Perhaps https://pypi.org/project/confusables/ ?

Some comments below if we do want to merge our own implementation of
this...

> ---
>  .pre-commit-config.yaml               |   5 ++
>  documentation/Makefile                |   1 +
>  documentation/tools/check-confusables | 115 ++++++++++++++++++++++++++
>  3 files changed, 121 insertions(+)
>  create mode 100755 documentation/tools/check-confusables
>=20
> diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
> index f2b73a481..876546f9a 100644
> --- a/.pre-commit-config.yaml
> +++ b/.pre-commit-config.yaml
> @@ -6,3 +6,8 @@ repos:
>          entry: ./documentation/tools/check-glossaries
>          language: python
>          pass_filenames: false
> +      - id: check-confusables
> +        name: Check for non-ASCII confusable characters
> +        entry: ./documentation/tools/check-confusables
> +        language: python
> +        files: \.rst$
> diff --git a/documentation/Makefile b/documentation/Makefile
> index fe0574537..87a6f8a8b 100644
> --- a/documentation/Makefile
> +++ b/documentation/Makefile
> @@ -37,6 +37,7 @@ clean:
> =20
>  checks:
>  	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
> +	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
> =20
>  stylecheck:
>  	vale sync
> diff --git a/documentation/tools/check-confusables b/documentation/tools/=
check-confusables
> new file mode 100755
> index 000000000..d9f27b337
> --- /dev/null
> +++ b/documentation/tools/check-confusables
> @@ -0,0 +1,115 @@
> +#!/usr/bin/env python3
> +

This needs a copyright and license header.

> +import argparse
> +import sys
> +
> +from pathlib import Path
> +
> +
> +def parse_arguments() -> argparse.Namespace:
> +    parser =3D argparse.ArgumentParser(
> +        description=3D"Check documentation sources for non-ASCII typogra=
phic "
> +                    "characters that should be plain ASCII")
> +
> +    parser.add_argument("files",
> +                        nargs=3D"*",
> +                        type=3DPath,
> +                        help=3D"Specific files to check; if none are giv=
en, "
> +                             "all *.rst files under --docs-dir are scann=
ed")
> +
> +    parser.add_argument("-d", "--docs-dir",
> +                        type=3DPath,
> +                        default=3DPath(__file__).resolve().parent.parent=
,
> +                        help=3D"Path to documentation/ directory in yoct=
o-docs")
> +
> +    return parser.parse_args()
> +
> +
> +# Map of "confusable" characters that are frequently introduced by edito=
rs,
> +# word processors or copy-pasting, to their plain ASCII replacement. The=
se
> +# look almost identical to regular ASCII but break tooling, e.g. a curly
> +# quote in a recipe example causes:
> +#
> +#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} =3D =E2=80=9Cf=
oo=E2=80=9D'
> +#
> +# Only these characters are flagged; legitimate non-ASCII such as box-dr=
awing
> +# characters used in directory trees, accented letters in contributor na=
mes
> +# and CJK characters are intentionally left alone.
> +confusables =3D {
> +    "=E2=80=98": "'",      # LEFT SINGLE QUOTATION MARK
> +    "=E2=80=99": "'",      # RIGHT SINGLE QUOTATION MARK
> +    "=E2=80=9C": '"',      # LEFT DOUBLE QUOTATION MARK
> +    "=E2=80=9D": '"',      # RIGHT DOUBLE QUOTATION MARK
> +    "=E2=80=B2": "'",      # PRIME
> +    "=E2=80=B3": '"',      # DOUBLE PRIME
> +    "=E2=80=93": "-",      # EN DASH
> +    "=E2=80=94": "--",     # EM DASH
> +    "=E2=80=90": "-",      # HYPHEN
> +    "=E2=80=91": "-",      # NON-BREAKING HYPHEN
> +    "=E2=88=92": "-",      # MINUS SIGN
> +    "=C2=A0": " ",      # NO-BREAK SPACE
> +    "=E2=80=AF": " ",      # NARROW NO-BREAK SPACE
> +    "=E2=80=8B": "",       # ZERO WIDTH SPACE
> +    "=EF=BB=BF": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
> +    "=C2=AD": "",       # SOFT HYPHEN
> +}

We should not directly use confusing unicode characters in this array,
we shoud use '\u' escape sequences.

I worry that the above is only a very small subset of the confusable
characters, and this list will expand over time.

> +
> +NO_BREAK_SPACE =3D "=C2=A0"

As above, better to use a '\u' escape sequence here.

> +
> +
> +def is_box_drawing(char: str) -> bool:
> +    # Box Drawing Unicode block (U+2500..U+257F), used for the directory
> +    # trees rendered in the manuals.
> +    return "=E2=94=80" <=3D char <=3D "=E2=95=BF"

... and probably here as well.

> +
> +
> +def check_file(path: Path, display: str) -> bool:
> +    found =3D False
> +
> +    with open(path, "r", encoding=3D"utf-8") as f:
> +        for lineno, line in enumerate(f, start=3D1):
> +            # The tree(1) command indents its directory listings with
> +            # no-break spaces; such listings are embedded verbatim in th=
e
> +            # manuals. A no-break space is therefore tolerated on any li=
ne
> +            # that also contains box-drawing characters (i.e. inside a
> +            # rendered directory tree), but still flagged elsewhere.
> +            in_tree =3D any(is_box_drawing(c) for c in line)
> +            for col, char in enumerate(line, start=3D1):
> +                if char not in confusables:
> +                    continue
> +                if char =3D=3D NO_BREAK_SPACE and in_tree:
> +                    continue
> +                replacement =3D confusables[char]
> +                hint =3D f"'{replacement}'" if replacement else "(remove=
)"
> +                print(f"WARNING: {display}:{lineno}:{col}: non-ASCII "
> +                      f"character U+{ord(char):04X} should be "
> +                      f"replaced with {hint}")
> +                found =3D True
> +
> +    return found
> +
> +
> +def main():
> +
> +    args =3D parse_arguments()
> +
> +    # When invoked with explicit files (e.g. by pre-commit, which passes=
 the
> +    # staged filenames) only those are checked; otherwise the whole tree=
 of
> +    # *.rst files under --docs-dir is scanned (e.g. by "make checks").
> +    if args.files:
> +        targets =3D [(path, str(path)) for path in args.files]
> +    else:
> +        docs_dir =3D Path(args.docs_dir)
> +        targets =3D [(path, str(path.relative_to(docs_dir)))
> +                   for path in sorted(docs_dir.rglob("*.rst"))]
> +
> +    exit_code =3D 0
> +    for path, display in targets:
> +        if check_file(path, display):
> +            exit_code =3D 1
> +
> +    sys.exit(exit_code)
> +
> +
> +if __name__ =3D=3D "__main__":
> +    main()

Best regards,

--=20
Paul Barker


--=-tpC2fA7WQohHUCgyNxyY
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----

iIcEABYKAC8WIQSzjPXf5Y1BDWhU2iCrY1Tsnbr0bgUCamiHUhEccGF1bEBwYmFy
a2VyLmRldgAKCRCrY1Tsnbr0bjajAP0bXRZMRrKtWIdCz3L8EsVJD1aLKNmkyKRJ
gnbqrmrCdgEAxmOXdWZSc5Urb11tT76HDzLvfSNMVxQZ6vjfeSHSNQc=
=BBxV
-----END PGP SIGNATURE-----

--=-tpC2fA7WQohHUCgyNxyY--