Re: [docs] [PATCH v3 5/5] tools: Add check-confusables pre-commit hook

On 7/28/26 1:41 PM, Paul Barker wrote:
> On Tue, 2026-07-28 at 12:07 +0300, Niko Mauno via lists.yoctoproject.org
> wrote:
>> From: Niko Mauno <[email protected]>
>>
>> Add a check-confusables script, in the same fashion as
>> check-glossaries, that scans the documentation .rst sources for
>> non-ASCII "confusable" characters (curly quotes, en/em dashes,
>> non-breaking and zero-width spaces, etc.) and reports each occurrence
>> with its location and suggested ASCII replacement, exiting non-zero if
>> any are found. This guards against the class of breakage fixed in the
>> preceding commit, e.g. curly quotes causing recipe ParseErrors.
>>
>> Legitimate non-ASCII such as box-drawing characters used in directory
>> trees, accented letters in contributor names and CJK characters are
>> intentionally left untouched. No-break spaces are likewise tolerated
>> on lines containing box-drawing characters, since the tree command
>> emits them as indentation in directory listings.
>>
>> Wire it up both as a local pre-commit hook, which checks the changed
>> files, and in the Makefile "checks" target, which scans the whole
>> tree, alongside check-glossaries.
>>
>> Suggested-by: Quentin Schulz <[email protected]>
>> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
>> Signed-off-by: Niko Mauno <[email protected]>
>=20
> Hi Niko,
>=20
> Are there any existing tools we can use instead of implementing this
> ourselves? Perhaps https://pypi.org/project/confusables/ ?
>=20
> Some comments below if we do want to merge our own implementation of
> this...
>=20

Hi Paul,

Thank you for these observations. It seems that confusables (and its=20
alternatives) would not directly resolve the exact problems that=20
check-confusables script addresses, therefore retained the=20
dependency-free approach in v4 which I just submitted. It addresses the=20
missing copyright/license header, and changes the literal glyphs to \u=20
escape sequences.

Also the topmost commit message now contains a segment explaining why=20
the "own short table" is used instead of depending on a 3rd party package.

-Niko



>> ---
>>   .pre-commit-config.yaml               |   5 ++
>>   documentation/Makefile                |   1 +
>>   documentation/tools/check-confusables | 115 ++++++++++++++++++++++++++
>>   3 files changed, 121 insertions(+)
>>   create mode 100755 documentation/tools/check-confusables
>>
>> diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
>> index f2b73a481..876546f9a 100644
>> --- a/.pre-commit-config.yaml
>> +++ b/.pre-commit-config.yaml
>> @@ -6,3 +6,8 @@ repos:
>>           entry: ./documentation/tools/check-glossaries
>>           language: python
>>           pass_filenames: false
>> +      - id: check-confusables
>> +        name: Check for non-ASCII confusable characters
>> +        entry: ./documentation/tools/check-confusables
>> +        language: python
>> +        files: \.rst$
>> diff --git a/documentation/Makefile b/documentation/Makefile
>> index fe0574537..87a6f8a8b 100644
>> --- a/documentation/Makefile
>> +++ b/documentation/Makefile
>> @@ -37,6 +37,7 @@ clean:
>>  =20
>>   checks:
>>   	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
>> +	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
>>  =20
>>   stylecheck:
>>   	vale sync
>> diff --git a/documentation/tools/check-confusables b/documentation/tools=
/check-confusables
>> new file mode 100755
>> index 000000000..d9f27b337
>> --- /dev/null
>> +++ b/documentation/tools/check-confusables
>> @@ -0,0 +1,115 @@
>> +#!/usr/bin/env python3
>> +
>=20
> This needs a copyright and license header.
>=20
>> +import argparse
>> +import sys
>> +
>> +from pathlib import Path
>> +
>> +
>> +def parse_arguments() -> argparse.Namespace:
>> +    parser =3D argparse.ArgumentParser(
>> +        description=3D"Check documentation sources for non-ASCII typogr=
aphic "
>> +                    "characters that should be plain ASCII")
>> +
>> +    parser.add_argument("files",
>> +                        nargs=3D"*",
>> +                        type=3DPath,
>> +                        help=3D"Specific files to check; if none are gi=
ven, "
>> +                             "all *.rst files under --docs-dir are scan=
ned")
>> +
>> +    parser.add_argument("-d", "--docs-dir",
>> +                        type=3DPath,
>> +                        default=3DPath(__file__).resolve().parent.paren=
t,
>> +                        help=3D"Path to documentation/ directory in yoc=
to-docs")
>> +
>> +    return parser.parse_args()
>> +
>> +
>> +# Map of "confusable" characters that are frequently introduced by edit=
ors,
>> +# word processors or copy-pasting, to their plain ASCII replacement. Th=
ese
>> +# look almost identical to regular ASCII but break tooling, e.g. a curl=
y
>> +# quote in a recipe example causes:
>> +#
>> +#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} =3D =E2=80=9C=
foo=E2=80=9D'
>> +#
>> +# Only these characters are flagged; legitimate non-ASCII such as box-d=
rawing
>> +# characters used in directory trees, accented letters in contributor n=
ames
>> +# and CJK characters are intentionally left alone.
>> +confusables =3D {
>> +    "=E2=80=98": "'",      # LEFT SINGLE QUOTATION MARK
>> +    "=E2=80=99": "'",      # RIGHT SINGLE QUOTATION MARK
>> +    "=E2=80=9C": '"',      # LEFT DOUBLE QUOTATION MARK
>> +    "=E2=80=9D": '"',      # RIGHT DOUBLE QUOTATION MARK
>> +    "=E2=80=B2": "'",      # PRIME
>> +    "=E2=80=B3": '"',      # DOUBLE PRIME
>> +    "=E2=80=93": "-",      # EN DASH
>> +    "=E2=80=94": "--",     # EM DASH
>> +    "=E2=80=90": "-",      # HYPHEN
>> +    "=E2=80=91": "-",      # NON-BREAKING HYPHEN
>> +    "=E2=88=92": "-",      # MINUS SIGN
>> +    "=C2=A0": " ",      # NO-BREAK SPACE
>> +    "=E2=80=AF": " ",      # NARROW NO-BREAK SPACE
>> +    "=E2=80=8B": "",       # ZERO WIDTH SPACE
>> +    "=EF=BB=BF": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
>> +    "=C2=AD": "",       # SOFT HYPHEN
>> +}
>=20
> We should not directly use confusing unicode characters in this array,
> we shoud use '\u' escape sequences.
>=20
> I worry that the above is only a very small subset of the confusable
> characters, and this list will expand over time.
>=20
>> +
>> +NO_BREAK_SPACE =3D "=C2=A0"
>=20
> As above, better to use a '\u' escape sequence here.
>=20
>> +
>> +
>> +def is_box_drawing(char: str) -> bool:
>> +    # Box Drawing Unicode block (U+2500..U+257F), used for the director=
y
>> +    # trees rendered in the manuals.
>> +    return "=E2=94=80" <=3D char <=3D "=E2=95=BF"
>=20
> ... and probably here as well.
>=20
>> +
>> +
>> +def check_file(path: Path, display: str) -> bool:
>> +    found =3D False
>> +
>> +    with open(path, "r", encoding=3D"utf-8") as f:
>> +        for lineno, line in enumerate(f, start=3D1):
>> +            # The tree(1) command indents its directory listings with
>> +            # no-break spaces; such listings are embedded verbatim in t=
he
>> +            # manuals. A no-break space is therefore tolerated on any l=
ine
>> +            # that also contains box-drawing characters (i.e. inside a
>> +            # rendered directory tree), but still flagged elsewhere.
>> +            in_tree =3D any(is_box_drawing(c) for c in line)
>> +            for col, char in enumerate(line, start=3D1):
>> +                if char not in confusables:
>> +                    continue
>> +                if char =3D=3D NO_BREAK_SPACE and in_tree:
>> +                    continue
>> +                replacement =3D confusables[char]
>> +                hint =3D f"'{replacement}'" if replacement else "(remov=
e)"
>> +                print(f"WARNING: {display}:{lineno}:{col}: non-ASCII "
>> +                      f"character U+{ord(char):04X} should be "
>> +                      f"replaced with {hint}")
>> +                found =3D True
>> +
>> +    return found
>> +
>> +
>> +def main():
>> +
>> +    args =3D parse_arguments()
>> +
>> +    # When invoked with explicit files (e.g. by pre-commit, which passe=
s the
>> +    # staged filenames) only those are checked; otherwise the whole tre=
e of
>> +    # *.rst files under --docs-dir is scanned (e.g. by "make checks").
>> +    if args.files:
>> +        targets =3D [(path, str(path)) for path in args.files]
>> +    else:
>> +        docs_dir =3D Path(args.docs_dir)
>> +        targets =3D [(path, str(path.relative_to(docs_dir)))
>> +                   for path in sorted(docs_dir.rglob("*.rst"))]
>> +
>> +    exit_code =3D 0
>> +    for path, display in targets:
>> +        if check_file(path, display):
>> +            exit_code =3D 1
>> +
>> +    sys.exit(exit_code)
>> +
>> +
>> +if __name__ =3D=3D "__main__":
>> +    main()
>=20
> Best regards,
>=20