Re: [docs] [PATCH v2 5/5] tools: Add check-confusables pre-commit hook

On 7/28/26 10:53 AM, Antonin Godard wrote:
> Hi,
>=20
> On Tue Jul 28, 2026 at 9:23 AM CEST, Niko Mauno via lists.yoctoproject.or=
g wrote:
>> From: Niko Mauno <[email protected]>
>>
>> Add a check-confusables script, in the same fashion as
>> check-glossaries, that scans the documentation .rst sources for
>> non-ASCII "confusable" characters (curly quotes, en/em dashes,
>> horizontal ellipsis, non-breaking and zero-width spaces, etc.) and
>> reports each occurrence with its location and suggested ASCII
>> replacement, exiting non-zero if any are found. This guards against
>> the class of breakage fixed in the preceding commit, e.g. curly quotes
>> causing recipe ParseErrors.
>>
>> Legitimate non-ASCII such as box-drawing characters used in directory
>> trees, accented letters in contributor names and CJK characters are
>> intentionally left untouched.
>>
>> Wire it up both as a local pre-commit hook and in the Makefile "checks"
>> target, alongside check-glossaries.
>>
>> Suggested-by: Quentin Schulz <[email protected]>
>> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
>> Signed-off-by: Niko Mauno <[email protected]>
>> ---
>>   .pre-commit-config.yaml               |  5 ++
>>   documentation/Makefile                |  1 +
>>   documentation/tools/check-confusables | 77 +++++++++++++++++++++++++++
>>   3 files changed, 83 insertions(+)
>>   create mode 100755 documentation/tools/check-confusables
>>
>> diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
>> index f2b73a481..d6008b609 100644
>> --- a/.pre-commit-config.yaml
>> +++ b/.pre-commit-config.yaml
>> @@ -6,3 +6,8 @@ repos:
>>           entry: ./documentation/tools/check-glossaries
>>           language: python
>>           pass_filenames: false
>> +      - id: check-confusables
>> +        name: Check for non-ASCII confusable characters
>> +        entry: ./documentation/tools/check-confusables
>> +        language: python
>> +        pass_filenames: false
>=20
> I think we should pass filenames and check only the modified files, no ne=
ed to
> checking the entire tree each time?
>=20
>> diff --git a/documentation/Makefile b/documentation/Makefile
>> index fe0574537..87a6f8a8b 100644
>> --- a/documentation/Makefile
>> +++ b/documentation/Makefile
>> @@ -37,6 +37,7 @@ clean:
>>  =20
>>   checks:
>>   	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
>> +	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
>>  =20
>>   stylecheck:
>>   	vale sync
>> diff --git a/documentation/tools/check-confusables b/documentation/tools=
/check-confusables
>> new file mode 100755
>> index 000000000..f79ee046c
>> --- /dev/null
>> +++ b/documentation/tools/check-confusables
>> @@ -0,0 +1,77 @@
>> +#!/usr/bin/env python3
>> +
>> +import argparse
>> +import sys
>> +
>> +from pathlib import Path
>> +
>> +
>> +def parse_arguments() -> argparse.Namespace:
>> +    parser =3D argparse.ArgumentParser(
>> +        description=3D"Check documentation sources for non-ASCII typogr=
aphic "
>> +                    "characters that should be plain ASCII")
>> +
>> +    parser.add_argument("-d", "--docs-dir",
>> +                        type=3DPath,
>> +                        default=3DPath(__file__).resolve().parent.paren=
t,
>> +                        help=3D"Path to documentation/ directory in yoc=
to-docs")
>> +
>> +    return parser.parse_args()
>> +
>> +
>> +# Map of "confusable" characters that are frequently introduced by edit=
ors,
>> +# word processors or copy-pasting, to their plain ASCII replacement. Th=
ese
>> +# look almost identical to regular ASCII but break tooling, e.g. a curl=
y
>> +# quote in a recipe example causes:
>> +#
>> +#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} =3D "foo"'
>=20
> I don't see the curly quote here
>=20
>> +#
>> +# Only these characters are flagged; legitimate non-ASCII such as box-d=
rawing
>> +# characters used in directory trees, accented letters in contributor n=
ames
>> +# and CJK characters are intentionally left alone.
>> +confusables =3D {
>> +    "=E2=80=98": "'",      # LEFT SINGLE QUOTATION MARK
>> +    "=E2=80=99": "'",      # RIGHT SINGLE QUOTATION MARK
>> +    "=E2=80=9C": '"',      # LEFT DOUBLE QUOTATION MARK
>> +    "=E2=80=9D": '"',      # RIGHT DOUBLE QUOTATION MARK
>> +    "=E2=80=B2": "'",      # PRIME
>> +    "=E2=80=B3": '"',      # DOUBLE PRIME
>> +    "=E2=80=93": "-",      # EN DASH
>> +    "=E2=80=94": "--",     # EM DASH
>> +    "=E2=80=90": "-",      # HYPHEN
>> +    "=E2=80=91": "-",      # NON-BREAKING HYPHEN
>> +    "=E2=88=92": "-",      # MINUS SIGN
>> +    "=E2=80=A6": "...",    # HORIZONTAL ELLIPSIS
>> +    "=C2=A0": " ",      # NO-BREAK SPACE
>=20
> Actually, since the `tree` command uses no-break spaces, I think it's OK =
to keep
> them otherwise we'd have to convert each tree output, which also might be
> confusing.
>=20
>> +    "=E2=80=AF": " ",      # NARROW NO-BREAK SPACE
>> +    "=E2=80=8B": "",       # ZERO WIDTH SPACE
>> +    "=EF=BB=BF": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
>> +    "=C2=AD": "",       # SOFT HYPHEN
>> +}
>> +
>> +
>> +def main():
>> +
>> +    args =3D parse_arguments()
>> +    exit_code =3D 0
>> +
>> +    for rst_path in sorted(Path(args.docs_dir).rglob("*.rst")):
>> +        rel =3D rst_path.relative_to(args.docs_dir)
>> +
>> +        with open(rst_path, "r", encoding=3D"utf-8") as f:
>> +            for lineno, line in enumerate(f, start=3D1):
>> +                for col, char in enumerate(line, start=3D1):
>> +                    if char in confusables:
>> +                        replacement =3D confusables[char]
>> +                        hint =3D (f"'{replacement}'" if replacement
>> +                                else "(remove)")
>> +                        print(f"WARNING: {rel}:{lineno}:{col}: non-ASCI=
I "
>> +                              f"character U+{ord(char):04X} should be "
>> +                              f"replaced with {hint}")
>> +                        exit_code =3D 1
>> +
>> +    sys.exit(exit_code)
>> +
>> +
>> +if __name__ =3D=3D "__main__":
>> +    main()
>=20
>=20
> Antonin


Thank You Antonin, I have now submitted v3 which should address the=20
observations You pointed out above, plus adds Quentin's Reviewed-by line=20
in the bottom three commits which remained unchanged since v1, and also=20
leaves out conversion of "horizontal ellipsis" characters which seemed=20
to be used for denoting git hash truncation and therefore seemed to add=20
just noise to the fix commit.
-Niko