[PATCH v2 5/5] tools: Add check-confusables pre-commit hook

From: Niko Mauno <[email protected]>

Add a check-confusables script, in the same fashion as
check-glossaries, that scans the documentation .rst sources for
non-ASCII "confusable" characters (curly quotes, en/em dashes,
horizontal ellipsis, non-breaking and zero-width spaces, etc.) and
reports each occurrence with its location and suggested ASCII
replacement, exiting non-zero if any are found. This guards against
the class of breakage fixed in the preceding commit, e.g. curly quotes
causing recipe ParseErrors.

Legitimate non-ASCII such as box-drawing characters used in directory
trees, accented letters in contributor names and CJK characters are
intentionally left untouched.

Wire it up both as a local pre-commit hook and in the Makefile "checks"
target, alongside check-glossaries.

Suggested-by: Quentin Schulz <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Niko Mauno <[email protected]>
---
 .pre-commit-config.yaml               |  5 ++
 documentation/Makefile                |  1 +
 documentation/tools/check-confusables | 77 +++++++++++++++++++++++++++
 3 files changed, 83 insertions(+)
 create mode 100755 documentation/tools/check-confusables

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index f2b73a481..d6008b609 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -6,3 +6,8 @@ repos:
         entry: ./documentation/tools/check-glossaries
         language: python
         pass_filenames: false
+      - id: check-confusables
+        name: Check for non-ASCII confusable characters
+        entry: ./documentation/tools/check-confusables
+        language: python
+        pass_filenames: false
diff --git a/documentation/Makefile b/documentation/Makefile
index fe0574537..87a6f8a8b 100644
--- a/documentation/Makefile
+++ b/documentation/Makefile
@@ -37,6 +37,7 @@ clean:
=20
 checks:
 	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
+	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
=20
 stylecheck:
 	vale sync
diff --git a/documentation/tools/check-confusables b/documentation/tools/ch=
eck-confusables
new file mode 100755
index 000000000..f79ee046c
--- /dev/null
+++ b/documentation/tools/check-confusables
@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+
+import argparse
+import sys
+
+from pathlib import Path
+
+
+def parse_arguments() -> argparse.Namespace:
+    parser =3D argparse.ArgumentParser(
+        description=3D"Check documentation sources for non-ASCII typograph=
ic "
+                    "characters that should be plain ASCII")
+
+    parser.add_argument("-d", "--docs-dir",
+                        type=3DPath,
+                        default=3DPath(__file__).resolve().parent.parent,
+                        help=3D"Path to documentation/ directory in yocto-=
docs")
+
+    return parser.parse_args()
+
+
+# Map of "confusable" characters that are frequently introduced by editors=
,
+# word processors or copy-pasting, to their plain ASCII replacement. These
+# look almost identical to regular ASCII but break tooling, e.g. a curly
+# quote in a recipe example causes:
+#
+#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} =3D "foo"'
+#
+# Only these characters are flagged; legitimate non-ASCII such as box-draw=
ing
+# characters used in directory trees, accented letters in contributor name=
s
+# and CJK characters are intentionally left alone.
+confusables =3D {
+    "=E2=80=98": "'",      # LEFT SINGLE QUOTATION MARK
+    "=E2=80=99": "'",      # RIGHT SINGLE QUOTATION MARK
+    "=E2=80=9C": '"',      # LEFT DOUBLE QUOTATION MARK
+    "=E2=80=9D": '"',      # RIGHT DOUBLE QUOTATION MARK
+    "=E2=80=B2": "'",      # PRIME
+    "=E2=80=B3": '"',      # DOUBLE PRIME
+    "=E2=80=93": "-",      # EN DASH
+    "=E2=80=94": "--",     # EM DASH
+    "=E2=80=90": "-",      # HYPHEN
+    "=E2=80=91": "-",      # NON-BREAKING HYPHEN
+    "=E2=88=92": "-",      # MINUS SIGN
+    "=E2=80=A6": "...",    # HORIZONTAL ELLIPSIS
+    "=C2=A0": " ",      # NO-BREAK SPACE
+    "=E2=80=AF": " ",      # NARROW NO-BREAK SPACE
+    "=E2=80=8B": "",       # ZERO WIDTH SPACE
+    "=EF=BB=BF": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
+    "=C2=AD": "",       # SOFT HYPHEN
+}
+
+
+def main():
+
+    args =3D parse_arguments()
+    exit_code =3D 0
+
+    for rst_path in sorted(Path(args.docs_dir).rglob("*.rst")):
+        rel =3D rst_path.relative_to(args.docs_dir)
+
+        with open(rst_path, "r", encoding=3D"utf-8") as f:
+            for lineno, line in enumerate(f, start=3D1):
+                for col, char in enumerate(line, start=3D1):
+                    if char in confusables:
+                        replacement =3D confusables[char]
+                        hint =3D (f"'{replacement}'" if replacement
+                                else "(remove)")
+                        print(f"WARNING: {rel}:{lineno}:{col}: non-ASCII "
+                              f"character U+{ord(char):04X} should be "
+                              f"replaced with {hint}")
+                        exit_code =3D 1
+
+    sys.exit(exit_code)
+
+
+if __name__ =3D=3D "__main__":
+    main()
--=20
2.47.3