[PATCH v5 2/2] tools: Add check-confusables pre-commit hook

From: Niko Mauno <[email protected]>

Add a check-confusables script, in the same fashion as
check-glossaries, that scans the documentation .rst sources for
non-ASCII "confusable" characters (curly quotes, en/em dashes,
non-breaking and zero-width spaces, etc.) and reports each occurrence
with its location and suggested ASCII replacement, exiting non-zero if
any are found. This guards against the class of breakage fixed in the
preceding commit, e.g. curly quotes causing recipe ParseErrors.

Legitimate non-ASCII such as box-drawing characters used in directory
trees, accented letters in contributor names and CJK characters are
intentionally left untouched. No-break spaces are likewise tolerated
on lines containing box-drawing characters, since the tree command
emits them as indentation in directory listings.

The set of flagged characters is intentionally small and curated
rather than exhaustive. A general-purpose dependency such as the
confusables PyPI package targets Unicode homoglyph detection against
the full confusables table; it would also flag the accented names, CJK
and box-drawing characters we deliberately keep, so we would still need
our own allow-list and replacement policy on top of it. A short,
dependency-free table kept in-tree matches check-glossaries and is
trivial to extend if a new problematic character shows up.

Wire it up both as a local pre-commit hook, which checks the changed
files, and in the Makefile "checks" target, which scans the whole
tree, alongside check-glossaries.

Suggested-by: Quentin Schulz <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Niko Mauno <[email protected]>
---
 .pre-commit-config.yaml               |   5 ++
 documentation/Makefile                |   1 +
 documentation/tools/check-confusables | 123 ++++++++++++++++++++++++++
 3 files changed, 129 insertions(+)
 create mode 100755 documentation/tools/check-confusables

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index f2b73a481..876546f9a 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -6,3 +6,8 @@ repos:
         entry: ./documentation/tools/check-glossaries
         language: python
         pass_filenames: false
+      - id: check-confusables
+        name: Check for non-ASCII confusable characters
+        entry: ./documentation/tools/check-confusables
+        language: python
+        files: \.rst$
diff --git a/documentation/Makefile b/documentation/Makefile
index fe0574537..87a6f8a8b 100644
--- a/documentation/Makefile
+++ b/documentation/Makefile
@@ -37,6 +37,7 @@ clean:
=20
 checks:
 	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
+	$(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
=20
 stylecheck:
 	vale sync
diff --git a/documentation/tools/check-confusables b/documentation/tools/ch=
eck-confusables
new file mode 100755
index 000000000..359386e7c
--- /dev/null
+++ b/documentation/tools/check-confusables
@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+#
+# Check documentation sources for non-ASCII typographic characters that
+# should be plain ASCII.
+#
+# Copyright (c) Vaisala Oyj. All rights reserved.
+#
+# SPDX-License-Identifier: MIT
+#
+
+import argparse
+import sys
+
+from pathlib import Path
+
+
+def parse_arguments() -> argparse.Namespace:
+    parser =3D argparse.ArgumentParser(
+        description=3D"Check documentation sources for non-ASCII typograph=
ic "
+                    "characters that should be plain ASCII")
+
+    parser.add_argument("files",
+                        nargs=3D"*",
+                        type=3DPath,
+                        help=3D"Specific files to check; if none are given=
, "
+                             "all *.rst files under --docs-dir are scanned=
")
+
+    parser.add_argument("-d", "--docs-dir",
+                        type=3DPath,
+                        default=3DPath(__file__).resolve().parent.parent,
+                        help=3D"Path to documentation/ directory in yocto-=
docs")
+
+    return parser.parse_args()
+
+
+# Map of "confusable" characters that are frequently introduced by editors=
,
+# word processors or copy-pasting, to their plain ASCII replacement. These
+# look almost identical to regular ASCII but break tooling, e.g. a curly
+# quote in a recipe example causes:
+#
+#   ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} =3D =E2=80=9Cfoo=
=E2=80=9D'
+#
+# Only these characters are flagged; legitimate non-ASCII such as box-draw=
ing
+# characters used in directory trees, accented letters in contributor name=
s
+# and CJK characters are intentionally left alone.
+confusables =3D {
+    "\u2018": "'",      # LEFT SINGLE QUOTATION MARK
+    "\u2019": "'",      # RIGHT SINGLE QUOTATION MARK
+    "\u201c": '"',      # LEFT DOUBLE QUOTATION MARK
+    "\u201d": '"',      # RIGHT DOUBLE QUOTATION MARK
+    "\u2032": "'",      # PRIME
+    "\u2033": '"',      # DOUBLE PRIME
+    "\u2013": "-",      # EN DASH
+    "\u2014": "--",     # EM DASH
+    "\u2010": "-",      # HYPHEN
+    "\u2011": "-",      # NON-BREAKING HYPHEN
+    "\u2212": "-",      # MINUS SIGN
+    "\u00a0": " ",      # NO-BREAK SPACE
+    "\u202f": " ",      # NARROW NO-BREAK SPACE
+    "\u200b": "",       # ZERO WIDTH SPACE
+    "\ufeff": "",       # ZERO WIDTH NO-BREAK SPACE / BOM
+    "\u00ad": "",       # SOFT HYPHEN
+}
+
+NO_BREAK_SPACE =3D "\u00a0"
+
+
+def is_box_drawing(char: str) -> bool:
+    # Box Drawing Unicode block (U+2500..U+257F), used for the directory
+    # trees rendered in the manuals.
+    return "\u2500" <=3D char <=3D "\u257f"
+
+
+def check_file(path: Path, display: str) -> bool:
+    found =3D False
+
+    with open(path, "r", encoding=3D"utf-8") as f:
+        for lineno, line in enumerate(f, start=3D1):
+            # The tree(1) command indents its directory listings with
+            # no-break spaces; such listings are embedded verbatim in the
+            # manuals. A no-break space is therefore tolerated on any line
+            # that also contains box-drawing characters (i.e. inside a
+            # rendered directory tree), but still flagged elsewhere.
+            in_tree =3D any(is_box_drawing(c) for c in line)
+            for col, char in enumerate(line, start=3D1):
+                if char not in confusables:
+                    continue
+                if char =3D=3D NO_BREAK_SPACE and in_tree:
+                    continue
+                replacement =3D confusables[char]
+                hint =3D f"'{replacement}'" if replacement else "(remove)"
+                print(f"WARNING: {display}:{lineno}:{col}: non-ASCII "
+                      f"character U+{ord(char):04X} should be "
+                      f"replaced with {hint}")
+                found =3D True
+
+    return found
+
+
+def main():
+
+    args =3D parse_arguments()
+
+    # When invoked with explicit files (e.g. by pre-commit, which passes t=
he
+    # staged filenames) only those are checked; otherwise the whole tree o=
f
+    # *.rst files under --docs-dir is scanned (e.g. by "make checks").
+    if args.files:
+        targets =3D [(path, str(path)) for path in args.files]
+    else:
+        docs_dir =3D Path(args.docs_dir)
+        targets =3D [(path, str(path.relative_to(docs_dir)))
+                   for path in sorted(docs_dir.rglob("*.rst"))]
+
+    exit_code =3D 0
+    for path, display in targets:
+        if check_file(path, display):
+            exit_code =3D 1
+
+    sys.exit(exit_code)
+
+
+if __name__ =3D=3D "__main__":
+    main()
--=20
2.47.3