From: Niko Mauno <[email protected]>
Add a check-confusables script, in the same fashion as
check-glossaries, that scans the documentation .rst sources for
non-ASCII "confusable" characters (curly quotes, en/em dashes,
non-breaking and zero-width spaces, etc.) and reports each occurrence
with its location and suggested ASCII replacement, exiting non-zero if
any are found. This guards against the class of breakage fixed by the
preceding "documentation: Replace non-ASCII confusable characters
with ASCII" commit, e.g. curly quotes causing recipe ParseErrors.
Legitimate non-ASCII such as box-drawing characters used in directory
trees, accented letters in contributor names and CJK characters are
intentionally left untouched. No-break spaces are likewise tolerated
on lines containing box-drawing characters, since the tree command
emits them as indentation in directory listings.
The set of flagged characters is intentionally small and curated
rather than exhaustive. A general-purpose dependency such as the
confusables PyPI package targets Unicode homoglyph detection against
the full confusables table; it would also flag the accented names, CJK
and box-drawing characters we deliberately keep, so we would still need
our own allow-list and replacement policy on top of it. A short,
dependency-free table kept in-tree matches check-glossaries and is
trivial to extend if a new problematic character shows up.
Wire it up both as a local pre-commit hook, which checks the changed
files, and in the Makefile "checks" target, which scans the whole
tree, alongside check-glossaries.
Suggested-by: Quentin Schulz <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Niko Mauno <[email protected]>
---
This applies on top of master-next, which already carries the preceding
"documentation: Replace non-ASCII confusable characters with ASCII"
patch from v5.
Changes since v5:
* Uppercase the CONFUSABLES map, so that both module-level globals
follow the same convention (per review feedback).
* Define NO_BREAK_SPACE ahead of the map and reuse it as the key for
U+00A0 (per review feedback).
* Pass the Path objects to check_file() directly and print them as
such, dropping the separate display strings (per review feedback).
* Realign the inline comments in the map, as one key is now a name.
* Kept flagging the other confusables on box-drawing lines rather
than skipping such lines wholesale: the no-break space is the only
character the tree command is known to emit there, so a wider
exemption would only create a blind spot.
* Reword the commit message reference to the preceding patch, which
is no longer part of this series.
.pre-commit-config.yaml | 5 ++
documentation/Makefile | 1 +
documentation/tools/check-confusables | 121 ++++++++++++++++++++++++++
3 files changed, 127 insertions(+)
create mode 100755 documentation/tools/check-confusables
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index f2b73a481..876546f9a 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -6,3 +6,8 @@ repos:
entry: ./documentation/tools/check-glossaries
language: python
pass_filenames: false
+ - id: check-confusables
+ name: Check for non-ASCII confusable characters
+ entry: ./documentation/tools/check-confusables
+ language: python
+ files: \.rst$
diff --git a/documentation/Makefile b/documentation/Makefile
index fe0574537..87a6f8a8b 100644
--- a/documentation/Makefile
+++ b/documentation/Makefile
@@ -37,6 +37,7 @@ clean:
checks:
$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
+ $(SOURCEDIR)/tools/check-confusables --docs-dir $(SOURCEDIR)
stylecheck:
vale sync
diff --git a/documentation/tools/check-confusables b/documentation/tools/check-confusables
new file mode 100755
index 000000000..36e8b22da
--- /dev/null
+++ b/documentation/tools/check-confusables
@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+#
+# Check documentation sources for non-ASCII typographic characters that
+# should be plain ASCII.
+#
+# Copyright (c) Vaisala Oyj. All rights reserved.
+#
+# SPDX-License-Identifier: MIT
+#
+
+import argparse
+import sys
+
+from pathlib import Path
+
+
+def parse_arguments() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(
+ description="Check documentation sources for non-ASCII typographic "
+ "characters that should be plain ASCII")
+
+ parser.add_argument("files",
+ nargs="*",
+ type=Path,
+ help="Specific files to check; if none are given, "
+ "all *.rst files under --docs-dir are scanned")
+
+ parser.add_argument("-d", "--docs-dir",
+ type=Path,
+ default=Path(__file__).resolve().parent.parent,
+ help="Path to documentation/ directory in yocto-docs")
+
+ return parser.parse_args()
+
+
+NO_BREAK_SPACE = "\u00a0"
+
+# Map of "confusable" characters that are frequently introduced by editors,
+# word processors or copy-pasting, to their plain ASCII replacement. These
+# look almost identical to regular ASCII but break tooling, e.g. a curly
+# quote in a recipe example causes:
+#
+# ERROR: ParseError ...: unparsed line: 'RDEPENDS:${PN} = “foo”'
+#
+# Only these characters are flagged; legitimate non-ASCII such as box-drawing
+# characters used in directory trees, accented letters in contributor names
+# and CJK characters are intentionally left alone.
+CONFUSABLES = {
+ "\u2018": "'", # LEFT SINGLE QUOTATION MARK
+ "\u2019": "'", # RIGHT SINGLE QUOTATION MARK
+ "\u201c": '"', # LEFT DOUBLE QUOTATION MARK
+ "\u201d": '"', # RIGHT DOUBLE QUOTATION MARK
+ "\u2032": "'", # PRIME
+ "\u2033": '"', # DOUBLE PRIME
+ "\u2013": "-", # EN DASH
+ "\u2014": "--", # EM DASH
+ "\u2010": "-", # HYPHEN
+ "\u2011": "-", # NON-BREAKING HYPHEN
+ "\u2212": "-", # MINUS SIGN
+ NO_BREAK_SPACE: " ", # NO-BREAK SPACE
+ "\u202f": " ", # NARROW NO-BREAK SPACE
+ "\u200b": "", # ZERO WIDTH SPACE
+ "\ufeff": "", # ZERO WIDTH NO-BREAK SPACE / BOM
+ "\u00ad": "", # SOFT HYPHEN
+}
+
+
+def is_box_drawing(char: str) -> bool:
+ # Box Drawing Unicode block (U+2500..U+257F), used for the directory
+ # trees rendered in the manuals.
+ return "\u2500" <= char <= "\u257f"
+
+
+def check_file(path: Path) -> bool:
+ found = False
+
+ with open(path, "r", encoding="utf-8") as f:
+ for lineno, line in enumerate(f, start=1):
+ # The tree(1) command indents its directory listings with
+ # no-break spaces; such listings are embedded verbatim in the
+ # manuals. A no-break space is therefore tolerated on any line
+ # that also contains box-drawing characters (i.e. inside a
+ # rendered directory tree), but still flagged elsewhere.
+ in_tree = any(is_box_drawing(c) for c in line)
+ for col, char in enumerate(line, start=1):
+ if char not in CONFUSABLES:
+ continue
+ if char == NO_BREAK_SPACE and in_tree:
+ continue
+ replacement = CONFUSABLES[char]
+ hint = f"'{replacement}'" if replacement else "(remove)"
+ print(f"WARNING: {path}:{lineno}:{col}: non-ASCII "
+ f"character U+{ord(char):04X} should be "
+ f"replaced with {hint}")
+ found = True
+
+ return found
+
+
+def main():
+
+ args = parse_arguments()
+
+ # When invoked with explicit files (e.g. by pre-commit, which passes the
+ # staged filenames) only those are checked; otherwise the whole tree of
+ # *.rst files under --docs-dir is scanned (e.g. by "make checks").
+ if args.files:
+ targets = args.files
+ else:
+ targets = sorted(args.docs_dir.rglob("*.rst"))
+
+ exit_code = 0
+ for path in targets:
+ if check_file(path):
+ exit_code = 1
+
+ sys.exit(exit_code)
+
+
+if __name__ == "__main__":
+ main()
base-commit: ce9e3b121a258933c65397160fc5889e0436397c
--
2.47.3
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.