Re: I/O uses default encoding argument
Adam Turner <[email protected]>
| Newsgroups | gmane.text.docutils.devel |
|---|---|
| Message-ID | <LO3P123MB2681EC7CB405CFB02D0B3946C2AC9@LO3P123MB2681.GBRP123.PROD.OUTLOOK.COM> |
Attached is a set of five patches rebased on current master -- I have updated the language in the deprecation warnings, used the encoding='locale' backport only for 3.7-3.9 (as 3.10 ``builtins.open`` knows about encoding='locale' natively), and updated the ``io.locale_encoding`` detection mechanism to ignore ``-X utf8``, as the system locale encoding doesn't change for the Python UTF-8 mode. A
0005-Update-HISTORY-and-RELEASE-NOTES.patch
(application/octet-stream, 1.9 KB)
From 21491c52deaded0d1a1b69dcf32905f389b004cf Mon Sep 17 00:00:00 2001 From: Adam Turner <[email protected]> Date: Sat, 11 Jun 2022 00:49:16 +0100 Subject: [PATCH 5/8] Update HISTORY and RELEASE-NOTES --- docutils/HISTORY.txt | 7 +++++++ docutils/RELEASE-NOTES.txt | 5 +++++ 2 files changed, 12 insertions(+) diff --git a/docutils/HISTORY.txt b/docutils/HISTORY.txt index 72afab754..db768da22 100644 --- a/docutils/HISTORY.txt +++ b/docutils/HISTORY.txt @@ -31,6 +31,13 @@ Changes Since 0.18.1 Let `Publisher.publish()` print info and prompt when waiting for input from a terminal (cf. https://clig.dev/#interactivity). +* docutils/io.py: + + - Add support for encoding='locale' (:pep:`597`). + - Deprecated ``docutils.io.locale_encoding`` + - Deprecated implicitly using locale encoding on input. Use encoding='locale' + to use the locale encoding. + * docutils/parsers/__init__.py - Aliases "markdown" and "commonmark" point to "commonmark_wrapper". diff --git a/docutils/RELEASE-NOTES.txt b/docutils/RELEASE-NOTES.txt index fe6ab1803..bd791b729 100644 --- a/docutils/RELEASE-NOTES.txt +++ b/docutils/RELEASE-NOTES.txt @@ -96,6 +96,8 @@ Future changes * Remove the ``--html-writer`` option of the ``buildhtml.py`` application (obsoleted by the `"writer" option`_) in Docutils 2.0. +* Change the default input encoding to UTF-8 in Docutils 1.0. + .. _old-format configuration files: docs/user/config.html#old-format-configuration-files .. _rst2html.py: docs/user/tools.html#rst2html-py @@ -122,6 +124,9 @@ Release 0.19b (unpublished) The PEP-writer template's header is updated to fix links and resemble the header of official PEPs. +* Added support for encoding='locale' (:pep:`597`), deprecated implicitly using + the locale encoding on input. + * Various bugfixes and improvements (see HISTORY_). .. _myst: https://pypi.org/project/myst-docutils -- 2.34.1.windows.1
0001-Canonicalise-UTF-8-references.patch
(application/octet-stream, 16.3 KB)
From 2acd8c6bf2372472488fdb0e2eea6d035d8459ce Mon Sep 17 00:00:00 2001 From: Adam Turner <[email protected]> Date: Fri, 10 Jun 2022 23:06:36 +0100 Subject: [PATCH 1/8] Canonicalise UTF-8 references - Use UTF-8 in prose text, error messages, and documentation - Use utf-8 in code or when referring to code - Use utf8 for LaTeX --- docutils/FAQ.txt | 2 +- docutils/docs/dev/todo.txt | 2 +- docutils/docs/ref/rst/directives.txt | 2 +- docutils/docs/user/config.txt | 6 +++--- docutils/docs/user/images/rsp.svg | 2 +- docutils/docs/user/images/s5-files.svg | 2 +- docutils/docs/user/rst/images/biohazard-bitmap-scaling.svg | 2 +- docutils/docs/user/rst/images/biohazard-bitmap.svg | 2 +- docutils/docs/user/rst/images/biohazard-scaling.svg | 2 +- docutils/docs/user/rst/images/biohazard.svg | 2 +- docutils/docs/user/rst/images/title-scaling.svg | 2 +- docutils/docs/user/rst/images/title.svg | 2 +- docutils/test/test_functional.py | 2 +- docutils/test/test_io.py | 2 +- 14 files changed, 16 insertions(+), 16 deletions(-) diff --git a/docutils/FAQ.txt b/docutils/FAQ.txt index 58e79b5f3..c781dec85 100644 --- a/docutils/FAQ.txt +++ b/docutils/FAQ.txt @@ -1111,7 +1111,7 @@ spaces. When encoded with UTF-8 and viewed with ordinary ASCII tools, these characters will appear to be multi-character garbage. You may have an decoding problem in your browser (or editor, etc.). -The encoding of the output is set to "utf-8", but your browser isn't +The encoding of the output is set to UTF-8, but your browser isn't recognizing that. You can either try to fix your browser (enable "UTF-8 character set", sometimes called "Unicode"), or choose a different encoding for the HTML output. You can also try diff --git a/docutils/docs/dev/todo.txt b/docutils/docs/dev/todo.txt index 32db35673..6e9c8ed58 100644 --- a/docutils/docs/dev/todo.txt +++ b/docutils/docs/dev/todo.txt @@ -135,7 +135,7 @@ Code cleanup and modernization: * Encoding of command line arguments can only be guessed: - * try UTF-8/strict first, then try the locale's encoding with + * try ``utf-8:strict`` first, then try the locale's encoding with strict error handling, then ASCII/replace? UTF-8 is almost 100% safe to try first; false positives are rare, diff --git a/docutils/docs/ref/rst/directives.txt b/docutils/docs/ref/rst/directives.txt index 68805814c..8990bf9a2 100644 --- a/docutils/docs/ref/rst/directives.txt +++ b/docutils/docs/ref/rst/directives.txt @@ -953,7 +953,7 @@ and the common options class_ and name_. .. [#ASCII-char] With Python 2, the values for the ``delimiter``, ``quote``, and ``escape`` options must be ASCII characters. (The csv module does not support Unicode and all non-ASCII characters are - encoded as multi-byte utf-8 string). This limitation does not exist + encoded as multi-byte UTF-8 string). This limitation does not exist under Python 3. diff --git a/docutils/docs/user/config.txt b/docutils/docs/user/config.txt index 9becf5a39..8b4137047 100644 --- a/docutils/docs/user/config.txt +++ b/docutils/docs/user/config.txt @@ -421,7 +421,7 @@ output_encoding The text encoding for output. -Default: "UTF-8". Options: ``--output-encoding, -o``. +Default: "utf-8". Options: ``--output-encoding, -o``. output_encoding_error_handler ----------------------------- @@ -455,7 +455,7 @@ record_dependencies Path to a file where Docutils will write a list of files that were required to generate the output, e.g. included files or embedded stylesheets [#dependencies]_. [#pwd]_ The format is one path per -line with forward slashes as separator, the encoding is ``utf8``. +line with forward slashes as separator, the encoding is UTF-8. Set to ``-`` in order to write dependencies to stdout. @@ -465,7 +465,7 @@ This option is particularly useful in conjunction with programs like ham.html: ham.txt $(shell cat hamdeps.txt) rst2html.py --record-dependencies=hamdeps.txt ham.txt ham.html -If the filesystem encoding differs from utf8, replace the ``cat`` +If the filesystem encoding differs from UTF-8, replace the ``cat`` command with a call to a converter, e.g.:: $(shell iconv -f utf8 -t latin1 hamdeps.txt) diff --git a/docutils/docs/user/images/rsp.svg b/docutils/docs/user/images/rsp.svg index 03445d3bc..1a003d975 100644 --- a/docutils/docs/user/images/rsp.svg +++ b/docutils/docs/user/images/rsp.svg @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<?xml version="1.0" encoding="utf-8" standalone="no"?> <!-- Created with Inkscape (http://www.inkscape.org/) --> <svg xmlns:dc="http://purl.org/dc/elements/1.1/" diff --git a/docutils/docs/user/images/s5-files.svg b/docutils/docs/user/images/s5-files.svg index a3e644a22..e71892353 100644 --- a/docutils/docs/user/images/s5-files.svg +++ b/docutils/docs/user/images/s5-files.svg @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<?xml version="1.0" encoding="utf-8" standalone="no"?> <!-- Created with Inkscape (http://www.inkscape.org/) --> <svg xmlns:dc="http://purl.org/dc/elements/1.1/" diff --git a/docutils/docs/user/rst/images/biohazard-bitmap-scaling.svg b/docutils/docs/user/rst/images/biohazard-bitmap-scaling.svg index 946e0768c..58bf38a2a 100644 --- a/docutils/docs/user/rst/images/biohazard-bitmap-scaling.svg +++ b/docutils/docs/user/rst/images/biohazard-bitmap-scaling.svg @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<?xml version="1.0" encoding="utf-8" standalone="no"?> <!-- Created with Inkscape (http://www.inkscape.org/) --> <svg diff --git a/docutils/docs/user/rst/images/biohazard-bitmap.svg b/docutils/docs/user/rst/images/biohazard-bitmap.svg index 8b92ff1b6..b54fb7b0a 100644 --- a/docutils/docs/user/rst/images/biohazard-bitmap.svg +++ b/docutils/docs/user/rst/images/biohazard-bitmap.svg @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<?xml version="1.0" encoding="utf-8" standalone="no"?> <!-- Created with Inkscape (http://www.inkscape.org/) --> <svg diff --git a/docutils/docs/user/rst/images/biohazard-scaling.svg b/docutils/docs/user/rst/images/biohazard-scaling.svg index 0a0260ba8..69d475cf8 100644 --- a/docutils/docs/user/rst/images/biohazard-scaling.svg +++ b/docutils/docs/user/rst/images/biohazard-scaling.svg @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="UTF-8"?> +<?xml version="1.0" encoding="utf-8"?> <svg version="1.0" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"> <title>Biohazard</title> <path d="m23.585 1.3778c-7.3e-5 1.1e-4 6.2e-5 0.0039 0 0.0039 0.0013-3.54e-4 0.0026-0.0032 0.0039-0.0039-4.78e-4 0-0.0034-2.3e-5 -0.0039 0zm-0.29329 0.01173c-0.0027 3.39e-4 -0.0051 0.0032-0.0078 0.0039 0.01047-6.47e-4 0.0208-0.0032 0.0313-0.0039-0.0026 3.1e-5 -0.0053-1.18e-4 -0.0078 0-0.0057 2.6e-4 -0.01016-6.94e-4 -0.01564 0zm1.4234 0.0078c0.02085 0.0016 0.04176 0.0023 0.06257 0.0039-0.02047-0.0016-0.04204-0.0032-0.06257-0.0039zm0.06257 0.0039c5.266 0.41855 9.4125 4.8332 9.4125 10.206 0 5.4708-4.3036 9.9328-9.7057 10.21v2.096c1.4749 0.23317 2.6044 1.514 2.6044 3.0541 0 0.37693-0.06786 0.73739-0.19161 1.0715l1.9161 1.1067c2.9612-4.473 8.9355-5.9326 13.644-3.2144 4.6969 2.7117 6.4311 8.5947 4.063 13.389 3.2501-6.3259 0.99004-14.167-5.2361-17.761-1.5578-0.89938-3.2176-1.4473-4.8959-1.6737 0.64243-1.5663 0.99716-3.2814 0.99716-5.0797 0-7.1369-5.5766-12.98-12.607-13.405zm-1.7167 0.0078c-6.9955 0.46205-12.537 6.2855-12.537 13.397 0 1.7471 0.33392 3.4168 0.94242 4.9467-1.6284 0.2386-3.2426 0.78109-4.7551 1.6543-6.1844 3.5706-8.4499 11.329-5.2947 17.632-2.2781-4.7717-0.53106-10.576 4.1255-13.264 4.7378-2.7354 10.753-1.2393 13.694 3.3004l1.8497-1.0676c-0.11624-0.32496-0.17597-0.67542-0.17597-1.0402 0-1.4747 1.0339-2.7081 2.4167-3.015v-2.139c-5.3544-0.32796-9.6041-4.7699-9.6041-10.206 0-5.3484 4.1065-9.7456 9.3381-10.198zm0.5983 12.858c-2.7025 0.06215-5.1939 0.9822-7.2382 2.4792 0.16332 0.24168 0.3388 0.47558 0.524 0.69997 0.18521 0.22438 0.38102 0.43966 0.58657 0.64522 0.20555 0.20555 0.42084 0.40137 0.64523 0.58656 0.16497 0.13617 0.34173 0.25876 0.51617 0.38323 1.509-0.9908 3.3162-1.572 5.2635-1.572 1.9473-8e-6 3.7505 0.58119 5.2595 1.572 0.17445-0.12446 0.35119-0.24706 0.51618-0.38323 0.22439-0.1852 0.43967-0.38101 0.64522-0.58656 0.20557-0.20556 0.40138-0.42084 0.58658-0.64522 0.1852-0.22439 0.36067-0.45829 0.524-0.69997-2.1182-1.5511-4.7161-2.4792-7.5315-2.4792-0.06599 0-0.13369-7.88e-4 -0.19944 0-0.0321 5.04e-4 -0.06572-7.33e-4 -0.09775 0zm-12.318 11.278c-0.28422 2.61 0.2112 5.324 1.6189 7.7622 1.4077 2.4383 3.5102 4.2281 5.9126 5.2869 0.12765-0.26228 0.24239-0.53297 0.34412-0.80556 0.10173-0.27257 0.19068-0.54821 0.2659-0.829 0.07525-0.2808 0.1395-0.56556 0.1877-0.85248 0.03545-0.21096 0.05374-0.42411 0.07431-0.63742-1.6126-0.81146-3.0189-2.0872-3.9926-3.7736-0.97365-1.6864-1.3744-3.5394-1.2709-5.3417-0.195-0.08884-0.39006-0.17939-0.59048-0.25419-0.27258-0.10171-0.54822-0.19066-0.82901-0.26591-0.28079-0.07523-0.56555-0.13558-0.85247-0.18379-0.28692-0.04821-0.57716-0.08499-0.86812-0.10558zm25.316 0.1525c-0.29098 0.02064-0.5812 0.05743-0.86812 0.10559-0.28692 0.04821-0.57169 0.10855-0.85249 0.1838-0.28078 0.07523-0.56033 0.16419-0.83292 0.26592-0.20042 0.07475-0.39157 0.16924-0.58656 0.25808 0.10353 1.8023-0.29725 3.6553-1.2709 5.3417-0.97364 1.6864-2.38 2.9582-3.9926 3.7697 0.02056 0.2133 0.03885 0.42645 0.07429 0.63741 0.0482 0.28692 0.10854 0.57168 0.18379 0.85249 0.07524 0.28079 0.1642 0.56035 0.26592 0.83291 0.10171 0.2726 0.21648 0.53936 0.34412 0.80165 2.4024-1.0589 4.5049-2.8448 5.9126-5.283 1.4077-2.4383 1.9071-5.1562 1.6228-7.7662zm-14.985 3.3004-1.8614 1.0793c2.3932 4.801 0.67248 10.702-4.0356 13.421-4.6849 2.7048-10.629 1.2814-13.604-3.1401 3.8616 5.9503 11.758 7.9008 17.972 4.3132 1.5053-0.86907 2.7751-1.9894 3.7931-3.273 1.0381 1.3493 2.3519 2.5212 3.9183 3.4255 6.2209 3.5917 14.127 1.6332 17.984-4.3328-2.9704 4.4407-8.9227 5.8772-13.616 3.1675-4.7378-2.7354-6.4495-8.6935-3.9887-13.511l-1.9357-1.1184c-0.56569 0.62984-1.3868 1.0284-2.2993 1.0284-0.92758 0-1.7603-0.41179-2.3267-1.0597z"/> diff --git a/docutils/docs/user/rst/images/biohazard.svg b/docutils/docs/user/rst/images/biohazard.svg index 0879dc2ba..847fd5ca6 100644 --- a/docutils/docs/user/rst/images/biohazard.svg +++ b/docutils/docs/user/rst/images/biohazard.svg @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="UTF-8"?> +<?xml version="1.0" encoding="utf-8"?> <svg version="1.0" width="48" height="48" xmlns="http://www.w3.org/2000/svg"> <title>Biohazard</title> <path d="m23.585 1.3778c-7.3e-5 1.1e-4 6.2e-5 0.0039 0 0.0039 0.0013-3.54e-4 0.0026-0.0032 0.0039-0.0039-4.78e-4 0-0.0034-2.3e-5 -0.0039 0zm-0.29329 0.01173c-0.0027 3.39e-4 -0.0051 0.0032-0.0078 0.0039 0.01047-6.47e-4 0.0208-0.0032 0.0313-0.0039-0.0026 3.1e-5 -0.0053-1.18e-4 -0.0078 0-0.0057 2.6e-4 -0.01016-6.94e-4 -0.01564 0zm1.4234 0.0078c0.02085 0.0016 0.04176 0.0023 0.06257 0.0039-0.02047-0.0016-0.04204-0.0032-0.06257-0.0039zm0.06257 0.0039c5.266 0.41855 9.4125 4.8332 9.4125 10.206 0 5.4708-4.3036 9.9328-9.7057 10.21v2.096c1.4749 0.23317 2.6044 1.514 2.6044 3.0541 0 0.37693-0.06786 0.73739-0.19161 1.0715l1.9161 1.1067c2.9612-4.473 8.9355-5.9326 13.644-3.2144 4.6969 2.7117 6.4311 8.5947 4.063 13.389 3.2501-6.3259 0.99004-14.167-5.2361-17.761-1.5578-0.89938-3.2176-1.4473-4.8959-1.6737 0.64243-1.5663 0.99716-3.2814 0.99716-5.0797 0-7.1369-5.5766-12.98-12.607-13.405zm-1.7167 0.0078c-6.9955 0.46205-12.537 6.2855-12.537 13.397 0 1.7471 0.33392 3.4168 0.94242 4.9467-1.6284 0.2386-3.2426 0.78109-4.7551 1.6543-6.1844 3.5706-8.4499 11.329-5.2947 17.632-2.2781-4.7717-0.53106-10.576 4.1255-13.264 4.7378-2.7354 10.753-1.2393 13.694 3.3004l1.8497-1.0676c-0.11624-0.32496-0.17597-0.67542-0.17597-1.0402 0-1.4747 1.0339-2.7081 2.4167-3.015v-2.139c-5.3544-0.32796-9.6041-4.7699-9.6041-10.206 0-5.3484 4.1065-9.7456 9.3381-10.198zm0.5983 12.858c-2.7025 0.06215-5.1939 0.9822-7.2382 2.4792 0.16332 0.24168 0.3388 0.47558 0.524 0.69997 0.18521 0.22438 0.38102 0.43966 0.58657 0.64522 0.20555 0.20555 0.42084 0.40137 0.64523 0.58656 0.16497 0.13617 0.34173 0.25876 0.51617 0.38323 1.509-0.9908 3.3162-1.572 5.2635-1.572 1.9473-8e-6 3.7505 0.58119 5.2595 1.572 0.17445-0.12446 0.35119-0.24706 0.51618-0.38323 0.22439-0.1852 0.43967-0.38101 0.64522-0.58656 0.20557-0.20556 0.40138-0.42084 0.58658-0.64522 0.1852-0.22439 0.36067-0.45829 0.524-0.69997-2.1182-1.5511-4.7161-2.4792-7.5315-2.4792-0.06599 0-0.13369-7.88e-4 -0.19944 0-0.0321 5.04e-4 -0.06572-7.33e-4 -0.09775 0zm-12.318 11.278c-0.28422 2.61 0.2112 5.324 1.6189 7.7622 1.4077 2.4383 3.5102 4.2281 5.9126 5.2869 0.12765-0.26228 0.24239-0.53297 0.34412-0.80556 0.10173-0.27257 0.19068-0.54821 0.2659-0.829 0.07525-0.2808 0.1395-0.56556 0.1877-0.85248 0.03545-0.21096 0.05374-0.42411 0.07431-0.63742-1.6126-0.81146-3.0189-2.0872-3.9926-3.7736-0.97365-1.6864-1.3744-3.5394-1.2709-5.3417-0.195-0.08884-0.39006-0.17939-0.59048-0.25419-0.27258-0.10171-0.54822-0.19066-0.82901-0.26591-0.28079-0.07523-0.56555-0.13558-0.85247-0.18379-0.28692-0.04821-0.57716-0.08499-0.86812-0.10558zm25.316 0.1525c-0.29098 0.02064-0.5812 0.05743-0.86812 0.10559-0.28692 0.04821-0.57169 0.10855-0.85249 0.1838-0.28078 0.07523-0.56033 0.16419-0.83292 0.26592-0.20042 0.07475-0.39157 0.16924-0.58656 0.25808 0.10353 1.8023-0.29725 3.6553-1.2709 5.3417-0.97364 1.6864-2.38 2.9582-3.9926 3.7697 0.02056 0.2133 0.03885 0.42645 0.07429 0.63741 0.0482 0.28692 0.10854 0.57168 0.18379 0.85249 0.07524 0.28079 0.1642 0.56035 0.26592 0.83291 0.10171 0.2726 0.21648 0.53936 0.34412 0.80165 2.4024-1.0589 4.5049-2.8448 5.9126-5.283 1.4077-2.4383 1.9071-5.1562 1.6228-7.7662zm-14.985 3.3004-1.8614 1.0793c2.3932 4.801 0.67248 10.702-4.0356 13.421-4.6849 2.7048-10.629 1.2814-13.604-3.1401 3.8616 5.9503 11.758 7.9008 17.972 4.3132 1.5053-0.86907 2.7751-1.9894 3.7931-3.273 1.0381 1.3493 2.3519 2.5212 3.9183 3.4255 6.2209 3.5917 14.127 1.6332 17.984-4.3328-2.9704 4.4407-8.9227 5.8772-13.616 3.1675-4.7378-2.7354-6.4495-8.6935-3.9887-13.511l-1.9357-1.1184c-0.56569 0.62984-1.3868 1.0284-2.2993 1.0284-0.92758 0-1.7603-0.41179-2.3267-1.0597z"/> diff --git a/docutils/docs/user/rst/images/title-scaling.svg b/docutils/docs/user/rst/images/title-scaling.svg index 0a6e597bb..6f881f0a4 100644 --- a/docutils/docs/user/rst/images/title-scaling.svg +++ b/docutils/docs/user/rst/images/title-scaling.svg @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="UTF-8"?> +<?xml version="1.0" encoding="utf-8"?> <svg width="203pt" height="24pt" version="1.1" viewBox="0 0 232 24" xmlns="http://www.w3.org/2000/svg"> <text x="-2.25" y="24" fill="#000000" diff --git a/docutils/docs/user/rst/images/title.svg b/docutils/docs/user/rst/images/title.svg index 06fc1895b..3a7477143 100644 --- a/docutils/docs/user/rst/images/title.svg +++ b/docutils/docs/user/rst/images/title.svg @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="UTF-8"?> +<?xml version="1.0" encoding="utf-8"?> <svg width="203pt" height="24pt" version="1.1" xmlns="http://www.w3.org/2000/svg"> <text x="-2.25" y="24" fill="#000000" diff --git a/docutils/test/test_functional.py b/docutils/test/test_functional.py index 18a2ddf67..6ddcb7bbe 100755 --- a/docutils/test/test_functional.py +++ b/docutils/test/test_functional.py @@ -157,7 +157,7 @@ expected output and check it in: no_expected = self.no_expected_template % { 'exp': expected_path, 'out': params['destination_path']} self.assertTrue(os.access(expected_path, os.R_OK), no_expected) - # samples are UTF8 encoded. 'rb' leads to errors with Python 3! + # samples are UTF-8 encoded. 'rb' leads to errors with Python 3! f = open(expected_path, 'r', encoding='utf-8') # Normalize line endings: expected = '\n'.join(f.read().splitlines()) diff --git a/docutils/test/test_io.py b/docutils/test/test_io.py index 55873f649..b120433fd 100755 --- a/docutils/test/test_io.py +++ b/docutils/test/test_io.py @@ -130,7 +130,7 @@ print("hello world") self.assertEqual(data, ['Some include text.\n']) def test_heuristics_no_utf8(self): - # if no encoding is given and decoding with utf-8 fails, + # if no encoding is given and decoding with 'utf-8' fails, # use either the locale encoding (if specified) or latin-1: if io.locale_encoding not in ('utf-8', 'utf8'): # in Py3k, the locale encoding is used without --input-encoding -- 2.34.1.windows.1
0002-Ignore-UTF-8-mode-when-detecting-locale-encoding.patch
(application/octet-stream, 1001 B)
From aa859da508fcaffff6eaa1e779669087b1ff4ad2 Mon Sep 17 00:00:00 2001 From: Adam Turner <[email protected]> Date: Thu, 16 Jun 2022 15:25:17 +0100 Subject: [PATCH 2/8] Ignore UTF-8 mode when detecting locale encoding --- docutils/docutils/io.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docutils/docutils/io.py b/docutils/docutils/io.py index 532477409..8ed00cd4f 100644 --- a/docutils/docutils/io.py +++ b/docutils/docutils/io.py @@ -26,7 +26,7 @@ from docutils import TransformSpec # or whether front-end tools need to call `locale.setlocale()` # before importing this module try: - locale_encoding = locale.getpreferredencoding(do_setlocale=False).lower() + locale_encoding = locale.getdefaultlocale()[1].lower() except ValueError as error: # OS X may set UTF-8 without language code # See https://bugs.python.org/issue18378 fixed in 3.8 # and https://sourceforge.net/p/docutils/bugs/298/. -- 2.34.1.windows.1
0003-Deprecate-docutils.io.locale_encoding.patch
(application/octet-stream, 6.4 KB)
From 3c2043550e5bc4b949c09657201165850bd4e7e4 Mon Sep 17 00:00:00 2001 From: Adam Turner <[email protected]> Date: Fri, 10 Jun 2022 23:46:01 +0100 Subject: [PATCH 3/8] Deprecate docutils.io.locale_encoding --- docutils/docutils/frontend.py | 5 +++- docutils/docutils/io.py | 30 ++++++++++++------- docutils/test/test_io.py | 15 ++++++++-- .../test_rst/test_directives/test_date.py | 3 +- 4 files changed, 39 insertions(+), 14 deletions(-) diff --git a/docutils/docutils/frontend.py b/docutils/docutils/frontend.py index aa3d9fbed..fa6ca066e 100644 --- a/docutils/docutils/frontend.py +++ b/docutils/docutils/frontend.py @@ -449,7 +449,10 @@ class OptionParser(optparse.OptionParser, docutils.SettingsSpec): """Lookup table for boolean configuration file settings.""" default_error_encoding = (getattr(sys.stderr, 'encoding', None) - or io.locale_encoding or 'ascii') + # use the private version of locale encoding to + # skip the deprecation warning + or io._locale_encoding # NoQA + or 'ascii') default_error_encoding_error_handler = 'backslashreplace' diff --git a/docutils/docutils/io.py b/docutils/docutils/io.py index 8ed00cd4f..b8b8f978b 100644 --- a/docutils/docutils/io.py +++ b/docutils/docutils/io.py @@ -20,27 +20,37 @@ from docutils import TransformSpec # Guess the locale's preferred encoding. -# If no valid guess can be made, locale_encoding is set to `None`: +# If no valid guess can be made, _locale_encoding is set to `None`: # # TODO: check whether this is set correctly with every OS and Python version # or whether front-end tools need to call `locale.setlocale()` # before importing this module try: - locale_encoding = locale.getdefaultlocale()[1].lower() + _locale_encoding = locale.getdefaultlocale()[1].lower() except ValueError as error: # OS X may set UTF-8 without language code # See https://bugs.python.org/issue18378 fixed in 3.8 # and https://sourceforge.net/p/docutils/bugs/298/. # Drop the special case after requiring Python >= 3.8 if "unknown locale: UTF-8" in error.args: - locale_encoding = "utf-8" + _locale_encoding = "utf-8" else: - locale_encoding = None + _locale_encoding = None except: # noqa any other problems determining the locale -> use None - locale_encoding = None + _locale_encoding = None try: - codecs.lookup(locale_encoding or '') + codecs.lookup(_locale_encoding or '') except LookupError: - locale_encoding = None + _locale_encoding = None + + +def __getattr__(name): + if name == "locale_encoding": + warnings.warn("'docutils.io.locale_encoding' is deprecated and will " + 'be removed in Docutils 1.0. Pass an explicit encoding, ' + "or 'locale' for the locale encoding.", + DeprecationWarning, stacklevel=2) + return _locale_encoding + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") class InputError(OSError): pass @@ -137,8 +147,8 @@ class Input(TransformSpec): # no BOM found. Start with UTF-8, because that only matches # data that *IS* UTF-8: encodings = ['utf-8', 'latin-1'] - if locale_encoding: - encodings.insert(1, locale_encoding) + if _locale_encoding: + encodings.insert(1, _locale_encoding) for enc in encodings: try: decoded = str(data, enc, self.error_handler) @@ -265,7 +275,7 @@ class ErrorOutput: """Where warning output is sent.""" self.encoding = (encoding or getattr(destination, 'encoding', None) - or locale_encoding or 'ascii') + or _locale_encoding or 'ascii') """The output character encoding.""" self.encoding_errors = encoding_errors diff --git a/docutils/test/test_io.py b/docutils/test/test_io.py index b120433fd..1331a9d6c 100755 --- a/docutils/test/test_io.py +++ b/docutils/test/test_io.py @@ -11,6 +11,7 @@ Test module for io.py. from io import StringIO, BytesIO import sys import unittest +import warnings from docutils import io @@ -132,11 +133,12 @@ print("hello world") def test_heuristics_no_utf8(self): # if no encoding is given and decoding with 'utf-8' fails, # use either the locale encoding (if specified) or latin-1: - if io.locale_encoding not in ('utf-8', 'utf8'): + if io._locale_encoding not in ('utf-8', 'utf8'): # NoQA # in Py3k, the locale encoding is used without --input-encoding # skipping the heuristic unless decoding fails. return - probed_encodings = (io.locale_encoding, 'latin-1') + # use private locale encoding to skip deprecation warning + probed_encodings = (io._locale_encoding, 'latin-1') # NoQA input = io.FileInput(source_path='data/latin1.txt') data = input.read() if input.successful_encoding not in probed_encodings: @@ -256,5 +258,14 @@ class ErrorOutputTests(unittest.TestCase): self.assertEqual(buf.getvalue(), 'b\ufffd u\xfc e\xfc b\xfc') +class DeprecationTests(unittest.TestCase): + def test_locale_encoding(self): + with warnings.catch_warnings(): + warnings.simplefilter('always', category=DeprecationWarning) + with self.assertWarnsRegex(DeprecationWarning, + 'docutils.io.locale_encoding'): + io.locale_encoding + + if __name__ == '__main__': unittest.main() diff --git a/docutils/test/test_parsers/test_rst/test_directives/test_date.py b/docutils/test/test_parsers/test_rst/test_directives/test_date.py index ce07d7e8d..6a297eb92 100755 --- a/docutils/test/test_parsers/test_rst/test_directives/test_date.py +++ b/docutils/test/test_parsers/test_rst/test_directives/test_date.py @@ -13,7 +13,8 @@ if __name__ == '__main__': from test_parsers import DocutilsTestSupport import time -from docutils.io import locale_encoding +# import the private version of locale encoding to skip the deprecation warning +from docutils.io import _locale_encoding as locale_encoding # NoQA def suite(): -- 2.34.1.windows.1
0004-Support-encoding-locale.patch
(application/octet-stream, 10 KB)
From 35c6ccd843cfec9cb4bafc44a977ff166259f966 Mon Sep 17 00:00:00 2001 From: Adam Turner <[email protected]> Date: Sat, 11 Jun 2022 00:02:56 +0100 Subject: [PATCH 4/8] Support encoding='locale' --- docutils/docutils/frontend.py | 12 +++----- docutils/docutils/io.py | 58 +++++++++++++++++++++++++++++++++-- docutils/test/test_CLI.py | 4 +-- docutils/test/test_io.py | 48 +++++++++++++++++++++++++++++ 4 files changed, 110 insertions(+), 12 deletions(-) diff --git a/docutils/docutils/frontend.py b/docutils/docutils/frontend.py index fa6ca066e..c085baa46 100644 --- a/docutils/docutils/frontend.py +++ b/docutils/docutils/frontend.py @@ -448,11 +448,7 @@ class OptionParser(optparse.OptionParser, docutils.SettingsSpec): 'off': False, 'no': False, 'false': False, '': False} """Lookup table for boolean configuration file settings.""" - default_error_encoding = (getattr(sys.stderr, 'encoding', None) - # use the private version of locale encoding to - # skip the deprecation warning - or io._locale_encoding # NoQA - or 'ascii') + default_error_encoding = getattr(sys.stderr, 'encoding', None) default_error_encoding_error_handler = 'backslashreplace' @@ -585,9 +581,11 @@ class OptionParser(optparse.OptionParser, docutils.SettingsSpec): {'default': 'strict', 'validator': validate_encoding_error_handler}), ('Specify text encoding and error handler for error output. ' 'Default: %s:%s.' - % (default_error_encoding, default_error_encoding_error_handler), + % (default_error_encoding or 'locale', + default_error_encoding_error_handler), ['--error-encoding', '-e'], - {'metavar': '<name[:handler]>', 'default': default_error_encoding, + {'metavar': '<name[:handler]>', + 'default': default_error_encoding or 'locale', 'validator': validate_encoding_and_error_handler}), ('Specify the error handler for unencodable characters in ' 'error output. Default: %s.' diff --git a/docutils/docutils/io.py b/docutils/docutils/io.py index b8b8f978b..e85172f53 100644 --- a/docutils/docutils/io.py +++ b/docutils/docutils/io.py @@ -53,6 +53,49 @@ def __getattr__(name): raise AttributeError(f"module {__name__!r} has no attribute {name!r}") +def _get_default_encoding(encoding): + """Determine appropriate default encoding. + + Returns input for any value apart from None and 'locale'. + Otherwise, returns the default locale encoding and raises a warning. + + From Docutils 1.0 the default encoding will be UTF-8 and this function will + be removed. + """ + if encoding is not None and encoding != 'locale': + return encoding + if encoding == 'locale' and sys.version_info[:2] >= (3, 10): + return 'locale' + # currently the code uses implicit casting to Boolean values via the use of + # 'or' short-circuiting -- the better test might be 'is not None', but for + # parity of implementation we use 'not' here. + if not _locale_encoding: + warnings.warn('Could not determine the locale encoding, assuming ' + 'ASCII. The fallback encoding will change to UTF-8 ' + 'in Docutils 1.0, pass an explicit input encoding of ' + "'ascii' to retain the current behaviour. Note UTF-8 " + 'is a strict superset of ASCII, so you should only ' + 'explicitly set the input encoding as ASCII if you ' + 'want to error on non-ASCII characters. Otherwise to ' + "silence this warning, use 'utf-8'.", + FutureWarning, stacklevel=2) + return 'ascii' + if encoding is None: + warnings.warn("The default encoding will change to 'utf-8' in " + "Docutils 1.0. Pass an explicit encoding, or 'locale' to" + 'continue using the locale encoding.', + FutureWarning, stacklevel=2) + return _locale_encoding + if encoding == 'locale': + # see locale.getencoding() in Python 3.11 or newer + enc = locale.getdefaultlocale()[1] + if enc is None or hasattr(sys, 'getandroidapilevel'): + # LANG not set or Android, default to UTF-8 + return 'utf-8' + return enc.lower() + return encoding + + class InputError(OSError): pass class OutputError(OSError): pass @@ -135,7 +178,8 @@ class Input(TransformSpec): if self.encoding: # We believe the user/application when the encoding is # explicitly given. - encodings = [self.encoding] + # Handle self.encoding == 'locale' here + encodings = [_get_default_encoding(self.encoding)] else: data_encoding = self.determine_encoding_from_data(data) if data_encoding: @@ -153,6 +197,12 @@ class Input(TransformSpec): try: decoded = str(data, enc, self.error_handler) self.successful_encoding = enc + if len(encodings) == 3 and enc == _locale_encoding: + warnings.warn( + "The default encoding will change to 'utf-8' in " + "Docutils 1.0. Pass an explicit encoding, or 'locale' " + 'to continue using the locale encoding.', + FutureWarning, stacklevel=2) # Return decoded, removing BOMs. return decoded.replace('\ufeff', '') except (UnicodeError, LookupError) as err: @@ -208,6 +258,8 @@ class Output(TransformSpec): def __init__(self, destination=None, destination_path=None, encoding=None, error_handler='strict'): + if isinstance(encoding, str) and encoding == 'locale': + encoding = _get_default_encoding(encoding) self.encoding = encoding """Text encoding for the output destination.""" @@ -274,8 +326,8 @@ class ErrorOutput: self.destination = destination """Where warning output is sent.""" - self.encoding = (encoding or getattr(destination, 'encoding', None) - or _locale_encoding or 'ascii') + self.encoding = _get_default_encoding( + encoding or getattr(destination, 'encoding', None)) """The output character encoding.""" self.encoding_errors = encoding_errors diff --git a/docutils/test/test_CLI.py b/docutils/test/test_CLI.py index d77daed2d..7a523b90c 100644 --- a/docutils/test/test_CLI.py +++ b/docutils/test/test_CLI.py @@ -64,8 +64,8 @@ class CliTests(unittest.TestCase): output, flags=re.DOTALL) # normalise error encoding default output = output.replace( - f'{frontend.OptionParser.default_error_encoding}:backslashreplace', - 'utf-8:backslashreplace') + (frontend.OptionParser.default_error_encoding or 'locale') + + ':backslashreplace', 'utf-8:backslashreplace') # compare to stored version with open('data/help/docutils.txt', encoding='utf-8') as samplefile: expected = samplefile.read() diff --git a/docutils/test/test_io.py b/docutils/test/test_io.py index 1331a9d6c..1f6ce22ce 100755 --- a/docutils/test/test_io.py +++ b/docutils/test/test_io.py @@ -11,6 +11,7 @@ Test module for io.py. from io import StringIO, BytesIO import sys import unittest +import unittest.mock import warnings from docutils import io @@ -266,6 +267,53 @@ class DeprecationTests(unittest.TestCase): 'docutils.io.locale_encoding'): io.locale_encoding + @unittest.mock.patch("docutils.io._locale_encoding", None) + def test_get_default_encoding_locale_encoding_falsy(self): + with self.assertWarnsRegex(FutureWarning, 'Could not determine'): + with warnings.catch_warnings(): + warnings.simplefilter('always', category=FutureWarning) + res = io._get_default_encoding(None) + self.assertEqual(res, 'ascii') + + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always', category=FutureWarning) + if sys.version_info[:2] >= (3, 10): + res = io._get_default_encoding('locale') + self.assertEqual(len(w), 0) + self.assertEqual(res, 'locale') + else: + with self.assertWarnsRegex(FutureWarning, + 'Could not determine'): + res = io._get_default_encoding('locale') + self.assertEqual(res, 'ascii') + + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always', category=FutureWarning) + res = io._get_default_encoding('dummy') + self.assertEqual(len(w), 0) + self.assertEqual(res, 'dummy') + + @unittest.skipIf(not io._locale_encoding, + 'Must have a truthy locale encoding') + def test_get_default_encoding_none(self): + with warnings.catch_warnings(): + warnings.simplefilter('always', category=FutureWarning) + with self.assertWarnsRegex(FutureWarning, 'The default encoding'): + res = io._get_default_encoding(None) + self.assertEqual(res, io._locale_encoding) + + @unittest.skipIf(not io._locale_encoding, + 'Must have a truthy locale encoding') + def test_get_default_encoding_locale(self): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always', category=FutureWarning) + res = io._get_default_encoding('locale') + self.assertEqual(len(w), 0) + if sys.version_info[:2] >= (3, 10): + self.assertEqual(res, 'locale') + else: + self.assertEqual(res, io._locale_encoding) + if __name__ == '__main__': unittest.main() -- 2.34.1.windows.1