bug#49870: [PATCH] Improve syntax source location accuracy
Vivien Kraus via "Bug reports for GUILE, GNU's Ubiquitous Extension Language" <[email protected]>
| Newsgroups | gmane.lisp.guile.bugs |
|---|---|
| Message-ID | <[email protected]> |
Dear guile developers,
The algorithm to compute the column number has a couple of flaws:
1. The text location may decrease (with #\backspace and #\return), which
makes source location ambiguous:
(syntax-case
(call-with-input-string "(a\r b)" read-syntax) ()
((a b)
(values (syntax-source #'a) (syntax-source #'b))))
=>
$1 = ((line . 0) (column . 1))
$2 = ((line . 0) (column . 1))
and:
(syntax-case
(call-with-input-string "('a\b\b\b b)" read-syntax) ()
((a b)
(values (syntax-source #'a) (syntax-source #'b))))
=>
$1 = ((line . 0) (column . 1))
$2 = ((line . 0) (column . 1))
This behavior is not desirable for programs that want to inspect the
source at this location.
2. Many unicode characters need to span multiple columns, especially
non-latin ones.
The best solution I found [1] was to use this algorithm:
1. Tabs stop every 8 columns (that’s what Guile already implements);
2. printable ASCII characters span 1 column;
3. other unprintable ASCII characters span 0 columns (I guess? That
would be compatible with #\alarm, which has its special test case so I
figure it’s important for it to have 0 width);
4. Compute the width of non-ASCII characters with wcwidth.
The wcwidth function takes a unicode character, and, depending on the
LC_CTYPE locale, determines how many columns it spans. It may fail if
LC_CTYPE is not set, and in that case return -1.
For this to work, we need to include the wcwidth gnulib module. You can
do it yourself, but in order not to forget it, I include the first 2
patches. Don’t be afraid if these patches are huge.
The real work is to first expose the wcwidth function to scheme, to
implement the algorithm for the suspendable ports (commit 3), and then
use wcwidth both in the C ports implementation and in the scheme
suspendable ports module (commit 4).
Best regards,
Vivien
[1]: https://www.gnu.org/prep/standards/html_node/Errors.html#Errors
0001-Update-Gnulib.patch
(text/x-patch, 582.6 KB) - not displayed
0002-gnulib-import-wcwidth.patch
(text/x-patch, 50.9 KB)
From ce997aa3a11ef8a9470e4d39cacf67d3893d8c38 Mon Sep 17 00:00:00 2001 From: Vivien Kraus <[email protected]> Date: Wed, 4 Aug 2021 01:04:39 +0200 Subject: [PATCH 2/4] gnulib: import wcwidth --- lib/Makefile.am | 40 +++- lib/uniwidth.in.h | 72 +++++++ lib/uniwidth/cjk.h | 37 ++++ lib/uniwidth/width.c | 468 ++++++++++++++++++++++++++++++++++++++++ lib/wcwidth.c | 73 +++++++ m4/gnulib-cache.m4 | 4 +- m4/gnulib-comp.m4 | 31 ++- m4/libunistring-base.m4 | 141 ++++++++++++ m4/wcwidth.m4 | 115 ++++++++++ 9 files changed, 966 insertions(+), 15 deletions(-) create mode 100644 lib/uniwidth.in.h create mode 100644 lib/uniwidth/cjk.h create mode 100644 lib/uniwidth/width.c create mode 100644 lib/wcwidth.c create mode 100644 m4/libunistring-base.m4 create mode 100644 m4/wcwidth.m4 diff --git a/lib/Makefile.am b/lib/Makefile.am index cb29c3136..be2a54569 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -136,7 +136,8 @@ # verify \ # vsnprintf \ # warnings \ -# wchar +# wchar \ +# wcwidth AUTOMAKE_OPTIONS = 1.11 gnits subdir-objects @@ -2787,9 +2788,7 @@ EXTRA_libgnu_la_SOURCES += strdup.c ## begin gnulib module streq -if gl_GNULIB_ENABLED_streq -endif EXTRA_DIST += streq.h ## end gnulib module streq @@ -3588,6 +3587,32 @@ EXTRA_DIST += unistd.in.h ## end gnulib module unistd +## begin gnulib module uniwidth/base + +BUILT_SOURCES += $(LIBUNISTRING_UNIWIDTH_H) + +uniwidth.h: uniwidth.in.h + $(AM_V_GEN)rm -f $@-t $@ && \ + { echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \ + cat $(srcdir)/uniwidth.in.h; \ + } > $@-t && \ + mv -f $@-t $@ +MOSTLYCLEANFILES += uniwidth.h uniwidth.h-t + +EXTRA_DIST += localcharset.h uniwidth.in.h + +## end gnulib module uniwidth/base + +## begin gnulib module uniwidth/width + +if LIBUNISTRING_COMPILE_UNIWIDTH_WIDTH +libgnu_la_SOURCES += uniwidth/width.c +endif + +EXTRA_DIST += uniwidth/cjk.h + +## end gnulib module uniwidth/width + ## begin gnulib module unsetenv if gl_GNULIB_ENABLED_unsetenv @@ -3830,6 +3855,15 @@ EXTRA_DIST += wctype.in.h ## end gnulib module wctype-h +## begin gnulib module wcwidth + + +EXTRA_DIST += wcwidth.c + +EXTRA_libgnu_la_SOURCES += wcwidth.c + +## end gnulib module wcwidth + ## begin gnulib module write diff --git a/lib/uniwidth.in.h b/lib/uniwidth.in.h new file mode 100644 index 000000000..0859254ef --- /dev/null +++ b/lib/uniwidth.in.h @@ -0,0 +1,72 @@ +/* Display width functions. + Copyright (C) 2001-2002, 2005, 2007, 2009-2021 Free Software Foundation, + Inc. + + This file is free software: you can redistribute it and/or modify + it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + This file is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. */ + +#ifndef _UNIWIDTH_H +#define _UNIWIDTH_H + +#include "unitypes.h" + +/* Get size_t. */ +#include <stddef.h> + +/* Get locale_charset() declaration. */ +#include "localcharset.h" + +#ifdef __cplusplus +extern "C" { +#endif + + +/* Display width. */ + +/* These functions are locale dependent. The encoding argument identifies + the encoding (e.g. "ISO-8859-2" for Polish). */ + +/* Determine number of column positions required for UC. */ +extern int + uc_width (ucs4_t uc, const char *encoding) + _UC_ATTRIBUTE_PURE; + +/* Determine number of column positions required for first N units + (or fewer if S ends before this) in S. */ +extern int + u8_width (const uint8_t *s, size_t n, const char *encoding) + _UC_ATTRIBUTE_PURE; +extern int + u16_width (const uint16_t *s, size_t n, const char *encoding) + _UC_ATTRIBUTE_PURE; +extern int + u32_width (const uint32_t *s, size_t n, const char *encoding) + _UC_ATTRIBUTE_PURE; + +/* Determine number of column positions required for S. */ +extern int + u8_strwidth (const uint8_t *s, const char *encoding) + _UC_ATTRIBUTE_PURE; +extern int + u16_strwidth (const uint16_t *s, const char *encoding) + _UC_ATTRIBUTE_PURE; +extern int + u32_strwidth (const uint32_t *s, const char *encoding) + _UC_ATTRIBUTE_PURE; + + +#ifdef __cplusplus +} +#endif + +#endif /* _UNIWIDTH_H */ diff --git a/lib/uniwidth/cjk.h b/lib/uniwidth/cjk.h new file mode 100644 index 000000000..1853ceb9d --- /dev/null +++ b/lib/uniwidth/cjk.h @@ -0,0 +1,37 @@ +/* Test for CJK encoding. + Copyright (C) 2001-2002, 2005-2007, 2009-2021 Free Software Foundation, Inc. + Written by Bruno Haible <[email protected]>, 2002. + + This file is free software: you can redistribute it and/or modify + it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + This file is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. */ + +#include "streq.h" + +static int +is_cjk_encoding (const char *encoding) +{ + if (0 + /* Legacy Japanese encodings */ + || STREQ_OPT (encoding, "EUC-JP", 'E', 'U', 'C', '-', 'J', 'P', 0, 0, 0) + /* Legacy Chinese encodings */ + || STREQ_OPT (encoding, "GB2312", 'G', 'B', '2', '3', '1', '2', 0, 0, 0) + || STREQ_OPT (encoding, "GBK", 'G', 'B', 'K', 0, 0, 0, 0, 0, 0) + || STREQ_OPT (encoding, "EUC-TW", 'E', 'U', 'C', '-', 'T', 'W', 0, 0, 0) + || STREQ_OPT (encoding, "BIG5", 'B', 'I', 'G', '5', 0, 0, 0, 0, 0) + /* Legacy Korean encodings */ + || STREQ_OPT (encoding, "EUC-KR", 'E', 'U', 'C', '-', 'K', 'R', 0, 0, 0) + || STREQ_OPT (encoding, "CP949", 'C', 'P', '9', '4', '9', 0, 0, 0, 0) + || STREQ_OPT (encoding, "JOHAB", 'J', 'O', 'H', 'A', 'B', 0, 0, 0, 0)) + return 1; + return 0; +} diff --git a/lib/uniwidth/width.c b/lib/uniwidth/width.c new file mode 100644 index 000000000..9ba07d331 --- /dev/null +++ b/lib/uniwidth/width.c @@ -0,0 +1,468 @@ +/* Determine display width of Unicode character. + Copyright (C) 2001-2002, 2006-2021 Free Software Foundation, Inc. + Written by Bruno Haible <[email protected]>, 2002. + + This file is free software: you can redistribute it and/or modify + it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + This file is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. */ + +#include <config.h> + +/* Specification. */ +#include "uniwidth.h" + +#include "cjk.h" + +/* + * Non-spacing attribute table. + * Consists of: + * - Non-spacing characters; generated from PropList.txt or + * "grep '^[^;]*;[^;]*;[^;]*;[^;]*;NSM;' UnicodeData.txt" + * - Format control characters; generated from + * "grep '^[^;]*;[^;]*;Cf;' UnicodeData.txt" + * - Zero width characters; generated from + * "grep '^[^;]*;ZERO WIDTH ' UnicodeData.txt" + */ +static const unsigned char nonspacing_table_data[38*64] = { + /* 0x0000-0x01ff */ + 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, /* 0x0000-0x003f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, /* 0x0040-0x007f */ + 0xff, 0xff, 0xff, 0xff, 0x00, 0x20, 0x00, 0x00, /* 0x0080-0x00bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x00c0-0x00ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0100-0x013f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0140-0x017f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0180-0x01bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x01c0-0x01ff */ + /* 0x0200-0x03ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0200-0x023f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0240-0x027f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0280-0x02bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x02c0-0x02ff */ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, /* 0x0300-0x033f */ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, /* 0x0340-0x037f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0380-0x03bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x03c0-0x03ff */ + /* 0x0400-0x05ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0400-0x043f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0440-0x047f */ + 0xf8, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0480-0x04bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x04c0-0x04ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0500-0x053f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0540-0x057f */ + 0x00, 0x00, 0xfe, 0xff, 0xff, 0xff, 0xff, 0xbf, /* 0x0580-0x05bf */ + 0xb6, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x05c0-0x05ff */ + /* 0x0600-0x07ff */ + 0x3f, 0x00, 0xff, 0x17, 0x00, 0x00, 0x00, 0x00, /* 0x0600-0x063f */ + 0x00, 0xf8, 0xff, 0xff, 0x00, 0x00, 0x01, 0x00, /* 0x0640-0x067f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0680-0x06bf */ + 0x00, 0x00, 0xc0, 0xbf, 0x9f, 0x3d, 0x00, 0x00, /* 0x06c0-0x06ff */ + 0x00, 0x80, 0x02, 0x00, 0x00, 0x00, 0xff, 0xff, /* 0x0700-0x073f */ + 0xff, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0740-0x077f */ + 0x00, 0x00, 0x00, 0x00, 0xc0, 0xff, 0x01, 0x00, /* 0x0780-0x07bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0xf8, 0x0f, 0x00, /* 0x07c0-0x07ff */ + /* 0x0800-0x09ff */ + 0x00, 0x00, 0xc0, 0xfb, 0xef, 0x3e, 0x00, 0x00, /* 0x0800-0x083f */ + 0x00, 0x00, 0x00, 0x0e, 0x00, 0x00, 0x00, 0x00, /* 0x0840-0x087f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0880-0x08bf */ + 0x00, 0x00, 0xf0, 0xff, 0xff, 0xff, 0xff, 0xff, /* 0x08c0-0x08ff */ + 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x14, /* 0x0900-0x093f */ + 0xfe, 0x21, 0xfe, 0x00, 0x0c, 0x00, 0x00, 0x00, /* 0x0940-0x097f */ + 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, /* 0x0980-0x09bf */ + 0x1e, 0x20, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, /* 0x09c0-0x09ff */ + /* 0x0a00-0x0bff */ + 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, /* 0x0a00-0x0a3f */ + 0x86, 0x39, 0x02, 0x00, 0x00, 0x00, 0x23, 0x00, /* 0x0a40-0x0a7f */ + 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, /* 0x0a80-0x0abf */ + 0xbe, 0x21, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, /* 0x0ac0-0x0aff */ + 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x90, /* 0x0b00-0x0b3f */ + 0x1e, 0x20, 0x40, 0x00, 0x0c, 0x00, 0x00, 0x00, /* 0x0b40-0x0b7f */ + 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0b80-0x0bbf */ + 0x01, 0x20, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0bc0-0x0bff */ + /* 0x0c00-0x0dff */ + 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, /* 0x0c00-0x0c3f */ + 0xc1, 0x3d, 0x60, 0x00, 0x0c, 0x00, 0x00, 0x00, /* 0x0c40-0x0c7f */ + 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, /* 0x0c80-0x0cbf */ + 0x00, 0x30, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, /* 0x0cc0-0x0cff */ + 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0d00-0x0d3f */ + 0x1e, 0x20, 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, /* 0x0d40-0x0d7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0d80-0x0dbf */ + 0x00, 0x04, 0x5c, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0dc0-0x0dff */ + /* 0x0e00-0x0fff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf2, 0x07, /* 0x0e00-0x0e3f */ + 0x80, 0x7f, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0e40-0x0e7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf2, 0x1b, /* 0x0e80-0x0ebf */ + 0x00, 0x3f, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0ec0-0x0eff */ + 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0xa0, 0x02, /* 0x0f00-0x0f3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xfe, 0x7f, /* 0x0f40-0x0f7f */ + 0xdf, 0xe0, 0xff, 0xfe, 0xff, 0xff, 0xff, 0x1f, /* 0x0f80-0x0fbf */ + 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x0fc0-0x0fff */ + /* 0x1000-0x11ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0xe0, 0xfd, 0x66, /* 0x1000-0x103f */ + 0x00, 0x00, 0x00, 0xc3, 0x01, 0x00, 0x1e, 0x00, /* 0x1040-0x107f */ + 0x64, 0x20, 0x00, 0x20, 0x00, 0x00, 0x00, 0x00, /* 0x1080-0x10bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10c0-0x10ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1100-0x113f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1140-0x117f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1180-0x11bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11c0-0x11ff */ + /* 0x1200-0x13ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1200-0x123f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1240-0x127f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1280-0x12bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x12c0-0x12ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1300-0x133f */ + 0x00, 0x00, 0x00, 0xe0, 0x00, 0x00, 0x00, 0x00, /* 0x1340-0x137f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1380-0x13bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x13c0-0x13ff */ + /* 0x1600-0x17ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1600-0x163f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1640-0x167f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1680-0x16bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16c0-0x16ff */ + 0x00, 0x00, 0x1c, 0x00, 0x00, 0x00, 0x1c, 0x00, /* 0x1700-0x173f */ + 0x00, 0x00, 0x0c, 0x00, 0x00, 0x00, 0x0c, 0x00, /* 0x1740-0x177f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb0, 0x3f, /* 0x1780-0x17bf */ + 0x40, 0xfe, 0x0f, 0x20, 0x00, 0x00, 0x00, 0x00, /* 0x17c0-0x17ff */ + /* 0x1800-0x19ff */ + 0x00, 0x78, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1800-0x183f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1840-0x187f */ + 0x60, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, /* 0x1880-0x18bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x18c0-0x18ff */ + 0x00, 0x00, 0x00, 0x00, 0x87, 0x01, 0x04, 0x0e, /* 0x1900-0x193f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1940-0x197f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1980-0x19bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x19c0-0x19ff */ + /* 0x1a00-0x1bff */ + 0x00, 0x00, 0x80, 0x09, 0x00, 0x00, 0x00, 0x00, /* 0x1a00-0x1a3f */ + 0x00, 0x00, 0x40, 0x7f, 0xe5, 0x1f, 0xf8, 0x9f, /* 0x1a40-0x1a7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xff, 0x7f, /* 0x1a80-0x1abf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1ac0-0x1aff */ + 0x0f, 0x00, 0x00, 0x00, 0x00, 0x00, 0xd0, 0x17, /* 0x1b00-0x1b3f */ + 0x04, 0x00, 0x00, 0x00, 0x00, 0xf8, 0x0f, 0x00, /* 0x1b40-0x1b7f */ + 0x03, 0x00, 0x00, 0x00, 0x3c, 0x3b, 0x00, 0x00, /* 0x1b80-0x1bbf */ + 0x00, 0x00, 0x00, 0x00, 0x40, 0xa3, 0x03, 0x00, /* 0x1bc0-0x1bff */ + /* 0x1c00-0x1dff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0xf0, 0xcf, 0x00, /* 0x1c00-0x1c3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1c40-0x1c7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1c80-0x1cbf */ + 0x00, 0x00, 0xf7, 0xff, 0xfd, 0x21, 0x10, 0x03, /* 0x1cc0-0x1cff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d00-0x1d3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d40-0x1d7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d80-0x1dbf */ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x3f, 0xf8, /* 0x1dc0-0x1dff */ + /* 0x2000-0x21ff */ + 0x00, 0xf8, 0x00, 0x00, 0x00, 0x7c, 0x00, 0x00, /* 0x2000-0x203f */ + 0x00, 0x00, 0x00, 0x00, 0xdf, 0xff, 0x00, 0x00, /* 0x2040-0x207f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x2080-0x20bf */ + 0x00, 0x00, 0xff, 0xff, 0xff, 0xff, 0x01, 0x00, /* 0x20c0-0x20ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x2100-0x213f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x2140-0x217f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x2180-0x21bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x21c0-0x21ff */ + /* 0x2c00-0x2dff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x2c00-0x2c3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x2c40-0x2c7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x2c80-0x2cbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x03, 0x00, /* 0x2cc0-0x2cff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x2d00-0x2d3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, /* 0x2d40-0x2d7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x2d80-0x2dbf */ + 0x00, 0x00, 0x00, 0x00, 0xff, 0xff, 0xff, 0xff, /* 0x2dc0-0x2dff */ + /* 0x3000-0x31ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x3c, 0x00, 0x00, /* 0x3000-0x303f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x3040-0x307f */ + 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, /* 0x3080-0x30bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x30c0-0x30ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x3100-0x313f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x3140-0x317f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x3180-0x31bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x31c0-0x31ff */ + /* 0xa600-0xa7ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xa600-0xa63f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0xf7, 0x3f, /* 0xa640-0xa67f */ + 0x00, 0x00, 0x00, 0xc0, 0x00, 0x00, 0x00, 0x00, /* 0xa680-0xa6bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, /* 0xa6c0-0xa6ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xa700-0xa73f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xa740-0xa77f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xa780-0xa7bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xa7c0-0xa7ff */ + /* 0xa800-0xa9ff */ + 0x44, 0x08, 0x00, 0x00, 0x60, 0x00, 0x00, 0x00, /* 0xa800-0xa83f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xa840-0xa87f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xa880-0xa8bf */ + 0x30, 0x00, 0x00, 0x00, 0xff, 0xff, 0x03, 0x00, /* 0xa8c0-0xa8ff */ + 0x00, 0x00, 0x00, 0x00, 0xc0, 0x3f, 0x00, 0x00, /* 0xa900-0xa93f */ + 0x80, 0xff, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xa940-0xa97f */ + 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc8, 0x13, /* 0xa980-0xa9bf */ + 0x00, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x00, /* 0xa9c0-0xa9ff */ + /* 0xaa00-0xabff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x7e, 0x66, 0x00, /* 0xaa00-0xaa3f */ + 0x08, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, /* 0xaa40-0xaa7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x9d, 0xc1, /* 0xaa80-0xaabf */ + 0x02, 0x00, 0x00, 0x00, 0x00, 0x30, 0x40, 0x00, /* 0xaac0-0xaaff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xab00-0xab3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xab40-0xab7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xab80-0xabbf */ + 0x00, 0x00, 0x00, 0x00, 0x20, 0x21, 0x00, 0x00, /* 0xabc0-0xabff */ + /* 0xfa00-0xfbff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xfa00-0xfa3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xfa40-0xfa7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xfa80-0xfabf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xfac0-0xfaff */ + 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, /* 0xfb00-0xfb3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xfb40-0xfb7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xfb80-0xfbbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xfbc0-0xfbff */ + /* 0xfe00-0xffff */ + 0xff, 0xff, 0x00, 0x00, 0xff, 0xff, 0x00, 0x00, /* 0xfe00-0xfe3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xfe40-0xfe7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xfe80-0xfebf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, /* 0xfec0-0xfeff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xff00-0xff3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xff40-0xff7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xff80-0xffbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0e, /* 0xffc0-0xffff */ + /* 0x10000-0x101ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10000-0x1003f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10040-0x1007f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10080-0x100bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x100c0-0x100ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10100-0x1013f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10140-0x1017f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10180-0x101bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, /* 0x101c0-0x101ff */ + /* 0x10200-0x103ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10200-0x1023f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10240-0x1027f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10280-0x102bf */ + 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, /* 0x102c0-0x102ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10300-0x1033f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x07, /* 0x10340-0x1037f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10380-0x103bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x103c0-0x103ff */ + /* 0x10a00-0x10bff */ + 0x6e, 0xf0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x87, /* 0x10a00-0x10a3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10a40-0x10a7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10a80-0x10abf */ + 0x00, 0x00, 0x00, 0x00, 0x60, 0x00, 0x00, 0x00, /* 0x10ac0-0x10aff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10b00-0x10b3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10b40-0x10b7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10b80-0x10bbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x10bc0-0x10bff */ + /* 0x11000-0x111ff */ + 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xff, /* 0x11000-0x1103f */ + 0x7f, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, /* 0x11040-0x1107f */ + 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x78, 0x26, /* 0x11080-0x110bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x110c0-0x110ff */ + 0x07, 0x00, 0x00, 0x00, 0x80, 0xef, 0x1f, 0x00, /* 0x11100-0x1113f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, /* 0x11140-0x1117f */ + 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x7f, /* 0x11180-0x111bf */ + 0x00, 0x1c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x111c0-0x111ff */ + /* 0x11200-0x113ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0xd3, 0x40, /* 0x11200-0x1123f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11240-0x1127f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11280-0x112bf */ + 0x00, 0x00, 0x00, 0x80, 0xf8, 0x07, 0x00, 0x00, /* 0x112c0-0x112ff */ + 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, /* 0x11300-0x1133f */ + 0x01, 0x00, 0x00, 0x00, 0xc0, 0x1f, 0x1f, 0x00, /* 0x11340-0x1137f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11380-0x113bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x113c0-0x113ff */ + /* 0x11400-0x115ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xff, /* 0x11400-0x1143f */ + 0x5c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11440-0x1147f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf8, 0x85, /* 0x11480-0x114bf */ + 0x0d, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x114c0-0x114ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11500-0x1153f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11540-0x1157f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3c, 0xb0, /* 0x11580-0x115bf */ + 0x01, 0x00, 0x00, 0x30, 0x00, 0x00, 0x00, 0x00, /* 0x115c0-0x115ff */ + /* 0x11600-0x117ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf8, 0xa7, /* 0x11600-0x1163f */ + 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11640-0x1167f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x28, 0xbf, 0x00, /* 0x11680-0x116bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x116c0-0x116ff */ + 0x00, 0x00, 0x00, 0xe0, 0xbc, 0x0f, 0x00, 0x00, /* 0x11700-0x1173f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11740-0x1177f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11780-0x117bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x117c0-0x117ff */ + /* 0x11c00-0x11dff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x7f, 0x3f, /* 0x11c00-0x11c3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11c40-0x11c7f */ + 0x00, 0x00, 0xfc, 0xff, 0xff, 0xfc, 0x6d, 0x00, /* 0x11c80-0x11cbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11cc0-0x11cff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11d00-0x11d3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11d40-0x11d7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11d80-0x11dbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x11dc0-0x11dff */ + /* 0x16a00-0x16bff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16a00-0x16a3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16a40-0x16a7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16a80-0x16abf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1f, 0x00, /* 0x16ac0-0x16aff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x7f, 0x00, /* 0x16b00-0x16b3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16b40-0x16b7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16b80-0x16bbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16bc0-0x16bff */ + /* 0x16e00-0x16fff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16e00-0x16e3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16e40-0x16e7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16e80-0x16ebf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16ec0-0x16eff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16f00-0x16f3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16f40-0x16f7f */ + 0x00, 0x80, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16f80-0x16fbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x16fc0-0x16fff */ + /* 0x1bc00-0x1bdff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1bc00-0x1bc3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1bc40-0x1bc7f */ + 0x00, 0x00, 0x00, 0x60, 0x0f, 0x00, 0x00, 0x00, /* 0x1bc80-0x1bcbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1bcc0-0x1bcff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1bd00-0x1bd3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1bd40-0x1bd7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1bd80-0x1bdbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1bdc0-0x1bdff */ + /* 0x1d000-0x1d1ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d000-0x1d03f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d040-0x1d07f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d080-0x1d0bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d0c0-0x1d0ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d100-0x1d13f */ + 0x00, 0x00, 0x00, 0x00, 0x80, 0x03, 0xf8, 0xff, /* 0x1d140-0x1d17f */ + 0xe7, 0x0f, 0x00, 0x00, 0x00, 0x3c, 0x00, 0x00, /* 0x1d180-0x1d1bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d1c0-0x1d1ff */ + /* 0x1d200-0x1d3ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d200-0x1d23f */ + 0x1c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d240-0x1d27f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d280-0x1d2bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d2c0-0x1d2ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d300-0x1d33f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d340-0x1d37f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d380-0x1d3bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1d3c0-0x1d3ff */ + /* 0x1da00-0x1dbff */ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x7f, 0xf8, /* 0x1da00-0x1da3f */ + 0xff, 0xff, 0xff, 0xff, 0xff, 0x1f, 0x20, 0x00, /* 0x1da40-0x1da7f */ + 0x10, 0x00, 0x00, 0xf8, 0xfe, 0xff, 0x00, 0x00, /* 0x1da80-0x1dabf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1dac0-0x1daff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1db00-0x1db3f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1db40-0x1db7f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1db80-0x1dbbf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1dbc0-0x1dbff */ + /* 0x1e000-0x1e1ff */ + 0x7f, 0xff, 0xff, 0xf9, 0xdb, 0x07, 0x00, 0x00, /* 0x1e000-0x1e03f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e040-0x1e07f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e080-0x1e0bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e0c0-0x1e0ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e100-0x1e13f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e140-0x1e17f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e180-0x1e1bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e1c0-0x1e1ff */ + /* 0x1e800-0x1e9ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e800-0x1e83f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e840-0x1e87f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e880-0x1e8bf */ + 0x00, 0x00, 0x7f, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e8c0-0x1e8ff */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e900-0x1e93f */ + 0xf0, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e940-0x1e97f */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x1e980-0x1e9bf */ + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 /* 0x1e9c0-0x1e9ff */ +}; +static const signed char nonspacing_table_ind[248] = { + 0, 1, 2, 3, 4, 5, 6, 7, /* 0x0000-0x0fff */ + 8, 9, -1, 10, 11, 12, 13, -1, /* 0x1000-0x1fff */ + 14, -1, -1, -1, -1, -1, 15, -1, /* 0x2000-0x2fff */ + 16, -1, -1, -1, -1, -1, -1, -1, /* 0x3000-0x3fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x4000-0x4fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x5000-0x5fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x6000-0x6fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x7000-0x7fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x8000-0x8fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x9000-0x9fff */ + -1, -1, -1, 17, 18, 19, -1, -1, /* 0xa000-0xafff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0xb000-0xbfff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0xc000-0xcfff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0xd000-0xdfff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0xe000-0xefff */ + -1, -1, -1, -1, -1, 20, -1, 21, /* 0xf000-0xffff */ + 22, 23, -1, -1, -1, 24, -1, -1, /* 0x10000-0x10fff */ + 25, 26, 27, 28, -1, -1, 29, -1, /* 0x11000-0x11fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x12000-0x12fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x13000-0x13fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x14000-0x14fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x15000-0x15fff */ + -1, -1, -1, -1, -1, 30, -1, 31, /* 0x16000-0x16fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x17000-0x17fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x18000-0x18fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x19000-0x19fff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x1a000-0x1afff */ + -1, -1, -1, -1, -1, -1, 32, -1, /* 0x1b000-0x1bfff */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 0x1c000-0x1cfff */ + 33, 34, -1, -1, -1, 35, -1, -1, /* 0x1d000-0x1dfff */ + 36, -1, -1, -1, 37, -1, -1, -1 /* 0x1e000-0x1efff */ +}; + +/* Determine number of column positions required for UC. */ +int +uc_width (ucs4_t uc, const char *encoding) +{ + /* Test for non-spacing or control character. */ + if ((uc >> 9) < 248) + { + int ind = nonspacing_table_ind[uc >> 9]; + if (ind >= 0) + if ((nonspacing_table_data[64*ind + ((uc >> 3) & 63)] >> (uc & 7)) & 1) + { + if (uc > 0 && uc < 0xa0) + return -1; + else + return 0; + } + } + else if ((uc >> 9) == (0xe0000 >> 9)) + { + if (uc >= 0xe0100) + { + if (uc <= 0xe01ef) + return 0; + } + else + { + if (uc >= 0xe0020 ? uc <= 0xe007f : uc == 0xe0001) + return 0; + } + } + /* Test for double-width character. + * Generated from "grep '^[^;]\{4,5\};[WF]' EastAsianWidth.txt" + * and "grep '^[^;]\{4,5\};[^WF]' EastAsianWidth.txt" + */ + if (uc >= 0x1100 + && ((uc < 0x1160) /* Hangul Jamo */ + || (uc >= 0x2329 && uc < 0x232b) /* Angle Brackets */ + || (uc >= 0x2e80 && uc < 0xa4d0 /* CJK ... Yi */ + && !(uc == 0x303f) && !(uc >= 0x4dc0 && uc < 0x4e00)) + || (uc >= 0xac00 && uc < 0xd7a4) /* Hangul Syllables */ + || (uc >= 0xf900 && uc < 0xfb00) /* CJK Compatibility Ideographs */ + || (uc >= 0xfe10 && uc < 0xfe20) /* Presentation Forms for Vertical */ + || (uc >= 0xfe30 && uc < 0xfe70) /* CJK Compatibility Forms */ + || (uc >= 0xff00 && uc < 0xff61) /* Fullwidth Forms */ + || (uc >= 0xffe0 && uc < 0xffe7) /* Fullwidth Signs */ + || (uc >= 0x20000 && uc <= 0x2ffff) /* Supplementary Ideographic Plane */ + || (uc >= 0x30000 && uc <= 0x3ffff) /* Tertiary Ideographic Plane */ + ) ) + return 2; + /* In ancient CJK encodings, Cyrillic and most other characters are + double-width as well. */ + if (uc >= 0x00A1 && uc < 0xFF61 && uc != 0x20A9 + && is_cjk_encoding (encoding)) + return 2; + return 1; +} diff --git a/lib/wcwidth.c b/lib/wcwidth.c new file mode 100644 index 000000000..7f11211dd --- /dev/null +++ b/lib/wcwidth.c @@ -0,0 +1,73 @@ +/* Determine the number of screen columns needed for a character. + Copyright (C) 2006-2007, 2010-2021 Free Software Foundation, Inc. + + This file is free software: you can redistribute it and/or modify + it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + This file is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public License + along with this program. If not, see <https://www.gnu.org/licenses/>. */ + +#include <config.h> + +/* Specification. */ +#include <wchar.h> + +/* Get iswprint. */ +#include <wctype.h> + +#include "localcharset.h" +#include "streq.h" +#include "uniwidth.h" + +/* Returns 1 if the current locale is an UTF-8 locale, 0 otherwise. */ +static inline int +is_locale_utf8 (void) +{ + const char *encoding = locale_charset (); + return STREQ_OPT (encoding, "UTF-8", 'U', 'T', 'F', '-', '8', 0, 0, 0, 0); +} + +#if GNULIB_WCHAR_SINGLE_LOCALE +/* When we know that the locale does not change, provide a speedup by + caching the value of is_locale_utf8. */ +static int cached_is_locale_utf8 = -1; +static inline int +is_locale_utf8_cached (void) +{ + if (cached_is_locale_utf8 < 0) + cached_is_locale_utf8 = is_locale_utf8 (); + return cached_is_locale_utf8; +} +#else +/* By default, don't make assumptions, hence no caching. */ +# define is_locale_utf8_cached is_locale_utf8 +#endif + +int +wcwidth (wchar_t wc) +#undef wcwidth +{ + /* In UTF-8 locales, use a Unicode aware width function. */ + if (is_locale_utf8_cached ()) + { + /* We assume that in a UTF-8 locale, a wide character is the same as a + Unicode character. */ + return uc_width (wc, "UTF-8"); + } + else + { + /* Otherwise, fall back to the system's wcwidth function. */ +#if HAVE_WCWIDTH + return wcwidth (wc); +#else + return wc == 0 ? 0 : iswprint (wc) ? 1 : -1; +#endif + } +} diff --git a/m4/gnulib-cache.m4 b/m4/gnulib-cache.m4 index 841bd6140..aeb671cd1 100644 --- a/m4/gnulib-cache.m4 +++ b/m4/gnulib-cache.m4 @@ -141,7 +141,8 @@ # verify \ # vsnprintf \ # warnings \ -# wchar +# wchar \ +# wcwidth # Specification in the form of a few gnulib-tool.m4 macro invocations: gl_LOCAL_DIR([gnulib-local]) @@ -241,6 +242,7 @@ gl_MODULES([ vsnprintf warnings wchar + wcwidth ]) gl_AVOID([lock unistr/base unistr/u8-mbtouc unistr/u8-mbtouc-unsafe unistr/u8-mbtoucr unistr/u8-prev unistr/u8-uctomb unitypes]) gl_SOURCE_BASE([lib]) diff --git a/m4/gnulib-comp.m4 b/m4/gnulib-comp.m4 index 0ae96ead7..029347507 100644 --- a/m4/gnulib-comp.m4 +++ b/m4/gnulib-comp.m4 @@ -255,6 +255,8 @@ AC_DEFUN([gl_EARLY], # Code from module trunc: # Code from module tzset: # Code from module unistd: + # Code from module uniwidth/base: + # Code from module uniwidth/width: # Code from module unsetenv: # Code from module useless-if-before-free: # Code from module vasnprintf: @@ -265,6 +267,7 @@ AC_DEFUN([gl_EARLY], # Code from module wchar: # Code from module wcrtomb: # Code from module wctype-h: + # Code from module wcwidth: # Code from module write: # Code from module xalloc-oversized: # Code from module xsize: @@ -795,10 +798,18 @@ AC_DEFUN([gl_INIT], gl_MATH_MODULE_INDICATOR([trunc]) gl_UNISTD_H gl_UNISTD_H_REQUIRE_DEFAULTS + gl_LIBUNISTRING_LIBHEADER([0.9.11], [uniwidth.h]) + gl_LIBUNISTRING_MODULE([0.9.8], [uniwidth/width]) gl_FUNC_VSNPRINTF gl_STDIO_MODULE_INDICATOR([vsnprintf]) gl_WCHAR_H gl_WCHAR_H_REQUIRE_DEFAULTS + gl_FUNC_WCWIDTH + if test $HAVE_WCWIDTH = 0 || test $REPLACE_WCWIDTH = 1; then + AC_LIBOBJ([wcwidth]) + gl_PREREQ_WCWIDTH + fi + gl_WCHAR_MODULE_INDICATOR([wcwidth]) gl_FUNC_WRITE if test $REPLACE_WRITE = 1; then AC_LIBOBJ([write]) @@ -849,7 +860,6 @@ AC_DEFUN([gl_INIT], gl_gnulib_enabled_sockets=false gl_gnulib_enabled_stat=false gl_gnulib_enabled_f9850631dca91859e9cddac9359921c0=false - gl_gnulib_enabled_streq=false gl_gnulib_enabled_sys_random=false gl_gnulib_enabled_tempname=false gl_gnulib_enabled_time_r=false @@ -1098,9 +1108,6 @@ AC_SUBST([LTALLOCA]) if { test $HAVE_MBRTOWC = 0 || test $REPLACE_MBRTOWC = 1; } && test $REPLACE_MBSTATE_T = 0; then func_gl_gnulib_m4code_mbsinit fi - if test $HAVE_MBRTOWC = 0 || test $REPLACE_MBRTOWC = 1; then - func_gl_gnulib_m4code_streq - fi fi } func_gl_gnulib_m4code_mbsinit () @@ -1356,12 +1363,6 @@ AC_SUBST([LTALLOCA]) gl_gnulib_enabled_f9850631dca91859e9cddac9359921c0=true fi } - func_gl_gnulib_m4code_streq () - { - if ! $gl_gnulib_enabled_streq; then - gl_gnulib_enabled_streq=true - fi - } func_gl_gnulib_m4code_sys_random () { if ! $gl_gnulib_enabled_sys_random; then @@ -1643,6 +1644,9 @@ AC_SUBST([LTALLOCA]) if test $ac_cv_func_vsnprintf = no || test $REPLACE_VSNPRINTF = 1; then func_gl_gnulib_m4code_vasnprintf fi + if test $HAVE_WCWIDTH = 0 || test $REPLACE_WCWIDTH = 1; then + func_gl_gnulib_m4code_3dcce957eadc896e63ab5f137947b410 + fi if test $REPLACE_WRITE = 1; then func_gl_gnulib_m4code_raise fi @@ -1691,7 +1695,6 @@ AC_SUBST([LTALLOCA]) AM_CONDITIONAL([gl_GNULIB_ENABLED_sockets], [$gl_gnulib_enabled_sockets]) AM_CONDITIONAL([gl_GNULIB_ENABLED_stat], [$gl_gnulib_enabled_stat]) AM_CONDITIONAL([gl_GNULIB_ENABLED_f9850631dca91859e9cddac9359921c0], [$gl_gnulib_enabled_f9850631dca91859e9cddac9359921c0]) - AM_CONDITIONAL([gl_GNULIB_ENABLED_streq], [$gl_gnulib_enabled_streq]) AM_CONDITIONAL([gl_GNULIB_ENABLED_sys_random], [$gl_gnulib_enabled_sys_random]) AM_CONDITIONAL([gl_GNULIB_ENABLED_tempname], [$gl_gnulib_enabled_tempname]) AM_CONDITIONAL([gl_GNULIB_ENABLED_time_r], [$gl_gnulib_enabled_time_r]) @@ -2106,6 +2109,9 @@ AC_DEFUN([gl_FILE_LIST], [ lib/tzset.c lib/unistd.c lib/unistd.in.h + lib/uniwidth.in.h + lib/uniwidth/cjk.h + lib/uniwidth/width.c lib/unsetenv.c lib/vasnprintf.c lib/vasnprintf.h @@ -2117,6 +2123,7 @@ AC_DEFUN([gl_FILE_LIST], [ lib/wcrtomb.c lib/wctype-h.c lib/wctype.in.h + lib/wcwidth.c lib/windows-initguard.h lib/write.c lib/xalloc-oversized.h @@ -2195,6 +2202,7 @@ AC_DEFUN([gl_FILE_LIST], [ m4/lib-ld.m4 m4/lib-link.m4 m4/lib-prefix.m4 + m4/libunistring-base.m4 m4/libunistring.m4 m4/limits-h.m4 m4/link.m4 @@ -2309,6 +2317,7 @@ AC_DEFUN([gl_FILE_LIST], [ m4/wchar_t.m4 m4/wcrtomb.m4 m4/wctype_h.m4 + m4/wcwidth.m4 m4/wint_t.m4 m4/write.m4 m4/xsize.m4 diff --git a/m4/libunistring-base.m4 b/m4/libunistring-base.m4 new file mode 100644 index 000000000..657bc0ded --- /dev/null +++ b/m4/libunistring-base.m4 @@ -0,0 +1,141 @@ +# libunistring-base.m4 serial 5 +dnl Copyright (C) 2010-2021 Free Software Foundation, Inc. +dnl This file is free software; the Free Software Foundation +dnl gives unlimited permission to copy and/or distribute it, +dnl with or without modifications, as long as this notice is preserved. + +dnl From Paolo Bonzini and Bruno Haible. + +dnl gl_LIBUNISTRING_MODULE([VERSION], [Module]) +dnl Declares that the source files of Module should be compiled, unless we +dnl are linking with libunistring and its version is >= the given VERSION. +dnl Defines an automake conditional LIBUNISTRING_COMPILE_$MODULE that is +dnl true if the source files of Module should be compiled. +dnl This macro is to be used for public libunistring API, not for +dnl undocumented API. +dnl +dnl You have to bump the VERSION argument to the next projected version +dnl number each time you make a change that affects the behaviour of the +dnl functions defined in Module (even if the sources of Module itself do not +dnl change). + +AC_DEFUN([gl_LIBUNISTRING_MODULE], +[ + AC_REQUIRE([gl_LIBUNISTRING_LIB_PREPARE]) + dnl Use the variables HAVE_LIBUNISTRING, LIBUNISTRING_VERSION from + dnl gl_LIBUNISTRING_CORE if that macro has been run. + AM_CONDITIONAL(AS_TR_CPP([LIBUNISTRING_COMPILE_$2]), + [gl_LIBUNISTRING_VERSION_CMP([$1])]) +]) + +dnl gl_LIBUNISTRING_LIBHEADER([VERSION], [HeaderFile]) +dnl Declares that HeaderFile should be created, unless we are linking +dnl with libunistring and its version is >= the given VERSION. +dnl HeaderFile should be relative to the lib directory and end in '.h'. +dnl Prepares for substituting LIBUNISTRING_HEADERFILE (to HeaderFile or empty). +dnl +dnl When we are linking with the already installed libunistring and its version +dnl is < VERSION, we create HeaderFile here, because we may compile functions +dnl (via gl_LIBUNISTRING_MODULE above) that are not contained in the installed +dnl version. +dnl When we are linking with the already installed libunistring and its version +dnl is > VERSION, we don't create HeaderFile here: it could cause compilation +dnl errors in other libunistring header files if some types are missing. +dnl +dnl You have to bump the VERSION argument to the next projected version +dnl number each time you make a non-comment change to the HeaderFile. + +AC_DEFUN([gl_LIBUNISTRING_LIBHEADER], +[ + AC_REQUIRE([gl_LIBUNISTRING_LIB_PREPARE]) + dnl Use the variables HAVE_LIBUNISTRING, LIBUNISTRING_VERSION from + dnl gl_LIBUNISTRING_CORE if that macro has been run. + if gl_LIBUNISTRING_VERSION_CMP([$1]); then + LIBUNISTRING_[]AS_TR_CPP([$2])='$2' + else + LIBUNISTRING_[]AS_TR_CPP([$2])= + fi + AC_SUBST([LIBUNISTRING_]AS_TR_CPP([$2])) +]) + +dnl Miscellaneous preparations/initializations. + +AC_DEFUN([gl_LIBUNISTRING_LIB_PREPARE], +[ + dnl Ensure that HAVE_LIBUNISTRING is fully determined at this point. + m4_ifdef([gl_LIBUNISTRING], [AC_REQUIRE([gl_LIBUNISTRING])]) + + AC_REQUIRE([AC_PROG_AWK]) + +dnl Sed expressions to extract the parts of a version number. +changequote(,) +gl_libunistring_sed_extract_major='/^[0-9]/{s/^\([0-9]*\).*/\1/p;q;} +i\ +0 +q +' +gl_libunistring_sed_extract_minor='/^[0-9][0-9]*[.][0-9]/{s/^[0-9]*[.]\([0-9]*\).*/\1/p;q;} +i\ +0 +q +' +gl_libunistring_sed_extract_subminor='/^[0-9][0-9]*[.][0-9][0-9]*[.][0-9]/{s/^[0-9]*[.][0-9]*[.]\([0-9]*\).*/\1/p;q;} +i\ +0 +q +' +changequote([,]) + + if test "$HAVE_LIBUNISTRING" = yes; then + LIBUNISTRING_VERSION_MAJOR=`echo "$LIBUNISTRING_VERSION" | sed -n -e "$gl_libunistring_sed_extract_major"` + LIBUNISTRING_VERSION_MINOR=`echo "$LIBUNISTRING_VERSION" | sed -n -e "$gl_libunistring_sed_extract_minor"` + LIBUNISTRING_VERSION_SUBMINOR=`echo "$LIBUNISTRING_VERSION" | sed -n -e "$gl_libunistring_sed_extract_subminor"` + fi +]) + +dnl gl_LIBUNISTRING_VERSION_CMP([VERSION]) +dnl Expands to a shell statement that evaluates to true if LIBUNISTRING_VERSION +dnl is less than the VERSION argument. +AC_DEFUN([gl_LIBUNISTRING_VERSION_CMP], +[ { test "$HAVE_LIBUNISTRING" != yes \ + || { + dnl AS_LITERAL_IF exists and works fine since autoconf-2.59 at least. + AS_LITERAL_IF([$1], + [dnl This is the optimized variant, that assumes the argument is a literal: + m4_pushdef([requested_version_major], + [gl_LIBUNISTRING_ARG_OR_ZERO(m4_bpatsubst([$1], [^\([0-9]*\).*], [\1]), [])]) + m4_pushdef([requested_version_minor], + [gl_LIBUNISTRING_ARG_OR_ZERO(m4_bpatsubst([$1], [^[0-9]*[.]\([0-9]*\).*], [\1]), [$1])]) + m4_pushdef([requested_version_subminor], + [gl_LIBUNISTRING_ARG_OR_ZERO(m4_bpatsubst([$1], [^[0-9]*[.][0-9]*[.]\([0-9]*\).*], [\1]), [$1])]) + test $LIBUNISTRING_VERSION_MAJOR -lt requested_version_major \ + || { test $LIBUNISTRING_VERSION_MAJOR -eq requested_version_major \ + && { test $LIBUNISTRING_VERSION_MINOR -lt requested_version_minor \ + || { test $LIBUNISTRING_VERSION_MINOR -eq requested_version_minor \ + && test $LIBUNISTRING_VERSION_SUBMINOR -lt requested_version_subminor + } + } + } + m4_popdef([requested_version_subminor]) + m4_popdef([requested_version_minor]) + m4_popdef([requested_version_major]) + ], + [dnl This is the unoptimized variant: + requested_version_major=`echo '$1' | sed -n -e "$gl_libunistring_sed_extract_major"` + requested_version_minor=`echo '$1' | sed -n -e "$gl_libunistring_sed_extract_minor"` + requested_version_subminor=`echo '$1' | sed -n -e "$gl_libunistring_sed_extract_subminor"` + test $LIBUNISTRING_VERSION_MAJOR -lt $requested_version_major \ + || { test $LIBUNISTRING_VERSION_MAJOR -eq $requested_version_major \ + && { test $LIBUNISTRING_VERSION_MINOR -lt $requested_version_minor \ + || { test $LIBUNISTRING_VERSION_MINOR -eq $requested_version_minor \ + && test $LIBUNISTRING_VERSION_SUBMINOR -lt $requested_version_subminor + } + } + } + ]) + } + }]) + +dnl gl_LIBUNISTRING_ARG_OR_ZERO([ARG], [ORIG]) expands to ARG if it is not the +dnl same as ORIG, otherwise to 0. +m4_define([gl_LIBUNISTRING_ARG_OR_ZERO], [m4_if([$1], [$2], [0], [$1])]) diff --git a/m4/wcwidth.m4 b/m4/wcwidth.m4 new file mode 100644 index 000000000..2ac2a5148 --- /dev/null +++ b/m4/wcwidth.m4 @@ -0,0 +1,115 @@ +# wcwidth.m4 serial 34 +dnl Copyright (C) 2006-2021 Free Software Foundation, Inc. +dnl This file is free software; the Free Software Foundation +dnl gives unlimited permission to copy and/or distribute it, +dnl with or without modifications, as long as this notice is preserved. + +AC_DEFUN([gl_FUNC_WCWIDTH], +[ + AC_REQUIRE([gl_WCHAR_H_DEFAULTS]) + AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles + + dnl Persuade glibc <wchar.h> to declare wcwidth(). + AC_REQUIRE([AC_USE_SYSTEM_EXTENSIONS]) + + AC_REQUIRE([gt_TYPE_WCHAR_T]) + AC_REQUIRE([gt_TYPE_WINT_T]) + + AC_CHECK_HEADERS_ONCE([wchar.h]) + AC_CHECK_FUNCS_ONCE([wcwidth]) + + AC_CHECK_DECLS([wcwidth], [], [], [[ + #include <wchar.h> + ]]) + if test $ac_cv_have_decl_wcwidth != yes; then + HAVE_DECL_WCWIDTH=0 + fi + + if test $ac_cv_func_wcwidth != yes; then + AC_CACHE_CHECK([whether wcwidth is a macro], + [gl_cv_func_wcwidth_macro], + [AC_EGREP_CPP([wchar_header_defines_wcwidth], [ +#include <wchar.h> +#ifdef wcwidth + wchar_header_defines_wcwidth +#endif], + [gl_cv_func_wcwidth_macro=yes], + [gl_cv_func_wcwidth_macro=no]) + ]) + fi + + if test $ac_cv_func_wcwidth = yes || test $gl_cv_func_wcwidth_macro = yes; then + HAVE_WCWIDTH=1 + dnl On Mac OS X 10.3, wcwidth(0x0301) (COMBINING ACUTE ACCENT) returns 1. + dnl On NetBSD 9.0, OpenBSD 5.0, MidnightBSD 1.1, + dnl wcwidth(0x05B0) (HEBREW POINT SHEVA) returns 1. + dnl On NetBSD 9.0, MidnightBSD 1.1, OSF/1 5.1, + dnl wcwidth(0x200B) (ZERO WIDTH SPACE) returns 1. + dnl On OpenBSD 5.8, wcwidth(0xFF1A) (FULLWIDTH COLON) returns 0. + dnl This leads to bugs in 'ls' (coreutils). + dnl On Solaris 11.4, wcwidth(0x2202) (PARTIAL DIFFERENTIAL) returns 2, + dnl even in Western locales. + AC_CACHE_CHECK([whether wcwidth works reasonably in UTF-8 locales], + [gl_cv_func_wcwidth_works], + [ + AC_RUN_IFELSE( + [AC_LANG_SOURCE([[ +#include <locale.h> +#include <wchar.h> +#if !HAVE_DECL_WCWIDTH +extern +# ifdef __cplusplus +"C" +# endif +int wcwidth (int); +#endif +int main () +{ + int result = 0; + if (setlocale (LC_ALL, "en_US.UTF-8") != NULL) + { + if (wcwidth (0x0301) > 0) + result |= 1; + if (wcwidth (0x05B0) > 0) + result |= 2; + if (wcwidth (0x200B) > 0) + result |= 4; + if (wcwidth (0xFF1A) == 0) + result |= 8; + if (wcwidth (0x2202) > 1) + result |= 16; + } + return result; +}]])], + [gl_cv_func_wcwidth_works=yes], + [gl_cv_func_wcwidth_works=no], + [ +changequote(,)dnl + case "$host_os" in + # Guess yes on glibc systems. + *-gnu* | gnu*) gl_cv_func_wcwidth_works="guessing yes";; + # Guess yes on musl systems. + *-musl*) gl_cv_func_wcwidth_works="guessing yes";; + # Guess yes on AIX 7 systems. + aix[7-9]*) gl_cv_func_wcwidth_works="guessing yes";; + *) gl_cv_func_wcwidth_works="$gl_cross_guess_normal";; + esac +changequote([,])dnl + ]) + ]) + case "$gl_cv_func_wcwidth_works" in + *yes) ;; + *no) REPLACE_WCWIDTH=1 ;; + esac + else + HAVE_WCWIDTH=0 + fi + dnl We don't substitute HAVE_WCWIDTH. We assume that if the system does not + dnl have the wcwidth function, then it does not declare it. +]) + +# Prerequisites of lib/wcwidth.c. +AC_DEFUN([gl_PREREQ_WCWIDTH], [ + AC_REQUIRE([AC_C_INLINE]) + : +]) -- 2.32.0
0003-Export-the-wcwidth-function.patch
(text/x-patch, 2.2 KB)
From 8026e94f6cf35c75a0a4f928c580b077f546a452 Mon Sep 17 00:00:00 2001 From: Vivien Kraus <[email protected]> Date: Tue, 3 Aug 2021 23:53:41 +0200 Subject: [PATCH 3/4] Export the wcwidth function The wcwidth function computes the number of columns that a character spans. It is in gnulib, but not documented. It is part of POSIX, and its own module in Gnulib [1]. [1]: https://www.gnu.org/software/gnulib/manual/html_node/wcwidth.html --- doc/ref/api-data.texi | 7 +++++++ libguile/chars.c | 12 ++++++++++++ libguile/chars.h | 1 + 3 files changed, 20 insertions(+) diff --git a/doc/ref/api-data.texi b/doc/ref/api-data.texi index 2ad13f5a5..a6647c3bf 100644 --- a/doc/ref/api-data.texi +++ b/doc/ref/api-data.texi @@ -2275,6 +2275,13 @@ with their meanings. @end multitable @end deffn +@deffn {Scheme Procedure} wcwidth chr +@deffnx {C Function} scm_wcwidth (chr) +Return the number of columns that @var{chr} spans. This function depends +on the @code{LC_CTYPE} locale category. Return @code{-1} if @var{chr} is +not recognized. +@end deffn + @rnindex char->integer @deffn {Scheme Procedure} char->integer chr @deffnx {C Function} scm_char_to_integer (chr) diff --git a/libguile/chars.c b/libguile/chars.c index fe55f9e2e..7b31e0601 100644 --- a/libguile/chars.c +++ b/libguile/chars.c @@ -504,6 +504,18 @@ SCM_DEFINE (scm_char_general_category, "char-general-category", 1, 0, 0, } #undef FUNC_NAME +SCM_DEFINE (scm_wcwidth, "wcwidth", 1, 0, 0, + (SCM chr), + "Return the number of columns that @var{chr} spans.") +#define FUNC_NAME s_scm_wcwidth +{ + int width; + SCM_VALIDATE_CHAR (1, chr); + width = wcwidth (SCM_CHAR(chr)); + return scm_from_int (width); +} +#undef FUNC_NAME + diff --git a/libguile/chars.h b/libguile/chars.h index f6d4c6354..1bf7c2aa2 100644 --- a/libguile/chars.h +++ b/libguile/chars.h @@ -86,6 +86,7 @@ SCM_API SCM scm_char_upcase (SCM chr); SCM_API SCM scm_char_downcase (SCM chr); SCM_API SCM scm_char_titlecase (SCM chr); SCM_API SCM scm_char_general_category (SCM chr); +SCM_API SCM scm_wcwidth (SCM chr); SCM_INLINE SCM scm_c_make_char (scm_t_wchar c); SCM_API scm_t_wchar scm_c_upcase (scm_t_wchar c); -- 2.32.0
0004-Use-wcwidth-to-compute-the-textual-port-column.patch
(text/x-patch, 8.6 KB)
From 25f118cab79ec613abd24a95b097c51d4a2c7b0b Mon Sep 17 00:00:00 2001 From: Vivien Kraus <[email protected]> Date: Tue, 3 Aug 2021 23:50:37 +0200 Subject: [PATCH 4/4] Use wcwidth to compute the textual port column The wcwidth function computes the number of columns that a character should span. By using it in the textual port column computation, we increase the precision of error reporting [1]. [1] https://www.gnu.org/prep/standards/html_node/Errors.html#Errors --- doc/ref/api-io.texi | 4 +- libguile/ports.c | 28 ++++++++----- module/ice-9/suspendable-ports.scm | 43 +++++++++----------- test-suite/tests/ports.test | 65 +++++++++++++++++++++++++++--- 4 files changed, 101 insertions(+), 39 deletions(-) diff --git a/doc/ref/api-io.texi b/doc/ref/api-io.texi index 87d4af496..70c7fdc61 100644 --- a/doc/ref/api-io.texi +++ b/doc/ref/api-io.texi @@ -503,7 +503,9 @@ The @code{put-string} procedure returns an unspecified value. Textual ports have a textual position associated with them: a line and a column. Reading in characters or writing them out advances the line and -the column appropriately. +the column appropriately. The textual position of a port increases +monotonically, and printable ASCII characters increase the position +strictly. @deffn {Scheme Procedure} port-column port @deffnx {Scheme Procedure} port-line port diff --git a/libguile/ports.c b/libguile/ports.c index c25c20709..d1d07819c 100644 --- a/libguile/ports.c +++ b/libguile/ports.c @@ -38,6 +38,7 @@ #include <uniconv.h> #include <unistd.h> #include <unistr.h> +#include <wchar.h> #ifdef HAVE_IO_H #include <io.h> @@ -1704,13 +1705,8 @@ update_port_position (SCM position, scm_t_wchar c) switch (c) { - case '\a': case EOF: break; - case '\b': - if (column > 0) - scm_port_position_set_column (position, scm_from_int (column - 1)); - break; case '\n': { long line = scm_to_long (scm_port_position_line (position)); @@ -1718,15 +1714,29 @@ update_port_position (SCM position, scm_t_wchar c) scm_port_position_set_column (position, SCM_INUM0); } break; - case '\r': - scm_port_position_set_column (position, SCM_INUM0); - break; case '\t': scm_port_position_set_column (position, scm_from_int (column + 8 - column % 8)); break; default: - scm_port_position_set_column (position, scm_from_int (column + 1)); + { + int width = 1; + if (c < 32) + { + /* non-printable ASCII character */ + width = 0; + } + if (c >= 128) + { + /* non-ASCII character */ + width = wcwidth (c); + if (width == -1) + { + width = 1; + } + } + scm_port_position_set_column (position, scm_from_int (column + width)); + } break; } } diff --git a/module/ice-9/suspendable-ports.scm b/module/ice-9/suspendable-ports.scm index a823f1d37..73e80afb4 100644 --- a/module/ice-9/suspendable-ports.scm +++ b/module/ice-9/suspendable-ports.scm @@ -559,29 +559,26 @@ (lambda (buf bv cur buffered) (slow-path)))) (define-inlinable (advance-port-position! pos char) - ;; FIXME: this cond is a speed hack; really we should just compile - ;; `case' better. - (cond - ;; FIXME: char>? et al should compile well. - ((<= (char->integer #\space) (char->integer char)) - (set-port-position-column! pos (1+ (port-position-column pos)))) - (else - (case char - ((#\alarm) #t) ; No change. - ((#\backspace) - (let ((col (port-position-column pos))) - (when (> col 0) - (set-port-position-column! pos (1- col))))) - ((#\newline) - (set-port-position-line! pos (1+ (port-position-line pos))) - (set-port-position-column! pos 0)) - ((#\return) - (set-port-position-column! pos 0)) - ((#\tab) - (let ((col (port-position-column pos))) - (set-port-position-column! pos (- (+ col 8) (remainder col 8))))) - (else - (set-port-position-column! pos (1+ (port-position-column pos)))))))) + (if (>= (char->integer char) 128) + ;; non-ASCII character + (let ((width (wcwidth char))) + (set-port-position-column! + pos + (+ (port-position-column pos) + (if (= width -1) 1 width)))) + ;; ASCII character + (case char + ((#\newline) + (set-port-position-line! pos (1+ (port-position-line pos))) + (set-port-position-column! pos 0)) + ((#\tab) + (let ((col (port-position-column pos))) + (set-port-position-column! pos (- (+ col 8) (remainder col 8))))) + (else + ;; non-printable ASCII characters have 0 width + (when (>= (char->integer char) 32) + ;; printable ASCII character + (set-port-position-column! pos (1+ (port-position-column pos)))))))) (define* (read-char #:optional (port (current-input-port))) (define (finish buf char) diff --git a/test-suite/tests/ports.test b/test-suite/tests/ports.test index cd87640ab..276e3c4de 100644 --- a/test-suite/tests/ports.test +++ b/test-suite/tests/ports.test @@ -1210,7 +1210,7 @@ (pass-if "x\\x08 backspace" (let ((port (open-output-string))) (display "x\x08" port) - (= 0 (port-column port)))) + (= 1 (port-column port)))) (pass-if "\\n" (let ((port (open-output-string))) @@ -1230,7 +1230,7 @@ (pass-if "x\\r" (let ((port (open-output-string))) (display "x\r" port) - (= 0 (port-column port)))) + (= 1 (port-column port)))) (pass-if "\\t" (let ((port (open-output-string))) @@ -1240,7 +1240,31 @@ (pass-if "x\\t" (let ((port (open-output-string))) (display "x\t" port) - (= 8 (port-column port))))) + (= 8 (port-column port)))) + + (pass-if "漢字 in C locale" + (setlocale LC_CTYPE "C") + (let ((port (open-output-string))) + (display "漢字" port) + (= 2 (port-column port)))) + + (pass-if "x漢字 in C locale" + (setlocale LC_CTYPE "C") + (let ((port (open-output-string))) + (display "x漢字" port) + (= 3 (port-column port)))) + + (pass-if "漢字 in UTF-8 locale" + (setlocale LC_CTYPE "en_US.UTF-8") + (let ((port (open-output-string))) + (display "漢字" port) + (= 4 (port-column port)))) + + (pass-if "x漢字 in UTF-8 locale" + (setlocale LC_CTYPE "en_US.UTF-8") + (let ((port (open-output-string))) + (display "x漢字" port) + (= 5 (port-column port))))) (with-test-prefix "input" @@ -1267,7 +1291,7 @@ (pass-if "x\\x08 backspace" (let ((port (open-input-string "x\x08"))) (while (not (eof-object? (read-char port)))) - (= 0 (port-column port)))) + (= 1 (port-column port)))) (pass-if "\\n" (let ((port (open-input-string "\n"))) @@ -1287,7 +1311,7 @@ (pass-if "x\\r" (let ((port (open-input-string "x\r"))) (while (not (eof-object? (read-char port)))) - (= 0 (port-column port)))) + (= 1 (port-column port)))) (pass-if "\\t" (let ((port (open-input-string "\t"))) @@ -1297,7 +1321,36 @@ (pass-if "x\\t" (let ((port (open-input-string "x\t"))) (while (not (eof-object? (read-char port)))) - (= 8 (port-column port)))))) + (= 8 (port-column port)))) + + (pass-if "漢字 in C locale" + (setlocale LC_CTYPE "C") + (let ((port (open-input-string "漢字"))) + (while (not (eof-object? (read-char port)))) + ;; There are 2 characters, but wcwidth failed so it + ;; returned 2. + (= 2 (port-column port)))) + + (pass-if "x漢字 in C locale" + (setlocale LC_CTYPE "C") + (let ((port (open-input-string "x漢字"))) + (while (not (eof-object? (read-char port)))) + (= 3 (port-column port)))) + + (pass-if "漢字 in UTF-8 locale" + (setlocale LC_CTYPE "en_US.UTF-8") + (let ((port (open-input-string "漢字"))) + (while (not (eof-object? (read-char port)))) + (setlocale LC_CTYPE "C") + ;; there are only 2 characters, but 4 columns + (= 4 (port-column port)))) + + (pass-if "x漢字 in UTF-8 locale" + (setlocale LC_CTYPE "en_US.UTF-8") + (let ((port (open-input-string "x漢字"))) + (while (not (eof-object? (read-char port)))) + (setlocale LC_CTYPE "C") + (= 5 (port-column port)))))) (with-test-prefix "port-line" -- 2.32.0