Re: new test failures on Cygwin and OpenBSD
Bruno Haible via Gnulib discussion list <[email protected]>
| Newsgroups | gmane.comp.lib.gnulib.bugs |
|---|---|
| Message-ID | <2821197.7ZnGzRAi6d@cagnes> |
Paul Eggert wrote:
> > So, for me, (3) is the right approach: In the "C" locale, map 0x80..0xFF not
> > to U+0080..U+00FF but to U+DF80..U+DFFF, like musl libc does it. This will
> > ensure that c32isprint returns 0 on these inputs, like isprint does.
>
> This sounds good to me as well. It is unfortunate that the "C" locale, which is supposed to be fast, will have a small performance penalty on OpenBSD and Cygwin (and I assume native MS-Windows?) due to Gnulib having to replace mbrtoc32. But that's better than incorrect behavior.
Yes, I agree.
I used the program below to look at the behaviour of various platforms:
Result:
wchar_t world char32_t world
glibc mbrtowc failed OK
musl OK (...wc=0xDFFF) OK
macOS OK (...wc=0x00FF) OK
FreeBSD OK (...wc=0x00FF) OK
NetBSD OK (...wc=0x00FF) OK
OpenBSD ≤ 7.3 OK (...wc=0x00FF) OK
OpenBSD OK (...wc=0x00FF) FAIL
AIX 32-bit OK (...wc=0x00FF) OK
AIX 64-bit OK (...wc=0x00FF) OK
Solaris 11 OK (...wc=0x00FF) OK
Solaris OpenIndiana OK (...wc=0x00FF) OK
Cygwin mbrtowc failed FAIL
mingw isprint=0 -- ...wc=0x00FF iswprint=1 FAIL
MSVC isprint=0 -- ...wc=0x00FF iswprint=1 FAIL
Android mbrtowc failed OK ("C" is UTF-8)
Haiku mbrtowc failed OK
From this list, I infer that the correct condition
- for the Cygwin, mingw, MSVC case is:
_GL_SMALL_WCHAR_T && !GNULIB_defined_mbstate_t
- for the OpenBSD case is:
!MBRTOWC_IN_C_LOCALE_MAYBE_EILSEQ && HAVE_MBRTOC32 && !MUSL_LIBC
Implemented through the three attached patches.
Bruno
================================================================================
#include <locale.h>
#include <wchar.h>
#include <ctype.h>
#include <wctype.h>
#include <stdio.h>
int main ()
{
int result;
if (setlocale (LC_ALL, "C") == NULL)
return 2;
result = 0;
{
int i;
for (i = 0xC0; i <= 0xFF; i++)
{
char c = i;
wchar_t wc;
mbstate_t mbs = { 0, };
if (mbrtowc (&wc, &c, 1, &mbs) > 1)
{
printf ("mbrtowc failed for c=0x%02X\n", i);
return 3;
}
int is1 = !!isprint ((unsigned char) c);
int is2 = !!iswprint (wc);
printf ("c=0x%02X isprint=%d -- wc=0x%04X iswprint=%d\n", i, is1, wc, is2);
if (is1 != is2)
result = 1;
}
}
if (result == 0)
printf ("OK\n");
return result;
}
================================================================================
0001-mbrtoc32-Make-c32isprint-compatible-with-isprint-in-.patch
(text/x-patch, 4.7 KB)
From d05f95bec14dcd5d58aef358a520f15ec64c381b Mon Sep 17 00:00:00 2001 From: Bruno Haible <[email protected]> Date: Thu, 27 Aug 2026 23:53:07 +0200 Subject: [PATCH 1/3] mbrtoc32: Make c32isprint compatible with isprint in the C locale. Reported by Paul Eggert in <https://lists.gnu.org/archive/html/bug-gnulib/2026-08/msg00222.html>. * lib/mbrtoc32.c (mbrtoc32) [!GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T]: On Cygwin and native Windows, in the C locale, map 0x80..0xFF differently. * lib/btoc32.c: Include hard-locale.h. (btoc32): [!GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T]: On Cygwin and native Windows, in the C locale, map 0x80..0xFF differently. * modules/btoc32 (Depends-on): Add hard-locale. --- ChangeLog | 15 ++++++++++++++- lib/btoc32.c | 17 +++++++++++++++++ lib/mbrtoc32.c | 14 ++++++++++++-- modules/btoc32 | 1 + 4 files changed, 44 insertions(+), 3 deletions(-) diff --git a/ChangeLog b/ChangeLog index a02b8878b4..53aa25ecb2 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,16 @@ +2026-08-27 Bruno Haible <[email protected]> + + mbrtoc32: Make c32isprint compatible with isprint in the C locale. + Reported by Paul Eggert in + <https://lists.gnu.org/archive/html/bug-gnulib/2026-08/msg00222.html>. + * lib/mbrtoc32.c (mbrtoc32) + [!GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T]: On Cygwin and native + Windows, in the C locale, map 0x80..0xFF differently. + * lib/btoc32.c: Include hard-locale.h. + (btoc32): [!GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T]: On Cygwin + and native Windows, in the C locale, map 0x80..0xFF differently. + * modules/btoc32 (Depends-on): Add hard-locale. + 2026-08-27 Bruno Haible <[email protected]> getlogin: Fix link error with option --with-systemd (regr. 2025-03-09). @@ -1862,7 +1875,7 @@ go back to the old way of doing things, without the recently-added USE_C_LOCALE stuff. Then make the following changes instead: * lib/quotearg.c [_QUOTEARG_AVOID_UCHAR_H]: Don’t include <uchar.h>. - (mbrto2c, mbstate_t) [_QUOTEARG_AVOID_UCHAR_H && defined __UCLIBC__ + (mbrtowc, mbstate_t) [_QUOTEARG_AVOID_UCHAR_H && defined __UCLIBC__ && !defined __UCLIBC_HAS_WCHAR__]: New macros. Also, don’t include <wchar.h> or <wctype.h>. (char32_t, c32isprint, mbrtoc32, GNULIB_MBRTOC32_REGULAR) diff --git a/lib/btoc32.c b/lib/btoc32.c index e4ca972db0..9af58b2582 100644 --- a/lib/btoc32.c +++ b/lib/btoc32.c @@ -26,6 +26,11 @@ #include <string.h> #include <wchar.h> +#if !GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T +# include "hard-locale.h" +# include <locale.h> +#endif + #if GL_CHAR32_T_IS_UNICODE # include "lc-charset-unicode.h" #endif @@ -51,6 +56,18 @@ btoc32 (int c) } return WEOF; #else +# if !GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T /* Cygwin, mingw, MSVC */ + if (!hard_locale (LC_CTYPE)) + { + /* In the "C" locale, map the bytes 0x80..0xFF to U+DF80..U+DFFF, so that + the c32is* functions return false on them, for consistency with the + <ctype.h> is* functions. */ + if (c != EOF) + return (c < 0x80 ? c : 0xDF00 + c); + else + return WEOF; + } +# endif /* In all known locale encodings, unibyte characters correspond only to characters in the BMP. */ wint_t wc = btowc (c); diff --git a/lib/mbrtoc32.c b/lib/mbrtoc32.c index 5f89e05460..ff4799ded1 100644 --- a/lib/mbrtoc32.c +++ b/lib/mbrtoc32.c @@ -92,7 +92,7 @@ mbrtoc32 (char32_t *pwc, const char *s, size_t n, mbstate_t *ps) # include "localcharset.h" # include "streq-opt.h" -# if MBRTOC32_IN_C_LOCALE_MAYBE_EILSEQ +# if MBRTOC32_IN_C_LOCALE_MAYBE_EILSEQ || _GL_SMALL_WCHAR_T # include "hard-locale.h" # include <locale.h> # endif @@ -291,7 +291,7 @@ mbrtoc32 (char32_t *pwc, const char *s, size_t n, mbstate_t *ps) return ret; -# elif _GL_SMALL_WCHAR_T +# elif _GL_SMALL_WCHAR_T /* Cygwin, mingw, MSVC */ /* Special-case all encodings that may produce wide character values > WCHAR_MAX. */ @@ -386,6 +386,16 @@ mbrtoc32 (char32_t *pwc, const char *s, size_t n, mbstate_t *ps) /* The conversion state is undefined, says POSIX. */ return (size_t)(-1); } + else if (!hard_locale (LC_CTYPE)) + { + /* In the "C" locale, map the bytes 0x80..0xFF to U+DF80..U+DFFF, so that + the c32is* functions return false on them, for consistency with the + <ctype.h> is* functions. */ + unsigned char c = (unsigned char) s[0]; + if (pwc != NULL) + *pwc = (c < 0x80 ? c : 0xDF00 + c); + return (c == 0 ? 0 : 1); + } else { wchar_t wc; diff --git a/modules/btoc32 b/modules/btoc32 index e2acebb402..d3e13be578 100644 --- a/modules/btoc32 +++ b/modules/btoc32 @@ -8,6 +8,7 @@ Depends-on: uchar-h mbszero mbrtoc32 +hard-locale btowc configure.ac: -- 2.53.0
0002-mbrtoc32-Make-c32isprint-compatible-with-isprint-in-.patch
(text/x-patch, 9.4 KB)
From 27353a8efc89559b855c9688a8c32183418267b7 Mon Sep 17 00:00:00 2001 From: Bruno Haible <[email protected]> Date: Fri, 28 Aug 2026 00:41:53 +0200 Subject: [PATCH 2/3] mbrtoc32: Make c32isprint compatible with isprint in the C locale. Reported by Paul Eggert in <https://lists.gnu.org/archive/html/bug-gnulib/2026-08/msg00222.html>. * m4/mbrtoc32.m4 (gl_MBRTOC32_C_LOCALE_LIKE_ISO_8859): New macro. (gl_FUNC_MBRTOC32): Define MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859 and set REPLACE_MBRTOC32 on platforms such as OpenBSD. * lib/mbrtoc32.c (mbrtoc32) [MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859]: In the C locale, map 0x80..0xFF differently. * lib/btoc32.c (btoc32) [MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859]: Likewise. * doc/posix-functions/mbrtoc32.texi: Mention the OpenBSD problem. --- ChangeLog | 12 +++++++ doc/posix-functions/mbrtoc32.texi | 5 +++ lib/btoc32.c | 28 ++++++++-------- lib/mbrtoc32.c | 23 +++++++++++-- m4/mbrtoc32.m4 | 55 +++++++++++++++++++++++++++++-- 5 files changed, 105 insertions(+), 18 deletions(-) diff --git a/ChangeLog b/ChangeLog index 53aa25ecb2..b829acc302 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,17 @@ 2026-08-27 Bruno Haible <[email protected]> + mbrtoc32: Make c32isprint compatible with isprint in the C locale. + Reported by Paul Eggert in + <https://lists.gnu.org/archive/html/bug-gnulib/2026-08/msg00222.html>. + * m4/mbrtoc32.m4 (gl_MBRTOC32_C_LOCALE_LIKE_ISO_8859): New macro. + (gl_FUNC_MBRTOC32): Define MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859 and + set REPLACE_MBRTOC32 on platforms such as OpenBSD. + * lib/mbrtoc32.c (mbrtoc32) [MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859]: + In the C locale, map 0x80..0xFF differently. + * lib/btoc32.c (btoc32) [MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859]: + Likewise. + * doc/posix-functions/mbrtoc32.texi: Mention the OpenBSD problem. + mbrtoc32: Make c32isprint compatible with isprint in the C locale. Reported by Paul Eggert in <https://lists.gnu.org/archive/html/bug-gnulib/2026-08/msg00222.html>. diff --git a/doc/posix-functions/mbrtoc32.texi b/doc/posix-functions/mbrtoc32.texi index 629a75dec4..72a373b3e6 100644 --- a/doc/posix-functions/mbrtoc32.texi +++ b/doc/posix-functions/mbrtoc32.texi @@ -22,6 +22,11 @@ @c https://sourceware.org/PR29511 glibc 2.35. @item +In the C or POSIX locales, +this function returns values like in ISO-8859-1 locales, +which leads to @code{c32isprint} etc.@: being inconsistent with @code{isprint}: +OpenBSD 7.9. +@item This function returns 0 instead of @code{(size_t) -2} when the input is empty: @c https://sourceware.org/PR16950 diff --git a/lib/btoc32.c b/lib/btoc32.c index 9af58b2582..31b3890723 100644 --- a/lib/btoc32.c +++ b/lib/btoc32.c @@ -26,7 +26,8 @@ #include <string.h> #include <wchar.h> -#if !GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T +#if MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859 \ + || (!GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T) # include "hard-locale.h" # include <locale.h> #endif @@ -41,6 +42,19 @@ _GL_EXTERN_INLINE wint_t btoc32 (int c) { +#if MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859 /* OpenBSD */ \ + || (!GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T) /* Cygwin, mingw, MSVC */ + if (!hard_locale (LC_CTYPE)) + { + /* In the "C" locale, map the bytes 0x80..0xFF to U+DF80..U+DFFF, so that + the c32is* functions return false on them, for consistency with the + <ctype.h> is* functions. */ + if (c != EOF) + return (c < 0x80 ? c : 0xDF00 + c); + else + return WEOF; + } +#endif #if HAVE_WORKING_MBRTOC32 && HAVE_WORKING_C32RTOMB && !_GL_WCHAR_T_IS_UCS4 /* The char32_t encoding of a multibyte character may be different than its wchar_t encoding. */ @@ -56,18 +70,6 @@ btoc32 (int c) } return WEOF; #else -# if !GNULIB_defined_mbstate_t && _GL_SMALL_WCHAR_T /* Cygwin, mingw, MSVC */ - if (!hard_locale (LC_CTYPE)) - { - /* In the "C" locale, map the bytes 0x80..0xFF to U+DF80..U+DFFF, so that - the c32is* functions return false on them, for consistency with the - <ctype.h> is* functions. */ - if (c != EOF) - return (c < 0x80 ? c : 0xDF00 + c); - else - return WEOF; - } -# endif /* In all known locale encodings, unibyte characters correspond only to characters in the BMP. */ wint_t wc = btowc (c); diff --git a/lib/mbrtoc32.c b/lib/mbrtoc32.c index ff4799ded1..826f367708 100644 --- a/lib/mbrtoc32.c +++ b/lib/mbrtoc32.c @@ -92,7 +92,9 @@ mbrtoc32 (char32_t *pwc, const char *s, size_t n, mbstate_t *ps) # include "localcharset.h" # include "streq-opt.h" -# if MBRTOC32_IN_C_LOCALE_MAYBE_EILSEQ || _GL_SMALL_WCHAR_T +# if (MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859 \ + || MBRTOC32_IN_C_LOCALE_MAYBE_EILSEQ \ + || _GL_SMALL_WCHAR_T) # include "hard-locale.h" # include <locale.h> # endif @@ -134,8 +136,10 @@ mbrtoc32 (char32_t *pwc, const char *s, size_t n, mbstate_t *ps) n = 1; } -# if (MBRTOC32_EMPTY_INPUT_BUG || _GL_SMALL_WCHAR_T \ - || (GNULIB_WCHAR_SINGLE_LOCALE && __GLIBC__ >= 2 && !__UCLIBC__)) +# if (MBRTOC32_EMPTY_INPUT_BUG \ + || (GNULIB_WCHAR_SINGLE_LOCALE && __GLIBC__ >= 2 && !__UCLIBC__) \ + || MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859 \ + || _GL_SMALL_WCHAR_T) if (n == 0) return (size_t) -2; # endif @@ -262,6 +266,19 @@ mbrtoc32 (char32_t *pwc, const char *s, size_t n, mbstate_t *ps) /* mbrtoc32() may produce different values for wc than mbrtowc(). Therefore use mbrtoc32(). */ +# if MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859 /* OpenBSD */ + if (!hard_locale (LC_CTYPE)) + { + /* In the "C" locale, map the bytes 0x80..0xFF to U+DF80..U+DFFF, so that + the c32is* functions return false on them, for consistency with the + <ctype.h> is* functions. */ + unsigned char c = (unsigned char) s[0]; + if (pwc != NULL) + *pwc = (c < 0x80 ? c : 0xDF00 + c); + return (c == 0 ? 0 : 1); + } +# endif + # if defined _WIN32 && !defined __CYGWIN__ char32_t wc; size_t ret = mbrtoc32 (&wc, s, n, ps); diff --git a/m4/mbrtoc32.m4 b/m4/mbrtoc32.m4 index 9eb9a7e223..2631695713 100644 --- a/m4/mbrtoc32.m4 +++ b/m4/mbrtoc32.m4 @@ -1,5 +1,5 @@ # mbrtoc32.m4 -# serial 25 +# serial 26 dnl Copyright (C) 2014-2026 Free Software Foundation, Inc. dnl This file is free software; the Free Software Foundation dnl gives unlimited permission to copy and/or distribute it, @@ -9,6 +9,7 @@ AC_DEFUN([gl_FUNC_MBRTOC32], [ AC_REQUIRE([gl_UCHAR_H_DEFAULTS]) + AC_REQUIRE([AC_CANONICAL_HOST]) AC_REQUIRE([AC_TYPE_MBSTATE_T]) dnl Determine REPLACE_MBSTATE_T, from which GNULIB_defined_mbstate_t is @@ -28,6 +29,7 @@ AC_DEFUN([gl_FUNC_MBRTOC32] else gl_MBRTOC32_EMPTY_INPUT gl_MBRTOC32_C_LOCALE + gl_MBRTOC32_C_LOCALE_LIKE_ISO_8859 gl_MBRTOC32_UTF8_LOCALE case "$gl_cv_func_mbrtoc32_empty_input" in *yes) ;; @@ -43,6 +45,17 @@ AC_DEFUN([gl_FUNC_MBRTOC32] REPLACE_MBRTOC32=1 ;; esac + case "$gl_cv_func_mbrtoc32_C_locale_sans_EILSEQ" in + *yes) + case "$gl_cv_func_mbrtoc32_C_locale_like_iso_8859" in + *yes) + AC_DEFINE([MBRTOC32_IN_C_LOCALE_MAYBE_LIKE_ISO_8859], [1], + [Define if the mbrtoc32 function in the C locale may work like in an ISO-8859-1 locale.]) + REPLACE_MBRTOC32=1 + ;; + esac + ;; + esac case "$gl_cv_func_mbrtoc32_utf8_locale_works" in *yes) ;; *) AC_DEFINE([MBRTOC32_MULTIBYTE_LOCALE_BUG], [1], @@ -167,7 +180,7 @@ AC_DEFUN([gl_MBRTOC32_EMPTY_INPUT] AC_DEFUN([gl_MBRTOC32_C_LOCALE], [ AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles - AC_CACHE_CHECK([whether the C locale is free of encoding errors], + AC_CACHE_CHECK([whether mbrtoc32 in the C locale is free of encoding errors], [gl_cv_func_mbrtoc32_C_locale_sans_EILSEQ], [AC_RUN_IFELSE( [AC_LANG_PROGRAM( @@ -204,6 +217,44 @@ AC_DEFUN([gl_MBRTOC32_C_LOCALE] ]) ]) +dnl Test whether mbrtoc32 in the C locale works like in an ISO-8859-1 locale. + +AC_DEFUN([gl_MBRTOC32_C_LOCALE_LIKE_ISO_8859], +[ + AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles + AC_CACHE_CHECK([whether mbrtoc32 in the C locale works like in an ISO-8859-1 locale], + [gl_cv_func_mbrtoc32_C_locale_like_iso_8859], + [AC_RUN_IFELSE( + [AC_LANG_PROGRAM( + [[#include <limits.h> + #include <locale.h> + #ifdef __HAIKU__ + #include <stdint.h> + #endif + #include <uchar.h> + ]], [[ + int i; + const char *locale = setlocale (LC_ALL, "C"); + if (! locale) + return 2; + char c = 0xE0; + char32_t wc; + mbstate_t mbs = { 0, }; + size_t ss = mbrtoc32 (&wc, &c, 1, &mbs); + if (ss == 1 && wc == 0x00E0) + return 3; + return 0; + ]])], + [gl_cv_func_mbrtoc32_C_locale_like_iso_8859=no], + [gl_cv_func_mbrtoc32_C_locale_like_iso_8859=yes], + [case "$host_os" in + *-musl* | midipix*) gl_cv_func_mbrtoc32_C_locale_like_iso_8859="no" ;; + *) gl_cv_func_mbrtoc32_C_locale_like_iso_8859="possibly yes" ;; + esac + ]) + ]) +]) + dnl Test whether mbrtoc32 works when it's fed the bytes one-by-one in an UTF-8 dnl locale. -- 2.53.0
0003-tests-Check-that-c32isprint-is-consistent-with-ispri.patch
(text/x-patch, 2.1 KB)
From de054f5dc8906dbac0054e113d7d4fd0430a3398 Mon Sep 17 00:00:00 2001 From: Bruno Haible <[email protected]> Date: Fri, 28 Aug 2026 00:43:56 +0200 Subject: [PATCH 3/3] tests: Check that c32isprint is consistent with isprint in the C locale. * tests/test-c32isprint.c: Include <ctype.h>. (main): In the C locale, check that c32isprint is consistent with isprint. --- ChangeLog | 5 +++++ tests/test-c32isprint.c | 15 +++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/ChangeLog b/ChangeLog index b829acc302..2254b198b1 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,10 @@ 2026-08-27 Bruno Haible <[email protected]> + tests: Check that c32isprint is consistent with isprint in the C locale. + * tests/test-c32isprint.c: Include <ctype.h>. + (main): In the C locale, check that c32isprint is consistent with + isprint. + mbrtoc32: Make c32isprint compatible with isprint in the C locale. Reported by Paul Eggert in <https://lists.gnu.org/archive/html/bug-gnulib/2026-08/msg00222.html>. diff --git a/tests/test-c32isprint.c b/tests/test-c32isprint.c index 9d91ef3640..42a5092d4f 100644 --- a/tests/test-c32isprint.c +++ b/tests/test-c32isprint.c @@ -21,6 +21,7 @@ #include "signature.h" SIGNATURE_CHECK (c32isprint, int, (wint_t)); +#include <ctype.h> #include <locale.h> #include <stdlib.h> #include <string.h> @@ -106,6 +107,20 @@ main (int argc, char *argv[]) { case '0': /* C locale; tested above. */ +#if !defined __ANDROID__ + /* On Android ≥ 5.0, the default locale is the "C.UTF-8" locale, not the + "C" locale. Furthermore, when you attempt to set the "C" or "POSIX" + locale via setlocale(), what you get is a "C" locale with UTF-8 + encoding, that is, effectively the "C.UTF-8" locale. */ + /* Check that c32isprint is consistent with isprint. */ + for (int c = 0x80; c < 0x100; c++) + { + ASSERT (!isprint ((unsigned char) c)); + buf[0] = (unsigned char) c; + is = for_character (buf, 1); + ASSERT (is == 0); + } +#endif return test_exit_status; case '1': -- 2.53.0