Re: wctomb() accepts out-of-range character in C-locale

[CC Bruno Haible, gnulib maintainer, to kick my memory]

Hi Jun,

On Mar 25 16:45, Jun T wrote:
> Dear newlib developers,
> (this is the first time I post to this list)
> 
> On recent Cygwin, the following C code output '1' (i.e., wide character
> 0x80 can be converted into a valid single-byte character in C-locale):
> 
> ---------------------------------------
> #include <stdio.h>
> #include <stdlib.h>
> #include <locale.h>
> 
> int main() {
>     char buf[MB_CUR_MAX];
>     setlocale(LC_ALL, "C");
>     printf("%d\n", wctomb(buf, 0x80));
>     return 0;
> }
> ---------------------------------------
> 
> On Linux it outputs '-1'.
> 
> It seems this is due to the following commit:
> 
> ------------------------------------------------
> commit 8a4318943875cd922601d34e54ce8a83ad2e733c
> Author: Corinna Vinschen <[email protected]>
> Date:   Mon Jul 31 12:44:16 2023 +0200
> 
>     Revert "* libc/stdlib/mbtowc_r.c (__ascii_mbtowc): Disallow conversion of"
> 
>     This reverts commit 2b77087a48ea56e77fca5aeab478c922f6473d7c.
> 
>     For some reason lost in time, commit 2b77087a48ea5 introduced
>     Cygwin-specific code treating single byte characters outside the
>     portable character set as illegal chars.  However, Cygwin was
>     always alone with this over-correct behaviour and it leads to
>     stuff like gnulib replacing functions defined in Cygwin with
>     their own implementation just due to that.
> ------------------------------------------------
> 
> Probably the function __ascii_wctomb() is used not only in C-locale
> but also in some other locales, and the commit is for "fixing"
> some problems in these locales?

No, __ascii_wctomb is by default used in "C".

> But a wide character >= 0x80 can't be converted into a valid
> character in C-loccale (7bit), I think.

Yes, I know, and that was what the original code from 2b77087a48 did.
But at the time I reverted this special handling, Bruno had reported a
change in gnulib in terms of fnmatch starting at
https://cygwin.com/pipermail/cygwin/2023-July/254017.html

During testing I found that gnulib was replacing various functions built
into Cygwin for several reasons, and one of them was that the conversion
of wide char to multibyte in the "C" locale was not transparently
converting chars from 0x80 up to 0xff.

I'm actually puzzled right now that this doesn't work in GLibc either.

Bruno, I really need your input here, because I just don't remember :(

Do you have an idea what gnulib configure test might have been the
trigger for the above revert?

And if GLibc also doesn't let chars >= 0x80 slip through, then Cygwin's
special handling was right.  But then this would introduce gnulib
trouble again...

Can you help us?


Thanks,
Corinna