Re: RFC: mbrtowc() and wcrtomb() implementation problems

> From: Keith Marshall <[email protected]>
> Date: Wed, 12 Feb 2020 22:03:17 +0000
> 
> 1) There is no true MBCS support; only conversions between wchar_t and
> DBCS codesets (corresponding to MB_LEN_MAX = 2) are supported.  Today,
> Windows supports MBCS codesets with MB_LEN_MAX = 5.
> 
> 2) Representation of wchar_t as UTF-16LE entities makes it impossible to
> have effective handling, in mbrtowc() and wcrtomb(), for code points
> which lie off the BMP.  Such code points are represented by surrogate
> pairs, in UTF-16, and there is no standards conformant mechanism for
> passing such surrogate pairs through the single wchar_t argument to
> either of these functions.
> 
> I can address the first of these issues; the second is more problematic.
>  I can imagine subverting use of the mbstate_t reference argument, to
> store intermediate state for partially completed surrogate conversions,
> but what would be an appropriate function return value, in such cases?
> Alternatively, should valid surrogate pair representations just be
> reported as invalid, for the purpose of such conversions?
> 
> What do you, the users of MinGW, think?

I think you can give up on 2).  Wide-character support in the CRT
routines is fundamentally broken on MS-Windows, due to the use of
UTF-16, for any codepoint beyond the BMP.  The example that cries the
loudest is 'wcschr', which accepts a single wchar_t value as its
second argument, which means you cannot search for any character
beyond the BMP.  Moreover, at least several functions internally don't
support codepoints beyond the BMP even if it were possible given the
API.  I forget the details, but I think I saw this with _wcslwr and
_wcsupr.

I don't think this can be fixed as long as wchar_t remains a 16-bit
data type.  People who need their MinGW programs to do better should
either (a) convert everything to UTF-8 and write their own code to
manipulate UTF-8 strings, or (b) use replacements such as Gnulib
(which, quite expectedly, uses a 32-bit data type for wide
characters).

So I think you should just document this as a Windows restriction, and
move on.

_______________________________________________
MinGW-Users mailing list
[email protected]

This list observes the Etiquette found at
http://www.mingw.org/Mailing_Lists.
We ask that you be polite and do the same.  Disregard for the list etiquette may cause your account to be moderated.

_______________________________________________
You may change your MinGW Account Options or unsubscribe at:
https://lists.osdn.me/mailman/listinfo/mingw-users
Also: mailto:[email protected]?subject=unsubscribe