Re: curses inch()/mvinch() mangle non-ASCII characters of an 8-bit locale on a wide build

--OYO47bQtVfL6INdT
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Aug 03, 2026 at 02:38:10PM +0300, Serhiy Storchaka wrote:
> One more thing about this, separate from whether the 8-bit limit is
> intended: winch() does not do what curs_inch.3x says.
>=20
> The page says these functions "extract only the low-order eight bits of t=
he

The page says (since 2026/07/4):

       These functions do not fail if the window contains cells of curses c=
om=E2=80=90
       plex  characters;  that is, if they contain characters with codes wi=
der
       than eight bits (or greater than 255 as an unsigned  decimal  intege=
r).
       The returned value depends upon the locale:

       =E2=80=A2   If  the locale encoding is UTF-8, ncurses extracts only =
the low-or=E2=80=90
           der eight bits of the character code from the cell.

       =E2=80=A2   If the locale encoding is not UTF-8, ncurses converts th=
e character
           from Unicode to the locale encoding,  and  extracts  the  low-or=
der
           eight bits from the resulting character.

Only eight bits are used, because the result is a chtype (and in ncurses
that holds only an 8-bit character).

> character code from the cell".=C2=A0 On a wide build they extract all of =
it --
> the
> bits above the eighth are not discarded, they are OR-ed into the attribute
> and colour fields.

thanks (the report as I read it was mainly for the non-UTF-8 case)

I'll make a further tweak to that

I could either do
	    result =3D ((chtype) UChar(CharOf(*cell)) | AttrOf(*cell));

or check if CharOf(*cell) is > 255, and return ERR (i.e., the better soluti=
on).

fwiw, X/Open Curses says

	These functions are only guaranteed to operate reliably on character
	sets in which each character fits into a single byte, whose attributes
	can be expressed using only constants with the A_ prefix.

=2E..so simply returning an error if it doesn't fit is an acceptable soluti=
on.

>=20
> =C2=A0 =C2=A0 setlocale(LC_ALL, "");
> =C2=A0 =C2=A0 initscr();
> =C2=A0 =C2=A0 start_color();
> =C2=A0 =C2=A0 init_pair(1, COLOR_RED, COLOR_BLACK);
> =C2=A0 =C2=A0 attrset(COLOR_PAIR(1));
> =C2=A0 =C2=A0 mvaddstr(0, 0, "=E2=82=AC");=C2=A0 =C2=A0 =C2=A0 /* EURO SI=
GN, in a UTF-8 locale */
> =C2=A0 =C2=A0 chtype c =3D mvinch(0, 0);
>=20
> gives, for a cell that was written with colour pair 1:
>=20
> =C2=A0 =C2=A0 winch()=C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D 0x000021ac
> =C2=A0 =C2=A0 & A_CHARTEXT=C2=A0 =C2=A0=3D 0xac
> =C2=A0 =C2=A0 & A_COLOR=C2=A0 =C2=A0 =C2=A0 =3D 0x00002100
> =C2=A0 =C2=A0 PAIR_NUMBER()=C2=A0 =3D 33
>=20
> 0x20 is the high byte of U+20AC; it lands in A_COLOR, so the pair reads b=
ack
> as 33 rather than 1.
>=20
> lib_winch.c:
>=20
> =C2=A0 =C2=A0 returnChtype((chtype) CharOf(win->_line[win->_cury].text[wi=
n->_
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| AttrOf(wi=
n->_line[win->_cury].text[win->_curx]))
>=20
> and on a wide build CharOf() is the raw wchar_t (curses.priv.h:1464), whi=
le
> the narrow ChCharOf() masks with A_CHARTEXT (curses.priv.h:1427).
> the wide one the same way would make the code agree with the manual
> would leave the rendition bits usable.
>=20
> This is not about 8-bit locales -- the example above is UTF-8. Eve
> that wants nothing but the rendition from inch() gets a wrong answe
>=20
> We have worked around it in Python's curses binding, so nothing is blocked
> on
> our side; I am only reporting that the behaviour and the documentation
> disagree.
>=20
> =C2=A0 =C2=A0 https://github.com/python/cpython/issues/153862
>=20
>=20
>=20

--=20
Thomas E. Dickey <[email protected]>
https://invisible-island.net

--OYO47bQtVfL6INdT
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQGzBAABCgAdFiEEGYgtkt2kxADCLA1WzCr0RyFnvgMFAmpw6UcACgkQzCr0RyFn
vgPEygwAgdjfOGwcVlmniY1vQu+dJTRdIbVmNucAsNbIjRUWUSHdl4SdOdIZlYdW
PRdEFg0OR5CsRNbQ/g4SRiJQOHgggLQ4PtG1++8AZeuXeGnrpBnk0eU32z5lIkdF
DyMlXFUIJCIJNmYmiPg8BbMrOousNoKd1JWUg9SLBF39mobe/mGHEk5t0Ww8uM7i
Cp1d3qddzBdbKFrP0zsBpIpRCDHH3dSkyf2Rb8Zj8ZRp5gbIy9xvXnKw5/TFjSnp
TpVWT0mtadPic6dpYcuD3iLRU8J7qje6uCLfcsmX9p8S4hmVZpDgTpGVJIsYslHU
9WWOyXADGfFmaBQ/AkGkBm17PpIb8wihRaqyn8mcj3zxXXTLnmTdZ1Q64sL0TOSN
K8MaddlJE5XxO1ZKRMsbMXGMzY687cFwanioTXp+zwz9+xduRVOboJ1TAVZlpImB
pXgSO/pd3sM7Jprl/tYKJSSXR7bQDmEx6xLoq4y/6EOEfHWs3Xq/OzaiI6hi6s7T
+AUY31ft
=4OgM
-----END PGP SIGNATURE-----

--OYO47bQtVfL6INdT--