| Newsgroups |
gmane.text.unicode.general |
| Message-ID |
<[email protected]> |
--2OYMHQXURATGUGQGORGOHnhgwp
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8
The SEW subsequently explained that the actual reason is due to insufficien=
t evidence of user community that would need to use the resulting mapping. =
Despite Win32 being a highly popular platform with plenty of backwards comp=
atibility and native UCS-2 terminal support, the specific use cases of inst=
alling codepages into Windows NT and using terminal tiles from Windows 3.1/=
95/98/ME are not sufficiently documented, making it difficult for any user =
communities to form around it. So it seems like the idea of standardizing l=
egacy Arabic terminal BMP mappings is a dead end for now. Dnia 17 kwietnia =
2026 22:59 [email protected] via Unicode < [email protected]=
rg > napisa=C5=82(a): The Recommendations in L2/26-100 claim that Micro=
soft's documentation of legacy Arabic encodings is available at https:/=
/learn.microsoft.com/en-us/typography/legacy/legacy_arabic_fonts. However, =
that article only demonstrates two encodings of TrueType fonts, which are u=
sed in Windows 3.1 but are completely different from the eight terminal enc=
odings. Unlike the TrueType encodings which represent internal shaping mapp=
ings and are not used for text interchange, the terminal encodings have bee=
n demonstrated to be directly used in text interchange through int 10h and =
ReadConsoleOutputA/WriteConsoleOutputA as already demonstrated in L2/26-077=
. The Recommendations also claim that the proposal does not demonstrate any=
need for interchange or encoding, but the proposal actually demonstrated s=
uch a need due to the logical extension of the Win32 terminal API to the fu=
nctions ReadConsoleOutputW/WriteConsoleOutputW, which are in Windows NT and=
may be used on the output of previously ran programs (including those that=
used the legacy Arabic terminal encodings), which given the CHAR_INFO stru=
cture, therefore implies a need for all the tiles to map to BMP for interch=
ange. I'm not objecting to the SEW's conclusion of "Users are e=
xpected to use PUA.", which can indeed be used to provide a mapping eve=
n if not standardized, but the reasoning given was flawed. Dnia 09 styczni=
a 2026 17:25 [email protected] < [email protected] > napisa=
=C5=82(a): The following Win32 C code will output 256 characters in system =
console codepage into the character grid, capture those character tiles in =
UCS-2 if possible, and then output the current console codepage number. #=
include <windows.h> #include <stdio.h> int main(){ HANDLE hC=
onsole=3DGetStdHandle(STD_OUTPUT_HANDLE); CHAR_INFO screen[256]; COORD si=
ze=3D{16,16,}; COORD pos=3D{0,0,}; SMALL_RECT rect=3D{0,0,15,15,}; for(i=
nt i=3D0;i<256;i++){ screen[i].Attributes=3D0xF0; screen[i].Char.Ascii=
Char=3Di; } WriteConsoleOutputA(hConsole,screen,size,pos,&rect); CHA=
R_INFO screenu[256]; if(ReadConsoleOutputW(hConsole,screenu,size,pos,&=
rect)){ for(int i=3D0;i<256;i++) printf("%04X ",screenu[i].Char=
.UnicodeChar); } else{ printf("error %08X\n",GetLastError()); }=
printf("codepage %u",GetConsoleOutputCP()); } In most cases, w=
henever a legacy Win32 codepage is used, the application can run on Windows=
NT to capture the UCS-2 mapping of those character cells to the BMP (altho=
ugh for CJK codepages a more complex setup would be necessary due to thousa=
nds of fullwidth characters with 2-byte sequences). However, in Arabic ve=
rsions of Windows 9x (95/98/ME) the resulting character set has many presen=
tation forms that are not in Unicode. This is the result when running on Wi=
ndows ME:=C2=A0 i.imgur.com https://i.imgur.com/QFm3SkI.png =C2=A0in 10=C3=
=9720 font, i.imgur.com https://i.imgur.com/KUbLQ0A.png =C2=A0in 10=C3=971=
8 font (same result also appears in Windows 95/98). 5=C3=9712, 7=C3=9712, 8=
=C3=9712, 10=C3=9718, 10=C3=9720, and 12=C3=9716 bitmap fonts have been att=
ested with that character set (VGAOEM.FON, 8514OEM.FON, DOSAPP.FON). The 10=
=C3=9720 font has slightly different mapping than the other sizes: 0x93 is =
=C3=B6 instead of =C3=B4, and 0x97 is missing (causing the following charac=
ters on the same line to be drawn at the wrong position). It also claims to=
be using codepage 720, but many characters differ from their CP720 mapping=
s, including the bundled=C2=A0CP_720.NLS mappings (for example, =D9=80 (U+0=
640 ARABIC TATWEEL) is 0x95 in CP720, but in the console 0x95 is =D8=B4 ins=
tead, and the tatweel is at 0xFF). On Windows 9x,=C2=A0ReadConsoleOutputW i=
s not supported so the UCS-2 mappings of the console character tiles cannot=
be captured (error 0x00000078 ERROR_CALL_NOT_IMPLEMENTED). When that pro=
gram runs on Arabic versions of Windows NT, the visual output is of the CP4=
37 character set if one of the bundled bitmap fonts is used ( i.imgur.com h=
ttps://i.imgur.com/RxjtxMH.png ), or the CP720 set if Lucida Console is use=
d, with the Arabic letters either having glitchy font substitution (NT 4.0,=
NT 5.0/2000) or the .notdef glyph (NT 5.1/XP and up). In fact, it seems th=
at the only Arabic bitmap fonts that occur in Windows NT are CP1256 fonts, =
which are not used in terminals. So this appears to be one of those permane=
nt Windows compatibility regressions that occured when Windows 9x ended, wh=
ere the terminals can no longer render legacy Arabic text. Even if the user=
managed to use registry hacks to set the font to Courier New or Simplified=
Arabic Fixed, it would still use the CP720 mapping which is not compatible=
with the Windows 9x set. It appears that in the Windows 9x Arabic termin=
al character set, 244 characters (=E2=80=87=EF=BA=80=EF=BA=81=EF=BA=82=EF=
=BA=83=EF=BA=84=EF=BA=85=EF=BA=87=EF=BA=88=EF=BA=8A=EF=BA=8B=EF=BA=8D=EF=BA=
=8E=EF=BA=8F=EF=BA=91=EF=BA=93=E2=96=BA=E2=97=84=E2=86=95=EF=BA=95=C2=B6=C2=
=A7=EF=BA=97=EF=BA=99=E2=86=91=E2=86=93=E2=86=92=E2=86=90=EF=BA=9B=EF=B9=B0=
=E2=96=B2=E2=96=BC !"#$%&'()*+,-./0123456789:;<=3D>?@ABCD=
EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~=EF=BA=9D=EF=BA=
=9F=EF=BA=A1=C3=A9=C3=A2=EF=BA=A3=C3=A0=EF=BA=A5=C3=A7=C3=AA=C3=AB=C3=A8=C3=
=AF=C3=AE=EF=BA=A7=EF=BA=A9=EF=BA=AB=EF=BA=AD=EF=BA=AF=C3=B4=EF=BA=B3=C3=BB=
=C3=B9=EF=BA=B7=EF=BA=BB=C2=A3=EF=BA=BF=EF=BB=81=EF=BB=85=EF=BB=89=EF=BB=8A=
=EF=BB=8B=EF=BB=8C are already in Unicode, but 12 characters are not in Uni=
code: =E2=80=A2 6 of them are pieces of lam-alef ligatures (0xDD, 0xDE, 0x=
F9, 0xFB, 0xFC, 0xFD) =E2=80=A2 2 of them are shadda with fathatan ligatur=
es without or with tatweel (0xD0, 0xD1) =E2=80=94 in some legacy Microsoft=
fonts, shadda with fathatan is mapped to private use U+E818 =E2=80=A2 4 o=
f them are disunifications of seen/sheen/sad/dad occuring either with or wi=
thout tail =E2=80=94=C2=A0=EF=B9=B3 (U+FE73 ARABIC TAIL FRAGMENT) was orig=
inally encoded in Unicode 3.2 for CP864 compatibility; in that codepage, th=
e forms of=C2=A0seen/sheen/sad/dad attach to the tail fragment =E2=80=94 f=
orms with included tail:=C2=A00x92, 0x95, 0x98, 0x8A =E2=80=94 forms witho=
ut tail (attaching to tail fragment like in CP864):=C2=A00xF3, 0xF4, 0xF5, =
0xF6 If someone tried to make a Win32 console implementation and tried to=
implement both Windows 9x Arabic terminal character set compatibility and =
wide string API (ReadConsoleOutputW) compatibility simultaneously, then the=
y would run into the issue that there is currently no standardized mapping =
to handle that scenario. What should Windows 9x Arabic console compatible i=
mplementations do in that case?=0D
--2OYMHQXURATGUGQGORGOHnhgwp
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8
<p>The SEW subsequently explained that the actual reason is due to insuffic=
ient evidence of user community that would need to use the resulting mappin=
g. Despite Win32 being a highly popular platform with plenty of backwards c=
ompatibility and native UCS-2 terminal support, the specific use cases of i=
nstalling codepages into Windows NT and using terminal tiles from Windows 3=
.1/95/98/ME are not sufficiently documented, making it difficult for any us=
er communities to form around it. So it seems like the idea of standardizin=
g legacy Arabic terminal BMP mappings is a dead end for now.</p><p></p><div=
class=3D"nh_extra"><blockquote style=3D"padding-top: 12px;" class=3D"nh_qo=
ute"><p style=3D"padding-bottom: 12px;"><strong>Dnia 17 kwietnia 2026 22:59=
</strong> <a href=3D"mailto:[email protected]" rel=3D"noopener noref=
errer nofollow" target=3D"_blank"><span style=3D"margin-left: 4px;">piotrun=
[email protected] via Unicode</span></a><span style=3D"margin-left: 4px;"> <=
[email protected] ></span> napisa=C5=82(a):</p><div id=3D"gwp228=
1a7f8"><div id=3D"gwp2281a7f8h"><div class=3D"gwp2281a7f8b" data-message-bo=
dy=3D"true" data-color-mode=3D"light"><p>The Recommendations in L2/26-100 c=
laim that Microsoft's documentation of legacy Arabic encodings is available=
at https://learn.microsoft.com/en-us/typography/legacy/legacy_arabic_fonts=
. However, that article only demonstrates two encodings of TrueType fonts, =
which are used in Windows 3.1 but are completely different from the eight t=
erminal encodings. Unlike the TrueType encodings which represent internal s=
haping mappings and are not used for text interchange, the terminal encodin=
gs have been demonstrated to be directly used in text interchange through i=
nt 10h and ReadConsoleOutputA/WriteConsoleOutputA as already demonstrated i=
n L2/26-077. The Recommendations also claim that the proposal does not demo=
nstrate any need for interchange or encoding, but the proposal actually dem=
onstrated such a need due to the logical extension of the Win32 terminal AP=
I to the functions ReadConsoleOutputW/WriteConsoleOutputW, which are in Win=
dows NT and may be used on the output of previously ran programs (including=
those that used the legacy Arabic terminal encodings), which given the CHA=
R_INFO structure, therefore implies a need for all the tiles to map to BMP =
for interchange. I'm not objecting to the SEW's conclusion of "Users are ex=
pected to use PUA.", which can indeed be used to provide a mapping even if =
not standardized, but the reasoning given was flawed.</p><p><br></p><div cl=
ass=3D"gwp2281a7f8_nh_extra"><blockquote style=3D"padding-top: 12px;" class=
=3D"gwp2281a7f8_nh_qoute"><p style=3D"padding-bottom: 12px;"><strong>Dnia 0=
9 stycznia 2026 17:25</strong> <span style=3D"margin-left: 4px;">piotrunio-=
[email protected] < [email protected] ></span> napisa=C5=82(a):</p><div i=
d=3D"gwp2281a7f8_gwpa05276c7"><div id=3D"gwp2281a7f8_gwpa05276c7h"><div cla=
ss=3D"gwp2281a7f8_gwpa05276c7b" data-message-body=3D"true" data-color-mode=
=3D"light"><div id=3D"gwp2281a7f8_gwpa05276c7_gwpa8b5f718"><div id=3D"gwp22=
81a7f8_gwpa05276c7_gwpa8b5f718h"><div data-color-mode=3D"light" class=3D"gw=
p2281a7f8_gwpa05276c7_gwpa8b5f718b" data-message-body=3D"true"><div id=3D"g=
wp2281a7f8_gwpa05276c7_gwpa8b5f718_gwpa8b5f718"><div id=3D"gwp2281a7f8_gwpa=
05276c7_gwpa8b5f718_gwpa8b5f718h"><div data-color-mode=3D"light" class=3D"g=
wp2281a7f8_gwpa05276c7_gwpa8b5f718_gwpa8b5f718b" data-message-body=3D"true"=
><div id=3D"gwp2281a7f8_gwpa05276c7_gwpa8b5f718_gwpa8b5f718_gwpa8b5f718"><d=
iv id=3D"gwp2281a7f8_gwpa05276c7_gwpa8b5f718_gwpa8b5f718_gwpa8b5f718h"><div=
data-color-mode=3D"light" class=3D"gwp2281a7f8_gwpa05276c7_gwpa8b5f718_gwp=
a8b5f718_gwpa8b5f718b" data-message-body=3D"true"><p>The following Win32 C =
code will output 256 characters in system console codepage into the charact=
er grid, capture those character tiles in UCS-2 if possible, and then outpu=
t the current console codepage number.<br></p><p><br></p><p>#include <wi=
ndows.h><br>#include <stdio.h><br>int main(){<br>HANDLE hConsole=
=3DGetStdHandle(STD_OUTPUT_HANDLE);<br>CHAR_INFO screen[256];<br>COORD size=
=3D{16,16,};<br>COORD pos=3D{0,0,};<br>SMALL_RECT rect=3D{0,0,15,15,};<br>f=
or(int i=3D0;i<256;i++){<br>screen[i].Attributes=3D0xF0;<br>screen[i].Ch=
ar.AsciiChar=3Di;<br>}<br>WriteConsoleOutputA(hConsole,screen,size,pos,&=
;rect);<br>CHAR_INFO screenu[256];<br>if(ReadConsoleOutputW(hConsole,screen=
u,size,pos,&rect)){<br>for(int i=3D0;i<256;i++) printf("%04X ",scree=
nu[i].Char.UnicodeChar);<br>}<br>else{<br>printf("error %08X\n",GetLastErro=
r());<br>}<br>printf("codepage %u",GetConsoleOutputCP());<br>}<br><br></p><=
p>In most cases, whenever a legacy Win32 codepage is used, the application =
can run on Windows NT to capture the UCS-2 mapping of those character cells=
to the BMP (although for CJK codepages a more complex setup would be neces=
sary due to thousands of fullwidth characters with 2-byte sequences).<br></=
p><p><br></p><p>However, in Arabic versions of Windows 9x (95/98/ME) the re=
sulting character set has many presentation forms that are not in Unicode. =
This is the result when running on Windows ME: <a href=3D"https://i.im=
gur.com/QFm3SkI.png" =3D"" rel=3D"noopener noreferrer" target=3D"_blank">ht=
tps://i.imgur.com/QFm3SkI.png</a> in 10=C3=9720 font, <a href=3D"https=
://i.imgur.com/KUbLQ0A.png" =3D"" rel=3D"noopener noreferrer" target=3D"_bl=
ank">https://i.imgur.com/KUbLQ0A.png</a> in 10=C3=9718 font (same resu=
lt also appears in Windows 95/98). 5=C3=9712, 7=C3=9712, 8=C3=9712, 10=C3=
=9718, 10=C3=9720, and 12=C3=9716 bitmap fonts have been attested with that=
character set (VGAOEM.FON, 8514OEM.FON, DOSAPP.FON). The 10=C3=9720 font h=
as slightly different mapping than the other sizes: 0x93 is =C3=B6 instead =
of =C3=B4, and 0x97 is missing (causing the following characters on the sam=
e line to be drawn at the wrong position). It also claims to be using codep=
age 720, but many characters differ from their CP720 mappings, including th=
e bundled CP_720.NLS mappings (for example, =D9=80 (U+0640 ARABIC TATW=
EEL) is 0x95 in CP720, but in the console 0x95 is =D8=B4 instead, and the t=
atweel is at 0xFF). On Windows 9x, ReadConsoleOutputW is not supported=
so the UCS-2 mappings of the console character tiles cannot be captured (e=
rror 0x00000078 ERROR_CALL_NOT_IMPLEMENTED).<br></p></div></div></div><p><b=
r></p><p>When that program runs on Arabic versions of Windows NT, the visua=
l output is of the CP437 character set if one of the bundled bitmap fonts i=
s used (<a href=3D"https://i.imgur.com/RxjtxMH.png" =3D"" rel=3D"noopener n=
oreferrer" target=3D"_blank">https://i.imgur.com/RxjtxMH.png</a>), or the C=
P720 set if Lucida Console is used, with the Arabic letters either having g=
litchy font substitution (NT 4.0, NT 5.0/2000) or the .notdef glyph (NT 5.1=
/XP and up). In fact, it seems that the only Arabic bitmap fonts that occur=
in Windows NT are CP1256 fonts, which are not used in terminals. So this a=
ppears to be one of those permanent Windows compatibility regressions that =
occured when Windows 9x ended, where the terminals can no longer render leg=
acy Arabic text. Even if the user managed to use registry hacks to set the =
font to Courier New or Simplified Arabic Fixed, it would still use the CP72=
0 mapping which is not compatible with the Windows 9x set.<br></p></div></d=
iv><p><br></p></div></div></div></div><div id=3D"gwp2281a7f8_gwpa05276c7_gw=
pa8b5f718"><div id=3D"gwp2281a7f8_gwpa05276c7_gwpa8b5f718h"><div data-color=
-mode=3D"light" class=3D"gwp2281a7f8_gwpa05276c7_gwpa8b5f718b" data-message=
-body=3D"true"><p>It appears that in the Windows 9x Arabic terminal charact=
er set, 244 characters (=E2=80=87=EF=BA=80=EF=BA=81=EF=BA=82=EF=BA=83=EF=BA=
=84=EF=BA=85=EF=BA=87=EF=BA=88=EF=BA=8A=EF=BA=8B=EF=BA=8D=EF=BA=8E=EF=BA=8F=
=EF=BA=91=EF=BA=93=E2=96=BA=E2=97=84=E2=86=95=EF=BA=95=C2=B6=C2=A7=EF=BA=97=
=EF=BA=99=E2=86=91=E2=86=93=E2=86=92=E2=86=90=EF=BA=9B=EF=B9=B0=E2=96=B2=E2=
=96=BC !"#$%&'()*+,-./0123456789:;<=3D>?@ABCDEFGHIJKLMNOPQRSTUVWX=
YZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~=EF=BA=9D=EF=BA=9F=EF=BA=A1=C3=A9=C3=
=A2=EF=BA=A3=C3=A0=EF=BA=A5=C3=A7=C3=AA=C3=AB=C3=A8=C3=AF=C3=AE=EF=BA=A7=EF=
=BA=A9=EF=BA=AB=EF=BA=AD=EF=BA=AF=C3=B4=EF=BA=B3=C3=BB=C3=B9=EF=BA=B7=EF=BA=
=BB=C2=A3=EF=BA=BF=EF=BB=81=EF=BB=85=EF=BB=89=EF=BB=8A=EF=BB=8B=EF=BB=8C=EF=
=BB=8D=EF=BB=8E=EF=BB=8F=EF=BB=90=EF=BB=91=EF=BB=93=EF=BB=95=EF=BB=97=EF=BB=
=99=EF=BB=9B=C2=AB=C2=BB=EF=B9=B1=E2=96=92=EF=B9=B2=E2=94=82=E2=94=A4=EF=B9=
=B4=EF=B9=B6=EF=B9=B7=EF=B9=B8=D9=A0=D9=A1=D9=A2=D9=A3=EF=B9=B9=EF=B9=BA=E2=
=94=90=E2=94=94=E2=94=B4=E2=94=AC=E2=94=9C=E2=94=80=E2=94=BC=EF=B9=BB=EF=B9=
=BE=D9=A4=D9=A5=D9=A6=D9=A7=D9=A8=D9=A9=D8=8C=EF=B9=BF=EF=B1=9E=EF=B1=9F=EF=
=B1=A0=EF=B3=B2=EF=B1=A1=EF=B3=B3=EF=B1=A2=E2=94=98=E2=94=8C=D8=9B=D8=9F=C2=
=A4=EF=BB=9D=EF=BB=9F=EF=BB=A1=EF=BB=A3=EF=BB=A5=EF=BB=A7=C2=B5=EF=BB=A9=EF=
=BB=AB=EF=BB=AC=EF=BB=AD=EF=BB=AF=EF=BB=B0=EF=BB=B1=EF=BB=B2=EF=BB=B3=EF=B3=
=B4=EF=B9=BC=EF=B9=BD=EF=BA=B1=EF=BA=B5=EF=BA=B9=EF=BA=BD=EF=B9=B3=C2=B0=C2=
=B7=E2=96=A0=D9=80) are already in Unicode, but 12 characters are not in Un=
icode:<br></p><p>=E2=80=A2 6 of them are pieces of lam-alef ligatures (0xDD=
, 0xDE, 0xF9, 0xFB, 0xFC, 0xFD)<br></p><p>=E2=80=A2 2 of them are shadda wi=
th fathatan ligatures without or with tatweel (0xD0, 0xD1)<br></p><p>=E2=80=
=94 in some legacy Microsoft fonts, shadda with fathatan is mapped to priva=
te use U+E818<br></p><p>=E2=80=A2 4 of them are disunifications of seen/she=
en/sad/dad occuring either with or without tail<br></p><p>=E2=80=94 =
=EF=B9=B3 (U+FE73 ARABIC TAIL FRAGMENT) was originally encoded in Unicode 3=
.2 for CP864 compatibility; in that codepage, the forms of seen/sheen/=
sad/dad attach to the tail fragment<br></p><p>=E2=80=94 forms with included=
tail: 0x92, 0x95, 0x98, 0x8A<br></p><p>=E2=80=94 forms without tail (=
attaching to tail fragment like in CP864): 0xF3, 0xF4, 0xF5, 0xF6<br><=
/p></div><p><br></p></div><p>If someone tried to make a Win32 console imple=
mentation and tried to implement both Windows 9x Arabic terminal character =
set compatibility and wide string API (ReadConsoleOutputW) compatibility si=
multaneously, then they would run into the issue that there is currently no=
standardized mapping to handle that scenario. What should Windows 9x Arabi=
c console compatible implementations do in that case?<br></p></div><p><br><=
/p></div></div></div></blockquote></div><p></p></div></div></div></blockquo=
te></div><p></p>
--2OYMHQXURATGUGQGORGOHnhgwp--