Re: TextProperty to TextList conversion problem on euckr locale

ksc5601.h needs to be updated to be consistent with iconv. otherwise,
charset_wctocs() can't find appropriate charset for those characters in
EUC-KR locale.

-federic

Choe Hwanjin wrote:
> I make utf8 text file for test. It contains utf8-encoded
> ksx1001(ksc5601) chars.  When I convert it to EUC-KR with iconv, no
> conversion error occur.
> 
> On X environment, I made TextProperty from Xutf8TextListToTextProperty
> and I converted the generated text property to the TextList by calling
> XmbTextPropertyToTextList. On UTF-8 locale, there was no error.  But on
> ko_KR.EUC-KR locale, some errors occured. Almost chars converted
> successfully, but below chars did not:
> 
> Unconvertable: U+00B7 [·(2)]
> Unconvertable: U+00A8 [¨(2)]
> Unconvertable: U+00AD [­(2)]
> Unconvertable: U+00B1 [±(2)]
> Unconvertable: U+00D7 [×(2)]
> Unconvertable: U+00F7 [÷(2)]
> Unconvertable: U+00B0 [°(2)]
> Unconvertable: U+00A7 [§(2)]
> Unconvertable: U+00B4 [´(2)]
> Unconvertable: U+00B8 [¸(2)]
> Unconvertable: U+00A1 [¡(2)]
> Unconvertable: U+00BF [¿(2)]
> Unconvertable: U+00A4 [¤(2)]
> Unconvertable: U+00B6 [¶(2)]
> Unconvertable: U+20AC [€(3)]
> Unconvertable: U+00AE [®(2)]
> Unconvertable: U+00C6 [Æ(2)]
> Unconvertable: U+00D0 [Ð(2)]
> Unconvertable: U+00AA [ª(2)]
> Unconvertable: U+00D8 [Ø(2)]
> Unconvertable: U+00BA [º(2)]
> Unconvertable: U+00DE [Þ(2)]
> Unconvertable: U+00BD [½(2)]
> Unconvertable: U+00BC [¼(2)]
> Unconvertable: U+00BE [¾(2)]
> Unconvertable: U+00E6 [æ(2)]
> Unconvertable: U+00F0 [ð(2)]
> Unconvertable: U+00F8 [ø(2)]
> Unconvertable: U+00DF [ß(2)]
> Unconvertable: U+00FE [þ(2)]
> Unconvertable: U+00B9 [¹(2)]
> Unconvertable: U+00B2 [²(2)]
> Unconvertable: U+00B3 [³(2)]
> 
> I think this is not desirable result.
> Above chars are define in ksx1001, so iconv converts them correctly.
> But on X, it fails. Is it right behavior?
> 
> So I made small test case code and attached those files.
> You can reproduce this problem by following this instructions:
> 
> $ gmake
> $ LANG=ko_KR.UTF-8 ./conversion < utf8.txt  # no error
> $ LANG=ko_KR.euckr ./conversion < utf8.txt  # error
> $ iconv -f utf8 -t euckr utf8.txt > /dev/null # no error
> 
> PS.
> I'm not a member of this mailing list, so please cc me.
> 
> 
> 
> ------------------------------------------------------------------------
> 
> CFLAGS = -I/usr/X11R6/include $(shell pkg-config --cflags glib-2.0)
> LIBS = -L/usr/X11R6/lib -lX11 $(shell pkg-config --libs glib-2.0)
> 
> all:
> 	gcc -Wall -ggdb $(CFLAGS) $(LIBS) -o conversion conversion.c 
> 
> 
> ------------------------------------------------------------------------
> 
> 
> #include <stdio.h>
> #include <string.h>
> #include <locale.h>
> #include <X11/Xlib.h>
> #include <X11/Xutil.h>
> #include <glib.h>
> 
> static Display *display;
> 
> void textpropertytest(char *utf8)
> {
>     int n, ret;
>     XTextProperty text_prop;
>     char *list[2];
>     char **return_list = NULL;
> 
>     list[0] = utf8;
>     list[1] = 0;
>     n = 1;
>     ret = Xutf8TextListToTextProperty(display, list, n,
> 				      XCompoundTextStyle,
> 				      &text_prop);
>     if (ret != Success) {
> 	fprintf(stderr, "Error on utf8 -> TextProperty (%d)\n", ret);
>     } else {
> 	ret = XmbTextPropertyToTextList(display, &text_prop, &return_list, &n);
> 	if (ret != Success) {
> 	    printf("Unconvertable: U+%04X [%s(%d)]\n",
> 		    g_utf8_get_char(utf8), utf8, strlen(utf8));
> 	}
> 	XFreeStringList(return_list);
>     }
> }
> 
> int
> main(void)
> {
>     char *locale_name;
>     char buf[256];
>     char utf8char[10];
>     const gchar *end;
>     gchar *ptr;
>     gchar *strend;
> 
>     locale_name = setlocale(LC_ALL, "");
>     fprintf(stderr, "Locale: %s\n", locale_name);
> 
>     display = XOpenDisplay(NULL);
>     if (display == NULL) {
> 	fprintf(stderr, "Error on XOpenDisplay\n");
> 	return 1;
>     }
> 
>     while (!feof(stdin)) {
> 	fgets(buf, sizeof(buf), stdin);
> 	if (g_utf8_validate(buf, -1, &end)) {
> 	    ptr = buf;
> 	    strend = strchr(buf, '\0');
> 	    while (ptr < strend) {
> 		memset(utf8char, 0, sizeof(utf8char));
> 		g_utf8_strncpy(utf8char, ptr, 1);
> 		if (strlen(utf8char) > 0)
> 		    textpropertytest(utf8char);
> 		ptr = g_utf8_find_next_char(ptr, NULL);
> 	    }
> 	} else {
> 	    fprintf(stderr, "Invalid utf8 string: %s\n", end);
> 	}
>     }
> 
>     XCloseDisplay(display);
> 
>     return 0;
> }
> 
> 
> ------------------------------------------------------------------------
> 
> A1A0:??·..…¨?­--?\?‘’
> A1B0:“”????????????±×
> A1C0:÷?????°?????????
> A1D0:???????§????????
> A1E0:????????????????
> A1F0:???????????????
> A2A0:????´~?????¸?¡¿
> A2B0:????¤?‰?????????
> A2C0:????????????????
> A2D0:??¶†‡???????????
> A2E0:No?™??TEL€®
> A2F0:
> A3A0:!"#$%&'()*+,-./
> A3B0:0123456789:;<=>?
> A3C0:@ABCDEFGHIJKLMNO
> A3D0:PQRSTUVWXYZ[?]^_
> A3E0:`abcdefghijklmno
> A3F0:pqrstuvwxyz{|}?
> A4A0:???????????????
> A4B0:????????????????
> A4C0:????????????????
> A4D0:????????????????
> A4E0:????????????????
> A4F0:???????????????
> A5A0:iiiiiiivvviviiviiiixx
> A5B0:IIIIIIIVVVIVIIVIIIIXX
> A5C0:???????????????
> A5D0:?????????
> A5E0:???????????????
> A5F0:?????????
> A6A0:???????????????
> A6B0:????????????????
> A6C0:????????????????
> A6D0:????????????????
> A6E0:?????
> A6F0:
> A7A0:???l???????????
> A7B0:????????????????
> A7C0:????????????????
> A7D0:????????????????
> A7E0:????????????????
> A7F0:
> A8A0:ÆЪ?IJL.?،ºÞ??
> A8B0:???????????????
> A8C0:?????????????(a)(b)(c)
> A8D0:(d)(e)(f)(g)(h)(i)(j)(k)(l)(m)(n)(o)(p)(q)(r)(s)
> A8E0:(t)(u)(v)(w)(x)(y)(z)(1)(2)(3)(4)(5)(6)(7)(8)(9)
> A8F0:(10)(11)(12)(13)(14)(15)½1/32/3¼¾1/83/85/87/8
> A9A0:æ?ð??ij?l.?øœßþ??
> A9B0:n???????????????
> A9C0:?????????????(a)(b)(c)
> A9D0:(d)(e)(f)(g)(h)(i)(j)(k)(l)(m)(n)(o)(p)(q)(r)(s)
> A9E0:(t)(u)(v)(w)(x)(y)(z)(1)(2)(3)(4)(5)(6)(7)(8)(9)
> A9F0:(10)(11)(12)(13)(14)(15)¹²³^(4)^(n)v(1)v(2)v(3)v(4)
> AAA0:???????????????
> AAB0:????????????????
> AAC0:????????????????
> AAD0:????????????????
> AAE0:????????????????
> AAF0:????
> ABA0:???????????????
> ABB0:????????????????
> ABC0:????????????????
> ABD0:????????????????
> ABE0:????????????????
> ABF0:???????
> ACA0:???????????????
> ACB0:????????????????
> ACC0:??
> ACD0:???????????????
> ACE0:????????????????
> ACF0:??
> ADA0:
> ADB0:
> ADC0:
> ADD0:
> ADE0:
> ADF0:
> AEA0:
> AEB0:
> AEC0:
> AED0:
> AEE0:
> AEF0:
> AFA0:
> AFB0:
> AFC0:
> AFD0:
> AFE0:
> AFF0:
> B0A0:???????????????
> B0B0:????????????????
> B0C0:????????????????
> B0D0:????????????????
> B0E0:????????????????
> B0F0:???????????????
> B1A0:???????????????
> B1B0:????????????????
> B1C0:????????????????
> B1D0:????????????????
> B1E0:????????????????
> B1F0:???????????????
> B2A0:???????????????
> B2B0:????????????????
> B2C0:????????????????
> B2D0:????????????????
> B2E0:????????????????
> B2F0:???????????????
> B3A0:???????????????
> B3B0:????????????????
> B3C0:????????????????
> B3D0:????????????????
> B3E0:????????????????
> B3F0:???????????????
> B4A0:???????????????
> B4B0:????????????????
> B4C0:????????????????
> B4D0:????????????????
> B4E0:????????????????
> B4F0:???????????????
> B5A0:???????????????
> B5B0:????????????????
> B5C0:????????????????
> B5D0:????????????????
> B5E0:????????????????
> B5F0:???????????????
> B6A0:???????????????
> B6B0:????????????????
> B6C0:????????????????
> B6D0:????????????????
> B6E0:????????????????
> B6F0:???????????????
> B7A0:???????????????
> B7B0:????????????????
> B7C0:????????????????
> B7D0:????????????????
> B7E0:????????????????
> B7F0:???????????????
> B8A0:???????????????
> B8B0:????????????????
> B8C0:????????????????
> B8D0:????????????????
> B8E0:????????????????
> B8F0:???????????????
> B9A0:???????????????
> B9B0:????????????????
> B9C0:????????????????
> B9D0:????????????????
> B9E0:????????????????
> B9F0:???????????????
> BAA0:???????????????
> BAB0:????????????????
> BAC0:????????????????
> BAD0:????????????????
> BAE0:????????????????
> BAF0:???????????????
> BBA0:???????????????
> BBB0:????????????????
> BBC0:????????????????
> BBD0:????????????????
> BBE0:????????????????
> BBF0:???????????????
> BCA0:???????????????
> BCB0:????????????????
> BCC0:????????????????
> BCD0:????????????????
> BCE0:????????????????
> BCF0:???????????????
> BDA0:???????????????
> BDB0:????????????????
> BDC0:????????????????
> BDD0:????????????????
> BDE0:????????????????
> BDF0:???????????????
> BEA0:???????????????
> BEB0:????????????????
> BEC0:????????????????
> BED0:????????????????
> BEE0:????????????????
> BEF0:???????????????
> BFA0:???????????????
> BFB0:????????????????
> BFC0:????????????????
> BFD0:????????????????
> BFE0:????????????????
> BFF0:???????????????
> C0A0:???????????????
> C0B0:????????????????
> C0C0:????????????????
> C0D0:????????????????
> C0E0:????????????????
> C0F0:???????????????
> C1A0:???????????????
> C1B0:????????????????
> C1C0:????????????????
> C1D0:????????????????
> C1E0:????????????????
> C1F0:???????????????
> C2A0:???????????????
> C2B0:????????????????
> C2C0:????????????????
> C2D0:????????????????
> C2E0:????????????????
> C2F0:???????????????
> C3A0:???????????????
> C3B0:????????????????
> C3C0:????????????????
> C3D0:????????????????
> C3E0:????????????????
> C3F0:???????????????
> C4A0:???????????????
> C4B0:????????????????
> C4C0:????????????????
> C4D0:????????????????
> C4E0:????????????????
> C4F0:???????????????
> C5A0:???????????????
> C5B0:????????????????
> C5C0:????????????????
> C5D0:????????????????
> C5E0:????????????????
> C5F0:???????????????
> C6A0:???????????????
> C6B0:????????????????
> C6C0:????????????????
> C6D0:????????????????
> C6E0:????????????????
> C6F0:???????????????
> C7A0:???????????????
> C7B0:????????????????
> C7C0:????????????????
> C7D0:????????????????
> C7E0:????????????????
> C7F0:???????????????
> C8A0:???????????????
> C8B0:????????????????
> C8C0:????????????????
> C8D0:????????????????
> C8E0:????????????????
> C8F0:???????????????
> C9A0:??I????????
> C9B0:???????????????
> C9C0:??????????????
> C9D0:???????????????
> C9E0:???????????????
> C9F0:??????????????
> CAA0:???????????????
> CAB0:????????????????
> CAC0:????????????????
> CAD0:????????????????
> CAE0:????????????????
> CAF0:???????????????
> CBA0:???????????????
> CBB0:????????????????
> CBC0:????????????????
> CBD0:????????????????
> CBE0:????????????????
> CBF0:???????????????
> CCA0:???????????????
> CCB0:????????????????
> CCC0:????????????????
> CCD0:????????????????
> CCE0:????????????????
> CCF0:???????????????
> CDA0:???????????????
> CDB0:????????????????
> CDC0:????????????????
> CDD0:????????????????
> CDE0:????????????????
> CDF0:???????????????
> CEA0:???????????????
> CEB0:????????????????
> CEC0:????????????????
> CED0:????????????????
> CEE0:????????????????
> CEF0:???????????????
> CFA0:???????????????
> CFB0:????????????????
> CFC0:????????????????
> CFD0:????????????????
> CFE0:????????????????
> CFF0:???????????????
> D0A0:???????????????
> D0B0:????????????????
> D0C0:????????????????
> D0D0:????????????????
> D0E0:????????????????
> D0F0:???????????????
> D1A0:???????????????
> D1B0:????????????????
> D1C0:????????????????
> D1D0:????????????????
> D1E0:????????????????
> D1F0:???????????????
> D2A0:???????????????
> D2B0:????????????????
> D2C0:????????????????
> D2D0:????????????????
> D2E0:????????????????
> D2F0:???????????????
> D3A0:???????????????
> D3B0:????????????????
> D3C0:????????????????
> D3D0:????????????????
> D3E0:????????????????
> D3F0:???????????????
> D4A0:???????????????
> D4B0:????????????????
> D4C0:????????????????
> D4D0:????????????????
> D4E0:????????????????
> D4F0:???????????????
> D5A0:???????????????
> D5B0:????????????????
> D5C0:????????????????
> D5D0:????????????????
> D5E0:????????????????
> D5F0:???????????????
> D6A0:???????????????
> D6B0:????????????????
> D6C0:????????????????
> D6D0:????????????????
> D6E0:????????????????
> D6F0:???????????????
> D7A0:???????????????
> D7B0:????????????????
> D7C0:????????????????
> D7D0:????????????????
> D7E0:????????????????
> D7F0:???????????????
> D8A0:???????????????
> D8B0:????????????????
> D8C0:????????????????
> D8D0:????????????????
> D8E0:????????????????
> D8F0:???????????????
> D9A0:???????????????
> D9B0:????????????????
> D9C0:????????????????
> D9D0:????????????????
> D9E0:????????????????
> D9F0:???????????????
> DAA0:???????????????
> DAB0:????????????????
> DAC0:????????????????
> DAD0:????????????????
> DAE0:????????????????
> DAF0:???????????????
> DBA0:???????????????
> DBB0:????????????????
> DBC0:????????????????
> DBD0:????????????????
> DBE0:????????????????
> DBF0:???????????????
> DCA0:???????????????
> DCB0:????????????????
> DCC0:????????????????
> DCD0:????????????????
> DCE0:????????????????
> DCF0:???????????????
> DDA0:???????????????
> DDB0:????????????????
> DDC0:????????????????
> DDD0:????????????????
> DDE0:????????????????
> DDF0:???????????????
> DEA0:???????????????
> DEB0:????????????????
> DEC0:????????????????
> DED0:????????????????
> DEE0:????????????????
> DEF0:???????????????
> DFA0:???????????????
> DFB0:????????????????
> DFC0:????????????????
> DFD0:????????????????
> DFE0:????????????????
> DFF0:???????????????
> E0A0:???????????????
> E0B0:????????????????
> E0C0:????????????????
> E0D0:????????????????
> E0E0:????????????????
> E0F0:???????????????
> E1A0:???????????????
> E1B0:????????????????
> E1C0:????????????????
> E1D0:????????????????
> E1E0:????????????????
> E1F0:???????????????
> E2A0:???????????????
> E2B0:????????????????
> E2C0:????????????????
> E2D0:????????????????
> E2E0:????????????????
> E2F0:???????????????
> E3A0:???????????????
> E3B0:????????????????
> E3C0:????????????????
> E3D0:????????????????
> E3E0:????????????????
> E3F0:???????????????
> E4A0:???????????????
> E4B0:????????????????
> E4C0:????????????????
> E4D0:????????????????
> E4E0:????????????????
> E4F0:???????????????
> E5A0:???????????????
> E5B0:????????????????
> E5C0:????????????????
> E5D0:????????????????
> E5E0:????????????????
> E5F0:???????????????
> E6A0:???????????????
> E6B0:????????????????
> E6C0:????????????????
> E6D0:????????????????
> E6E0:????????????????
> E6F0:???????????????
> E7A0:???????????????
> E7B0:????????????????
> E7C0:????????????????
> E7D0:????????????????
> E7E0:????????????????
> E7F0:???????????????
> E8A0:???????????????
> E8B0:????????????????
> E8C0:????????????????
> E8D0:????????????????
> E8E0:????????????????
> E8F0:???????????????
> E9A0:???????????????
> E9B0:????????????????
> E9C0:????????????????
> E9D0:????????????????
> E9E0:????????????????
> E9F0:???????????????
> EAA0:???????????????
> EAB0:????????????????
> EAC0:????????????????
> EAD0:????????????????
> EAE0:????????????????
> EAF0:???????????????
> EBA0:???????????????
> EBB0:????????????????
> EBC0:????????????????
> EBD0:????????????????
> EBE0:????????????????
> EBF0:???????????????
> ECA0:???????????????
> ECB0:????????????????
> ECC0:????????????????
> ECD0:????????????????
> ECE0:????????????????
> ECF0:???????????????
> EDA0:???????????????
> EDB0:????????????????
> EDC0:????????????????
> EDD0:????????????????
> EDE0:????????????????
> EDF0:???????????????
> EEA0:???????????????
> EEB0:????????????????
> EEC0:????????????????
> EED0:????????????????
> EEE0:????????????????
> EEF0:???????????????
> EFA0:???????????????
> EFB0:????????????????
> EFC0:????????????????
> EFD0:????????????????
> EFE0:????????????????
> EFF0:???????????????
> F0A0:???????????????
> F0B0:????????????????
> F0C0:????????????????
> F0D0:????????????????
> F0E0:????????????????
> F0F0:???????????????
> F1A0:???????????????
> F1B0:????????????????
> F1C0:????????????????
> F1D0:????????????????
> F1E0:????????????????
> F1F0:???????????????
> F2A0:???????????????
> F2B0:????????????????
> F2C0:????????????????
> F2D0:????????????????
> F2E0:????????????????
> F2F0:???????????????
> F3A0:???????????????
> F3B0:????????????????
> F3C0:????????????????
> F3D0:????????????????
> F3E0:????????????????
> F3F0:???????????????
> F4A0:???????????????
> F4B0:????????????????
> F4C0:????????????????
> F4D0:????????????????
> F4E0:????????????????
> F4F0:???????????????
> F5A0:???????????????
> F5B0:????????????????
> F5C0:????????????????
> F5D0:????????????????
> F5E0:????????????????
> F5F0:???????????????
> F6A0:???????????????
> F6B0:????????????????
> F6C0:????????????????
> F6D0:????????????????
> F6E0:????????????????
> F6F0:???????????????
> F7A0:???????????????
> F7B0:????????????????
> F7C0:????????????????
> F7D0:????????????????
> F7E0:????????????????
> F7F0:???????????????
> F8A0:???????????????
> F8B0:????????????????
> F8C0:????????????????
> F8D0:????????????????
> F8E0:????????????????
> F8F0:???????????????
> F9A0:???????????????
> F9B0:????????????????
> F9C0:????????????????
> F9D0:????????????????
> F9E0:????????????????
> F9F0:???????????????
> FAA0:???????????????
> FAB0:????????????????
> FAC0:????????????????
> FAD0:????????????????
> FAE0:????????????????
> FAF0:???????????????
> FBA0:???????????????
> FBB0:????????????????
> FBC0:????????????????
> FBD0:????????????????
> FBE0:????????????????
> FBF0:???????????????
> FCA0:???????????????
> FCB0:????????????????
> FCC0:????????????????
> FCD0:????????????????
> FCE0:????????????????
> FCF0:???????????????
> FDA0:???????????????
> FDB0:????????????????
> FDC0:????????????????
> FDD0:????????????????
> FDE0:????????????????
> FDF0:???????????????
> FEA0:
> FEB0:
> FEC0:
> FED0:
> FEE0:
> FEF0:


--