Re: GuessUnicodeCharset

On Mon, 27 Sep 2004 00:59:43 +0300 (EEST) Nerijus Baliunas <[email protected]> wrote:

NB> GuessUnicodeCharset() is not working sometimes with Lithuanian
NB> texts, when the first non ASCII character (ð, þ for example) is in
NB> ISO-8859-1 or -2 (in addition to -4 or -13, i.e. Baltic encodings).
NB> Then ISO-8859-1 or -2 is chosen for conversion from UTF-8 and
NB> the text is garbled, as it usually has characters which are not in -1
NB> or -2.

 I did think of a possibility of something like this happening but wanted
to wait for a first real-life case of such thing happening before thinking
about it. Well, I didn't have to wait for a long time ;-)

NB> I thought of 2 solutions:
NB> * find all non ASCII characters instead of the first only (or at least
NB> 3-5), and analyze all of them.

 Yes, this is what I thought of as well. Definitely not all but maybe
the first 10-20. Or, if we want to be really meticulous, look at some
characters in the beginning, some in the middle and some in the end.

 It is not difficult to do, but I probably won't have time to do it this
week, so if you can do it, it would be great.

 TIA,
VZ



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php