Re: Sybase: Charset conversion

Sebastian Podjasek wrote:
> 
> Below are few interesting lines from TDSDUMP
> 
> log.c:190:Starting log file for FreeTDS 0.82
>         on 2009-11-24 17:09:44 with debug flags 0x4fff.
> iconv.c:197:names for ISO-8859-1: ISO-8859-1
> iconv.c:197:names for UTF-8: UTF-8
> iconv.c:197:names for UCS-2LE: UCS-2LE
> iconv.c:197:names for UCS-2BE: UCS-2BE
> iconv.c:363:iconv to convert client-side data to the "UTF-8" character
> set
> iconv.c:516:tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
...
> token.c:316:looking for login token, got  e3(ENVCHANGE)
> token.c:108:tds_process_default_tokens() marker is e3(ENVCHANGE)
> token.c:2356:server indicated charset change to "utf8"
> iconv.c:985:setting server single-byte charset to "UTF-8"
> iconv.c:516:tds_iconv_info_init: converting "ISO-8859-1"->"UTF-8"

Line 516 looks like a problem: the server said it will send UTF-8, and
iconv.c says it will convert "ISO-8859-1" to "UTF-8".  If so, FreeTDS
didn't interpret the message correctly.  

When we look at the data, we find

> 0070 00 00 e1 80 02 00 00 00-d1 08 c5 9c 00 00 60 cf |........ ......`.|
> 0080 c0 00 3e 00 00 00 00 50-00 6f 01 42 01 05 00 63 |..>....P .o.B...c|
> 0090 00 7a 00 65 00 6e 00 69-00 65 00 20 00 7a 00 20 |.z.e.n.i .e. .z. |
> 00a0 00 54 00 43 00 47 00 20-00 6a 00 65 00 73 00 74 |.T.C.G.  .j.e.s.t|
> 00b0 00 20 00 7a 00 61 00 6b-01 42 00 75 00 63 00 6f |. .z.a.k .B.u.c.o|
> 00c0 00 6e 00 65 fd 10 00 01-00 01 00 00 00          |.n.e.... .....|
> 
> token.c:510:processing result tokens.  marker is  61(ROWFMT2)
> token.c:1713:tds5_process_result
> mem.c:563:tds_free_all_results()
> token.c:1737:num_cols=2
> token.c:1840:col 0:
> token.c:1841:   column_name=[dt_made]
> token.c:1847:   flags=20 utype=80 type=111 varint=1
> token.c:1850:   colsize=8 prec=0 scale=0
> token.c:3294:adjust_character_column_size:
>         Server charset: UCS-2LE
>         Server column_size: 640
>         Client charset: UTF-8
>         Client column_size: 1280
> token.c:1840:col 1:
> token.c:1841:   column_name=[c_text]
> token.c:1847:   flags=20 utype=34 type=225 varint=5
> token.c:1850:   colsize=1280 prec=0 scale=0

Problem is, that's neither UTF-8 nor ISO 8859-1.  I think it's utf-16be. 
If so, the data would read something like:

	Po??czenie z TCG jest zak?ucone

(where I have no glyphs for some Polish characters.)  Does that look
right?  

If so, I see two problems:

1.  The server is announcing UTF-8 but the client thinks it's getting ISO
8859-1. 
2.  The server isn't sending UTF-8 but UTF-16.  That would be because the
data are incorrectly encoded on the server.  (The DBA tells the server the
encoding of the data.  When bulk-loading, it's up the to the DBA to ensure
any data added use the column's encoding, because the server can't know.  

HTH.  

--jkl