Re: Status of UTF-8 support in 0.83 dev

It appears that a conversion failure occurs in tds_put_data, near line 1477:

    /* we need to convert data before */
    /* TODO this can be a waste of memory... */
    s = tds_convert_string(tds, curcol->char_conv, s, colsize, &colsize);
    if (!s) {
        /* FIXME this is a bad place to return error... */
        /* TODO on memory failure we should compute converted size and use chunks */
        return TDS_FAIL;
    }
    converted = 1;

Occurs when string contains UTF-8 characters encoded with 3 bytes.
2-byte chars pass without problem...

When I print the string of curcol->column_data I get some strange characters at
the end of the string... Hence UTF-8 characters should be displayed without prob
by using fprintf, as long as LANG=en_US.utf8 ... no?

I will double check the UTF-8 string I put in my buffers, as well as the size
passed to SQLBindParameter, but when I print the string in the ODBC program,
it displays without additional characters at the end...

In my ODBC program:

    UTF-8 = [abcéô鴻]  ( size= 10 )

In tds.c code:

    UTF-8 = [abcéô鴻01000]  ( size= 10 )

...

Seb

Sebastien FLAESCH wrote:
> Thank you James for looking at this!
> 
> UTF-8 support is critical for one of of customers in Asia...
> 
> I have identified a first issue: I must of course use N'xxx' string
> constants when inserting UNICODE/UTF-8 data in the database.
> In previous sample program, the string constants where identified as
> non-Unicode strings in the current database locale. Sorry...
> 
> I have added a SELECT to fetch inserted rows, and it seems to be fine
> so far. I have checked the values in the database with SQL Server's
> query tool and the characters are correct.
> 
> Still the insert statement using UTF-8 data fails with:
> 
> Error -1 at: SQLExecute 1.2 StmtH
> Diagnostic info:
>   SQL State: 42000
>   SQL code : 8005
>   Message  : [FreeTDS][SQL Server]The incoming tabular data stream
>              (TDS) remote procedure call (RPC) protocol stream is
>              incorrect. Parameter 4: The parameter name is invalid.
> 
> 
> Note that the first insert with latin characters "abcéÓÎ" / "abcÛÊé"
> (2 bytes long) works, it's the second insert using Asian characters
> (3 bytes long)...
> 
> 
> FYI I have rebuild freetds with iconv 1.12 now:
> 
> [root@fox ~]# ldd -r /opt3/dbs/tds/0.83/lib/libtdsodbc.so
>         libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x00111000)
>         librt.so.1 => /lib/tls/librt.so.1 (0x00a7b000)
>         libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00957000)
>         libc.so.6 => /lib/tls/libc.so.6 (0x001ed000)
>         /lib/ld-linux.so.2 (0x00739000)
> 
> Thanks,
> Seb
> 
> 
> James K. Lowden wrote:
>> Sebastien FLAESCH wrote:
>>> Log shows now:
>>>
>>> log.c:190:Starting log file for FreeTDS 0.83.dev.20080812
>>>          on 2008-08-12 18:11:48 with debug flags 0x4fff.
>>> iconv.c:197:names for ISO-8859-1: ISO-8859-1
>>> iconv.c:197:names for UTF-8: UTF-8
>>> iconv.c:197:names for UCS-2LE: UCS-2LE
>>> iconv.c:197:names for UCS-2BE: UCS-2BE
>>> iconv.c:363:iconv to convert client-side data to the "UTF-8" character
>>> set iconv.c:516:tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
>>> iconv.c:516:tds_iconv_info_init: converting "ISO-8859-1"->"UCS-2LE"
>>
>> Hi Sebastien,
>> I modified the log messages from iconv.c to make them clearer.  HTH.
>>> But still, I get:
>>>
>>>
>>> Error -1 at: SQLExecute 1.2 StmtH
>>> Diagnostic info:
>>>    SQL State: 42000
>>>    SQL code : 8005
>>>    Message  : [FreeTDS][SQL Server]The incoming tabular data stream
>>>    (TDS) remote procedure call (RPC) protocol stream is incorrect.
>>>    Parameter 4: The parameter name is invalid.
>>
>> This happens because of these errors in the log:
>>
>>> query.c:1326:tds_put_data_info putting status 
>>> query.c:1420:tds_put_data: colsize = 9
>>> query.c:1449:tds_put_data: not null param varint_size = 2
>>> util.c:334:tdserror(0x98cb050, 0x98cc3d8, 2402, 0)
>>> odbc.c:2106:msgno 2402 20003
>>> util.c:368:tdserror: client library returned TDS_INT_CANCEL(2)
>>> util.c:389:tdserror: returning TDS_INT_CANCEL(2)
>>> query.c:1326:tds_put_data_info putting status 
>>> query.c:1420:tds_put_data: colsize = 9
>>> query.c:1449:tds_put_data: not null param varint_size = 2
>>> util.c:334:tdserror(0x98cb050, 0x98cc3d8, 2402, 0)
>>> odbc.c:2106:msgno 2402 20003
>>> util.c:368:tdserror: client library returned TDS_INT_CANCEL(2)
>>> util.c:389:tdserror: returning TDS_INT_CANCEL(2)
>>> util.c:162:Changed query state from QUERYING to PENDING
>>> net.c:768:Sending packet
>>
>> The resulting sp_execute call is incorrectly formed, and the server
>> complains. 
>> 2402 is TDSEICONVO and indicates an invalid multibyte input sequence was
>> encountered while sending data to the server. 
>> I'm too tired to debug it further.  I would guess the parameter data are
>> assumed to be in ISO-8859-1, not in the client's encoding. 
>> Regards,
>> --jkl
>> _______________________________________________
>> FreeTDS mailing list
>> [email protected]
>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> FreeTDS mailing list
> [email protected]
> http://lists.ibiblio.org/mailman/listinfo/freetds

_______________________________________________
FreeTDS mailing list
[email protected]
http://lists.ibiblio.org/mailman/listinfo/freetds