Re: Status of UTF-8 support in 0.83 dev

James K. Lowden wrote:
> Hi Sebastien, 
> 
>> It appears that a conversion failure occurs in tds_put_data, near line
>> 1477:
>>
>>     /* we need to convert data before */
>>     /* TODO this can be a waste of memory... */
>>     s = tds_convert_string(tds, curcol->char_conv, s, colsize,
>>     &colsize); if (!s) {
>>         /* FIXME this is a bad place to return error... */
>>         /* TODO on memory failure we should compute converted size and
>>         use chunks */ return TDS_FAIL;
>>     }
>>     converted = 1;
>>
>> Occurs when string contains UTF-8 characters encoded with 3 bytes.
>> 2-byte chars pass without problem...
> 
> I marked your message for later investigation.  
> 
> When you say 3-byte characters fail, do you mean that when you prefix the
> string with 'N' in a simple SQL statement, the data are mangled on the
> server?  That would imply a logic error using iconv.  
> 
> If we're talking about unprefixed strings, the problem could be on the
> server (as we both know now).  
> 
> Or, are you talking about prepared statement parameters?

I talk about parameterized queries, using SQLBindParameter() to pass UTF-8
strings... String literals with/without N prefix work fine for me.

See attachments...


> 
> I'm at a disadvantage here because I don't read any ideographic language
> and don't have CJK fonts installed.  To work on this problem -- not
> something I'm spending a lot of time on now in the height of summer -- we
> need to put together a unit test using known code points.  For example, we
> could use a known Japanese string, perhaps some famous phrase or haiku,
> encoded as UTF-8 in a file.  Then the unit test could formulate an SQL
> INSERT statement to write the text to a table, select the row, and write a
> new file.  By comparing input to output, even an illiterate like me could
> know whether or not it worked correctly. Then we could repeat the process
> using a parameterized query, finishing up with an output parameter.  
> 
> If you will work up the unit test, I'll be able to at least help find any
> bugs in the guts of the data transmission logic.  (Remember ODBC isn't my
> game, but I do want parameterized queries to work, and I'm the fellow who
> stuffed iconv into the middle of FreeTDS.) 

Yes I can help such ODBC test, but will be off next 2 weeks...

> 
>> When I print the string of curcol->column_data I get some strange
>> characters at the end of the string... Hence UTF-8 characters should be
>> displayed without prob by using fprintf, as long as LANG=en_US.utf8 ...
>> no?
> 
> There are no null-terminated strings internally, and all character data in
> TDS have a length prefix.  

Yep that's was my conclusion too.

> 
> Regards, 
> 
> --jkl
> _______________________________________________
> FreeTDS mailing list
> [email protected]
> http://lists.ibiblio.org/mailman/listinfo/freetds
>