freebcp UTF8 to UCS-2LE importing

Will McQueen <[email protected]>
Newsgroups gmane.comp.db.tds.freetds
Message-ID <C1174DCAFE2FA74CB0797105D205E7B1010AE99BC3@ars-ex07mail1.corp.athenahealth.com>
Hi, I've been searching the email lists for exactly my problem but haven't quite found it.  Let me start with my problem.

I have a UTF8 formatted data file that I'm importing into a SQL Server 2008 table which has columns typed as nvarchar and the import seems to truncate the data to half the size it should be.  Here are some tdsdump snippets:

log.c:196:Starting log file for FreeTDS 0.91RC2
...
iconv.c:349:setting up conversions for client charset "UTF8"
...
token.c:3423:adjust_character_column_size:
        Server charset: UCS-2LE
        Server column_size: 192
        Client charset: UTF-8
        Client column_size: 384
token.c:1489:tds7_get_data_info:
        colname = CONTEXT_NAME (12 bytes)
        type = 39 (varchar)
        server's type = 231 (x UCS-2 varchar)
        column_varint_size = 2
        column_size = 384 (192 on server)
...
bcp.c:1141:parsing host column 2
bcp.c:1160:host column 2 uses bcpcol 2 (0xbaf3390)
bcp.c:1218:prefix_len = 0 collen = 0
bcp.c:1488:_bcp_measure_terminated_field(0xbaf8a40, 0xbaf8860, 1)
bcp.c:1249:_bcp_measure_terminated_field returned 46
bcp.c:1258:Adjusted collen is 46.
iconv.c:874:tds_iconv_fread: read 46 of 46 bytes; outbuf has 46 left.
iconv.c:891:tds_iconv_fread: error 7: Argument list too long.
bcp.c:1379:Data read from hostfile: collen is now 46, data_is_null is 0
dblib.c:2198:dbconvert(0xbab7460, SYBCHAR, 0xbaee340, 46, SYBVARCHAR, 0xbab7fe0, 384)
...

I think the error in bcp.c is line 1258 where it thinks the adjusted collen should be 46.  I think it should actually be 92, as the 46 bytes read in is variable length UTF8 data.  It could be 46 1-byte characters or it could be 12 multi-byte characters but in the worst case its 46 characters in which case converting it to UCS-2LE will require 92 bytes in the outbuffer.  The error 7 from tds_iconv_fread seems to confirm this, as the E2BIG error is returned from iconv when the outbuffer isn't large enough to hold the resulting string.

Another indication this is the issue is if I switch my client charset to ISO-8859-1.  My data is almost entirely valid ISO-8859-1 data so I can process many rows this way and in that case it properly adjusts the collen to 92.

I'll see if I can work on a patch that can be applied, but I'm curious if anyone else has any feedback for me.

Thanks,
Will
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.