Re: Sending Unicode (UTF-8) data to dbrpcparam - SQL Server receives blank string

Actually git version does what you want, if you look dbrpcparam

1) http://gitorious.org/freetds/freetds/blobs/master/src/dblib/rpc.c#line160
2) http://gitorious.org/freetds/freetds/blobs/Branch-0_91/src/dblib/rpc.c#line164

#1 is the last git version, #2 is the last 0.91, git have some lines

	if (type == SYBVARCHAR && IS_TDS7_PLUS(dbproc->tds_socket->conn))
		type = XSYBNVARCHAR;

that turns type to XSYBNVARCHAR instead of SYBVARCHAR. You should get
same result on 0.91 with

	if (type == SYBVARCHAR && IS_TDS7_PLUS(dbproc->tds_socket))
		type = XSYBNVARCHAR;

Frediano

2013/2/28 Marc Abramowitz <[email protected]>:
> Some more info…
>
> I realized that CP1252 is a single byte encoding that isn't capable of representing Unicode strings and I read
> in the FreeTDS manual that it encodes things to UCS-2 to send to the database and I
> found examples of that in tds/query.c — so I borrowed from some other spots
> in tds/query.c and changed the code to this:
>
> diff --git a/src/tds/query.c b/src/tds/query.c
> index 168c4d5..95e5a4b 100644
> --- a/src/tds/query.c
> +++ b/src/tds/query.c
> @@ -1621,7 +1621,10 @@ tds_put_data(TDSSOCKET * tds, TDSCOLUMN * curcol)
>   /* we need to convert data before */
>   /* TODO this can be a waste of memory... */
>   converted = 1;
> - s = tds_convert_string(tds, curcol->char_conv, s, colsize, &output_size);
> + tdsdump_log(TDS_DBG_INFO1, "tds_put_data: calling tds_convert_string on \"%s\" (strlen %d); colsize = %d; client_charset = \"%s\"; server_charset = \"%s\"\n",
> +    s, (int) strlen(s), (int) colsize, tds->char_convs[client2ucs2]->client_charset.name, tds->char_convs[client2ucs2]->server_charset.name);
> + s = tds_convert_string(tds, tds->char_convs[client2ucs2], s, colsize, &output_size);
> + tdsdump_log(TDS_DBG_INFO1, "tds_put_data: result of tds_convert_string => \"%s\"\n", s);
>   colsize = (TDS_INT)output_size;
>   if (!s) {
>   /* on conversion error put a empty string */
>
> And that seems to help although now I get back a result that is encoded in UCS-2
> (not sure if that is a FreeTDS bug or a pymssql bug).
>
> Freetds log:
>
> query.c:1437:tds_put_data_info putting param_name.
> query.c:1471:tds_put_data_info putting status.
> query.c:1572:tds_put_data: colsize = 20
> query.c:1629:tds_put_data: calling tds_convert_string on "Здравствуй^A" (strlen 21); colsize = 20; client_charset = "UTF-8"; server_charset = "UCS-2LE"
> query.c:1632:tds_put_data: result of tds_convert_string => "^W^D4^D@^D0^D2^DA^DB^D2^DC^D9^Dt"
> query.c:1648:tds_put_data: not null param varint_size = 2
> util.c:156:Changed query state from QUERYING to PENDING
> net.c:741:Sending packet
> 0000 03 01 00 54 00 00 01 00-14 00 73 00 6f 00 6d 00 |...T.... ..s.o.m.|
> 0010 65 00 50 00 72 00 6f 00-63 00 57 00 69 00 74 00 |e.P.r.o. c.W.i.t.|
> 0020 68 00 4f 00 6e 00 65 00-50 00 61 00 72 00 61 00 |h.O.n.e. P.a.r.a.|
> 0030 6d 00 00 00 00 00 a7 14-00 09 04 d0 00 34 14 00 |m....... .....4..|
> 0040 17 04 34 04 40 04 30 04-32 04 41 04 42 04 32 04 |[email protected]. 2.A.B.2.|
> 0050 43 04 39 04            -                        |C.9.|
>
> Result:
>
> AssertionError: u'\x17\x044\x04@\x040\x042\x04A\x04B\x042\x04C\x049\x04!' != u'\u0417\u0434\u0440\u0430\u0432\u0441\u0442\u0432\u0443\u0439!'
>
> I am expecting to get back the Unicode string on the right, but what I get back is the UTF-16 string on the left.
>
> From: Marc Abramowitz <[email protected]<mailto:[email protected]>>
> Date: Wednesday, February 27, 2013 2:58 PM
> To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>
> Subject: Re: Sending Unicode (UTF-8) data to dbrpcparam - SQL Server receives blank string
>
> A little more info…
>
> If I add the following:
>
> diff --git a/src/tds/query.c b/src/tds/query.c
> index 168c4d5..17f227c 100644
> --- a/src/tds/query.c
> +++ b/src/tds/query.c
> @@ -1621,7 +1621,10 @@ tds_put_data(TDSSOCKET * tds, TDSCOLUMN * curcol)
>                 /* we need to convert data before */
>                 /* TODO this can be a waste of memory... */
>                 converted = 1;
> +               tdsdump_log(TDS_DBG_INFO1, "tds_put_data: calling tds_convert_string on \"%s\"; colsize = %d; client_charset = \"%s\"; server_charset = \"%s\"\n",
> +                           s, colsize, curcol->char_conv->client_charset.name, curcol->char_conv->server_charset.name);
>                 s = tds_convert_string(tds, curcol->char_conv, s, colsize, &output_size);
> +               tdsdump_log(TDS_DBG_INFO1, "tds_put_data: result of tds_convert_string => \"%s\"\n", s);
>                 colsize = (TDS_INT)output_size;
>                 if (!s) {
>                         /* on conversion error put a empty string */
>
> Here's what I get:
>
> query.c:1624:tds_put_data: calling tds_convert_string on "Здравствуй^A"; colsize = 20; client_charset = "UTF-8"; server_charset = "CP1252"
> util.c:331:tdserror(0x101506eb0, 0x10159fdb0, 2402, 0)
> dblib.c:7929:dbperror(0x10159fac0, 2402, 0)
> dblib.c:7981:2402: "Error converting characters into server's character set. Some character(s) could not be converted"
> dblib.c:8002:"Error converting characters into server's character set. Some character(s) could not be converted", client returns 2 (INT_CANCEL)
> util.c:361:tdserror: client library returned TDS_INT_CANCEL(2)
> util.c:384:tdserror: returning TDS_INT_CANCEL(2)
> query.c:1627:tds_put_data: result of tds_convert_string => "(null)"
>
> Should I be suspicious of the "^A" (null byte?) tacked on to the end of my string? Maybe that 's why the conversion is failing?
>
> _______________________________________________
> FreeTDS mailing list
> [email protected]
> http://lists.ibiblio.org/mailman/listinfo/freetds
_______________________________________________
FreeTDS mailing list
[email protected]
http://lists.ibiblio.org/mailman/listinfo/freetds