Re: SQLGetData() returns TEXT length minus one on Solaris 64b with SQL Server 2005
Frediano Ziglio <[email protected]>
| Newsgroups | gmane.comp.db.tds.freetds |
|---|---|
| Message-ID | <CAHt6W4ea0vx+6ZdFUvvjNvYOH6tA8cRfXowWs6m=EiqbxiVrLw@mail.gmail.com> |
2013/2/6 Sebastien FLAESCH <[email protected]>: > FYI: > > We have tested with this configure option: > > --disable-libiconv > > and now it works... > > Whatever iconv issue it is, I believe that FreeTDS should check iconv status > and stop with a fatal error if iconv is not able to handle charset conversions. > > Character set conversion is critical, when subject of data loss or corruption > with no special warning for the user. > > Seb > Mmm... not that good. It means that Solaris iconv does something that is not expected by FreeTDS. Probably FreeTDS expect a 1 to 1 conversion while lib iconv provided by Solaris does not respect this. I should write a test if libiconv linked in is FreeTDS-aware. Probably using GNU libiconv would fix your issues. However is hard to reproduce using a libiconv that works as expected. Frediano > On 02/05/2013 03:22 PM, Sebastien FLAESCH wrote: >> It appears that compiling with libiconv does not help. >> Seb >> >> >> On 02/05/2013 12:46 PM, Sebastien FLAESCH wrote: >>> On 02/04/2013 06:57 PM, Frediano Ziglio wrote: >>>> 2013/2/4 Sebastien FLAESCH<[email protected]>: >>>>> Frediano, >>>>> >>>>> I believe it's related to the client charset conversions... >>>>> >>>>> I did following simple test: >>>>> >>>>> My database collation is CP1252, so to avoid iconv usage, in the ODBC >>>>> data source definition, I have now: >>>>> >>>>> ClientCharset = CP1252 >>>>> >>>>> The program uses only pure ASCII chars, so this is ok. >>>>> >>>>> Now there is no more iconv error in the log, and the first call to >>>>> SQLGetData() returns 10000 bytes as expected... >>>>> >>>>> If fact for character data, ODBC returns the char pieces with an >>>>> additional \0 terminator. >>>>> >>>>> So, with a buffer of 4096 bytes: >>>>> >>>>> Call #1 to SQLGetData() returns len=10000 bytes (ok) >>>>> Call #2 to SQLGetData() returns len=5905 bytes >>>>> there was 5905 bytes left to read, because only 4095 bytes could >>>>> be written to the buffer, because of additional \0 terminator. >>>>> 10000 - 4095 = 5905 ... >>>>> Call #3 to SQLGetData() returns len=1810 bytes >>>>> 5905 - 4095 = 1810 ... >>>>> Last call: 1810 bytes fit in the 4096 buffer. >>>>> >>>> >>>> Well, this is expected. But I don't understand what's going wrong with Solaris. >>>> Could be possible that Solaris do not convert some characters and so >>>> it convert a byte characters to 0 bytes ?? >>> >>> Possible... >>> >>>> Are you sure that there is only ASCII characters (0-127) ?? >>> >>> Pure ASCII. >>> >>>> Why should Solaris report some warning converting ASCII ?? >>> >>> My C locale is set to POSIX/C, I don't know why it ends as ISO-8859-1 >>> for the iconv conversions in FreeTDS. Maybe it's the default. >>> >>> Is there some mapping between the C locale (LANG/LC_ALL) and the iconv >>> locale name (I know these can be different names, how is this mapped?) >>> >>>> A dump would be helpful. Send privately to me if it contains sensitive >>>> information. >>> >>> I send you a dump. >>> >>>> You could try using libiconv. >>> >>> We'll try this too. >>> >>>>> I hope this helps... >>>>> >>>>> Seb >>>>> >>>> >>>> Frediano >>>> >>>> >>>>> >>>>> On 02/03/2013 12:12 PM, Frediano Ziglio wrote: >>>>>> Hi, >>>>>> where is SQL_DATA_AT_EXEC in your code? >>>>>> >>>>>> Which DM are you using? Is possible that for some reason DM call some >>>>>> additional function to do some checks. This could discovered enabling >>>>>> FreeTDS dump. >>>>>> >>>>>> Quire strange it works correctly on Linux but not on Solaris. >>>>>> Does data contains no-ASCII characters? >>>>>> How long are usually these TEXT fields? >>>>>> Which TDS protocol version are you using? >>>>>> There is a specific getdata test for this stuff and should work. >>>>>> >>>>>> Regards, >>>>>> Frediano >>>>>> >>>>>> 2013/2/2 Sebastien FLAESCH<[email protected]>: >>>>>>> Hi Frediano, >>>>>>> >>>>>>> FYI, we use a single-byte encoding (LANG=POSIX) ... >>>>>>> >>>>>>> Here is the code we use, nothing particular, this is the same code we >>>>>>> use for different ODBC drivers... >>>>>>> >>>>>>> #define BLOB_BLOCK_SIZE 4096 >>>>>>> .... >>>>>>> static SQLRETURN getBlobPieces(SqlStatement * st, SQLUSMALLINT fpos, >>>>>>> SQLSMALLINT ctype, MyLocator * loc, >>>>>>> FILE * out, SQLLEN * plen) >>>>>>> { >>>>>>> SQLRETURN r = 0; >>>>>>> char buf[BLOB_BLOCK_SIZE]; >>>>>>> SQLLEN len; >>>>>>> int i = 0; >>>>>>> while (1) { >>>>>>> r = SQLGetData(st->stmtHandle, fpos, ctype, >>>>>>> (SQLPOINTER) buf, sizeof(buf),&len); >>>>>>> if (!SQL_SUCCEEDED(r) || len == 0 || len == SQL_NULL_DATA) >>>>>>> break; >>>>>>> if (len> (SQLLEN) sizeof(buf) || len == SQL_NO_TOTAL) >>>>>>> len = sizeof(buf); >>>>>>> /* Char data piece get \0, these must be removed! */ >>>>>>> if (ctype == SQL_C_CHAR) { >>>>>>> if (buf[len - 1] == '\0') { >>>>>>> len--; >>>>>>> } >>>>>>> } >>>>>>> fwrite(buf, 1, len, out); >>>>>>> } >>>>>>> *plen = len; >>>>>>> return r; >>>>>>> } >>>>>>> >>>>>>> Am I missusing SQLGetData()? As you can see, I modify the len after >>>>>>> Does this affect the next call to SQLGetData()? >>>>>>> >>>>>>> Anyway, the INITIAL len returned by SQLGetData() is for sure: >>>>>>> >>>>>>> (total-bytes - 1) >>>>>>> >>>>>>> When using UTF-8, I get SQL_NO_TOTAL in the first call, but in ASCII, >>>>>>> I get the (total size - 1)... >>>>>>> >>>>>>> Seb >>>>>>> >>>>>>> On 02/02/2013 08:52 AM, Frediano Ziglio wrote: >>>>>>>> Il giorno 31/gen/2013, alle ore 14:28, Sebastien FLAESCH<[email protected]> ha scritto: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Context: >>>>>>>>> >>>>>>>>> $ uname -a >>>>>>>>> SunOS s64x100 5.10 Generic_120012-14 i86pc i386 i86pc >>>>>>>>> >>>>>>>>> FreeTDS versions tested: >>>>>>>>> - freetds-0.92.dev.20120312 >>>>>>>>> - freetds-dev.0.92.377 >>>>>>>>> >>>>>>>>> When fetching a TEXT from the database (SQL Server 2005), using SQL_DATA_AT_EXEC, >>>>>>>>> it appears that the SQLGetData() function returns the size of the text data, >>>>>>>>> minus one... >>>>>>>>> >>>>>>>>> I have tested the same FreeTDS version on a Linux 32b, and SQLGetData() returns >>>>>>>>> the correct length as expected... >>>>>>>>> >>>>>>>> >>>>>>>> Strange! How do you read the length? >>>>>>>> >>>>>>>>> I will continue to investigate, but I wanted to ask if this is a know issue. >>>>>>>>> >>>>>>>>> When connecting to SQL Server 2008 R2, this does not occur, because SQLGetData() >>>>>>>>> returns SQL_NO_TOTAL and we fetch text pieces in a different way... >>>>>>>>> >>>>>>>> >>>>>>>> This is done to deal with different encodings. >>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Seb >>>>>>>> >>>>>>>> Frediano >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> FreeTDS mailing list >>>>>>>> [email protected] >>>>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> FreeTDS mailing list >>>>>>> [email protected] >>>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds >>>>>> _______________________________________________ >>>>>> FreeTDS mailing list >>>>>> [email protected] >>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds >>>>>> >>>>> >>>>> _______________________________________________ >>>>> FreeTDS mailing list >>>>> [email protected] >>>>> http://lists.ibiblio.org/mailman/listinfo/freetds >>>> _______________________________________________ >>>> FreeTDS mailing list >>>> [email protected] >>>> http://lists.ibiblio.org/mailman/listinfo/freetds >>>> >>> >>> _______________________________________________ >>> FreeTDS mailing list >>> [email protected] >>> http://lists.ibiblio.org/mailman/listinfo/freetds >>> >> >> _______________________________________________ >> FreeTDS mailing list >> [email protected] >> http://lists.ibiblio.org/mailman/listinfo/freetds >> > > _______________________________________________ > FreeTDS mailing list > [email protected] > http://lists.ibiblio.org/mailman/listinfo/freetds