Re: Conversion emoji symbols to UTF-8

It's not clear which library are you using. Note that SQL_VARIANT is
not supported in TDS 7.0 so the server send data as normal text. Could
be some paths/libraries do not support correctly SQL_VARIANT.

Frediano


2016-04-20 13:06 GMT+01:00 Velichko Yuriy <[email protected]>:
> I have tried FreeTDS from here, and all work fine. So the error on my side.
> Will try to find.
>
> On 20 April 2016 at 14:41, Velichko Yuriy <[email protected]> wrote:
>
>>
>>
>> On 20 April 2016 at 14:07, Frediano Ziglio <[email protected]> wrote:
>>
>>> 2016-04-20 12:01 GMT+01:00 Velichko Yuriy <[email protected]>:
>>> > Your query works for me to:
>>> >
>>> > select cast(cast(0x420435044104420420003dd84ddc20007400650073007400 as
>>> > nvarchar(200)) as sql_variant)
>>> >
>>> > Here an example for broken result:
>>> >
>>> > create sequence s1;
>>> > select sys_seq.start_value FROM sys.sequences sys_seq where name = 's1';
>>> >
>>>
>>> Is this related to emoji/encoding ?
>>>
>> Already no, I think. Should I create new thread?
>>
>>
>>> Can you post a
>>>
>>> TDSDUMP. I don't have a mssql 2012 to test with (it's
>>> tested by AppVeyor).
>>>
>>> Frediano
>>>
>>> > On 20 April 2016 at 13:13, Frediano Ziglio <[email protected]> wrote:
>>> >
>>> >> 2016-04-20 9:39 GMT+01:00 Velichko Yuriy <[email protected]>:
>>> >> > One more strange thing.
>>> >> >
>>> >> > 1. As you mentioned the default property use utf-16 differs on win
>>> (TRUE)
>>> >> > and mac (FALSE). So I must to set "use utf-16 = yes" to avoid
>>> different
>>> >> > behaviour, Good that it will be fixed in 1.0
>>> >>
>>> >> no, the default is the same, just that you are using the trivial iconv
>>> >> provided with FreeTDS which is not recommended.
>>> >>
>>> >> > 2.
>>> >> >
>>> >> > tds version = 7.2 - has no effect for emoji
>>> >> > symbols
>>> >> > just need to use
>>> >> > utf-16
>>> >> > .
>>> >> >
>>> >>
>>> >> Good, so now should work for every protocol version.
>>> >>
>>> >> > 3
>>> >> >
>>> >> >
>>> >> >
>>> >> > tds version = 7.2
>>> >> > - brokes another kind of strings.
>>> >>
>>> >> Which ones?
>>> >>
>>> >> > After I set "
>>> >> >
>>> >> > tds version = 7.2
>>> >> >
>>> >> > "
>>> >> > I got few broken tests.
>>> >>
>>> >> Which tests ?
>>> >>
>>> >> > I found out that strings returned for sql_variant has the same
>>> problems
>>> >> > like for emoji symbols - string starts with
>>> >> >
>>> >> > \0 symbol and has non zero length.
>>> >> > Looks like similar conversion issue. Using "
>>> >> >
>>> >> > tds version = 7.
>>> >> > 0
>>> >> > "
>>> >> > - fixes this issue - returned value is valid.
>>> >>
>>> >> Weird. I'm trying this query on tsql
>>> >>
>>> >> select cast(cast(0x420435044104420420003dd84ddc20007400650073007400 as
>>> >> nvarchar(200)) as sql_variant)
>>> >>
>>> >> (which is based on your previous string) and looks the strings are
>>> >> always returned correctly.
>>> >> I tried from 7.0 to 7.3 without issues.
>>> >>
>>> >> > To test this you can create Sequence and run query:
>>> >> > select sys_seq.start_value FROM sys.sequences sys_seq
>>> >> >
>>> >>
>>> >> Can you post all queries you are using? Which exact sequence is not
>>> >> working?
>>> >> Are you trying some invalid coding?
>>> >>
>>> >> Frediano
>>> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > On 19 April 2016 at 13:53, Velichko Yuriy <[email protected]>
>>> >> wrote:
>>> >> >
>>> >> >> Yes, without "use utf-16" symbols are broken.
>>> >> >> Set
>>> >> >> use utf-16 = yes
>>> >> >> and
>>> >> >> tds version = 7.2
>>> >> >> fixes this problem on mac.
>>> >> >>
>>> >> >> If tds version = 7.0 - symbols are broken.
>>> >> >>
>>> >> >> On 19 April 2016 at 13:26, Frediano Ziglio <[email protected]>
>>> wrote:
>>> >> >>
>>> >> >>> 2016-04-18 21:46 GMT+01:00 Daniel Fazekas <[email protected]>:
>>> >> >>> > It is actually odd that merely upgrading solved that emoji issue.
>>> >> >>> > Emojis being largely in the Supplementary Multilingual Plane of
>>> >> Unicode
>>> >> >>> require true UTF-16 support rather than the legacy UCS-2 encoding
>>> on
>>> >> the
>>> >> >>> server communication side.
>>> >> >>> >
>>> >> >>> > In FreeTDS terms, that means that in addition to version 0.95+
>>> you
>>> >> >>> ought to include the following extra setting in your configuration
>>> >> file:
>>> >> >>> > use utf-16 = yes
>>> >> >>> >
>>> >> >>> > So a complete sample configuration would look something like
>>> >> >>> > tds version = 7.2
>>> >> >>> > client charset = UTF-8
>>> >> >>> > use utf-16 = yes
>>> >> >>> >
>>> >> >>> > Was it a Windows platform peculiarity that UTF-16 worked even
>>> with
>>> >> that
>>> >> >>> required setting missing?
>>> >> >>>
>>> >> >>> This as replacements iconv use utf16 in place of ucs2.
>>> >> >>> This however reminds me that "use utf-16" option default should be
>>> >> >>> changed to 1 (true) for 1.0 version!
>>> >> >>> I'm doing it right now.
>>> >> >>>
>>> >> >>> Frediano
>>> >> >>>
>>> >> >>> > In any case, I'd add that to your configuration just to be sure.
>>> >> >>> > It certaintly was necessary for me on Mac and Linux.
>>> >> >>> >
>>> >> >>> >> On Apr 18, 2016, at 12:09, Velichko Yuriy <
>>> [email protected]
>>> >> >
>>> >> >>> wrote:
>>> >> >>> >>
>>> >> >>> >> With version 0.95.91 all works fine. looks like this issue was
>>> fixed
>>> >> >>> >> earlier.
>>> >> >>> >>
>>> >> >>> >>>>>>>>> On 17 Apr 2016 12:59, "Velichko Yuriy" <
>>> >> >>> [email protected]> wrote:
>>> >> >>> >>>>>>>>> My .conf includes next rows:
>>> >> >>> >>>>>>>>>    tds version = 7.0
>>> >> >>> >>>>>>>>>    client charset = UTF-8
>>> >> >>> >