Re: Conversion emoji symbols to UTF-8

2016-04-20 12:01 GMT+01:00 Velichko Yuriy <[email protected]>:
> Your query works for me to:
>
> select cast(cast(0x420435044104420420003dd84ddc20007400650073007400 as
> nvarchar(200)) as sql_variant)
>
> Here an example for broken result:
>
> create sequence s1;
> select sys_seq.start_value FROM sys.sequences sys_seq where name = 's1';
>

Is this related to emoji/encoding ?

Can you post a TDSDUMP. I don't have a mssql 2012 to test with (it's
tested by AppVeyor).

Frediano

> On 20 April 2016 at 13:13, Frediano Ziglio <[email protected]> wrote:
>
>> 2016-04-20 9:39 GMT+01:00 Velichko Yuriy <[email protected]>:
>> > One more strange thing.
>> >
>> > 1. As you mentioned the default property use utf-16 differs on win (TRUE)
>> > and mac (FALSE). So I must to set "use utf-16 = yes" to avoid different
>> > behaviour, Good that it will be fixed in 1.0
>>
>> no, the default is the same, just that you are using the trivial iconv
>> provided with FreeTDS which is not recommended.
>>
>> > 2.
>> >
>> > tds version = 7.2 - has no effect for emoji
>> > symbols
>> > just need to use
>> > utf-16
>> > .
>> >
>>
>> Good, so now should work for every protocol version.
>>
>> > 3
>> >
>> >
>> >
>> > tds version = 7.2
>> > - brokes another kind of strings.
>>
>> Which ones?
>>
>> > After I set "
>> >
>> > tds version = 7.2
>> >
>> > "
>> > I got few broken tests.
>>
>> Which tests ?
>>
>> > I found out that strings returned for sql_variant has the same problems
>> > like for emoji symbols - string starts with
>> >
>> > \0 symbol and has non zero length.
>> > Looks like similar conversion issue. Using "
>> >
>> > tds version = 7.
>> > 0
>> > "
>> > - fixes this issue - returned value is valid.
>>
>> Weird. I'm trying this query on tsql
>>
>> select cast(cast(0x420435044104420420003dd84ddc20007400650073007400 as
>> nvarchar(200)) as sql_variant)
>>
>> (which is based on your previous string) and looks the strings are
>> always returned correctly.
>> I tried from 7.0 to 7.3 without issues.
>>
>> > To test this you can create Sequence and run query:
>> > select sys_seq.start_value FROM sys.sequences sys_seq
>> >
>>
>> Can you post all queries you are using? Which exact sequence is not
>> working?
>> Are you trying some invalid coding?
>>
>> Frediano
>>
>> >
>> >
>> >
>> > On 19 April 2016 at 13:53, Velichko Yuriy <[email protected]>
>> wrote:
>> >
>> >> Yes, without "use utf-16" symbols are broken.
>> >> Set
>> >> use utf-16 = yes
>> >> and
>> >> tds version = 7.2
>> >> fixes this problem on mac.
>> >>
>> >> If tds version = 7.0 - symbols are broken.
>> >>
>> >> On 19 April 2016 at 13:26, Frediano Ziglio <[email protected]> wrote:
>> >>
>> >>> 2016-04-18 21:46 GMT+01:00 Daniel Fazekas <[email protected]>:
>> >>> > It is actually odd that merely upgrading solved that emoji issue.
>> >>> > Emojis being largely in the Supplementary Multilingual Plane of
>> Unicode
>> >>> require true UTF-16 support rather than the legacy UCS-2 encoding on
>> the
>> >>> server communication side.
>> >>> >
>> >>> > In FreeTDS terms, that means that in addition to version 0.95+ you
>> >>> ought to include the following extra setting in your configuration
>> file:
>> >>> > use utf-16 = yes
>> >>> >
>> >>> > So a complete sample configuration would look something like
>> >>> > tds version = 7.2
>> >>> > client charset = UTF-8
>> >>> > use utf-16 = yes
>> >>> >
>> >>> > Was it a Windows platform peculiarity that UTF-16 worked even with
>> that
>> >>> required setting missing?
>> >>>
>> >>> This as replacements iconv use utf16 in place of ucs2.
>> >>> This however reminds me that "use utf-16" option default should be
>> >>> changed to 1 (true) for 1.0 version!
>> >>> I'm doing it right now.
>> >>>
>> >>> Frediano
>> >>>
>> >>> > In any case, I'd add that to your configuration just to be sure.
>> >>> > It certaintly was necessary for me on Mac and Linux.
>> >>> >
>> >>> >> On Apr 18, 2016, at 12:09, Velichko Yuriy <[email protected]
>> >
>> >>> wrote:
>> >>> >>
>> >>> >> With version 0.95.91 all works fine. looks like this issue was fixed
>> >>> >> earlier.
>> >>> >>
>> >>> >>>>>>>>> On 17 Apr 2016 12:59, "Velichko Yuriy" <
>> >>> [email protected]> wrote:
>> >>> >>>>>>>>> My .conf includes next rows:
>> >>> >>>>>>>>>    tds version = 7.0
>> >>> >>>>>>>>>    client charset = UTF-8
>> >>> >
>> >>> > _______________________________________________
>> >>> > FreeTDS mailing list
>> >>> > [email protected]
>> >>> > http://lists.ibiblio.org/mailman/listinfo/freetds
>> >>> _______________________________________________
>> >>> FreeTDS mailing list
>> >>> [email protected]
>> >>> http://lists.ibiblio.org/mailman/listinfo/freetds
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> С уважением, Величко Юрий.
>> >>
>> >
>> >
>> >
>> > --
>> > С уважением, Величко Юрий.
>> > _______________________________________________
>> > FreeTDS mailing list
>> > [email protected]
>> > http://lists.ibiblio.org/mailman/listinfo/freetds
>> _______________________________________________
>> FreeTDS mailing list
>> [email protected]
>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>
>
>
>
> --
> С уважением, Величко Юрий.
> _______________________________________________
> FreeTDS mailing list
> [email protected]
> http://lists.ibiblio.org/mailman/listinfo/freetds
_______________________________________________
FreeTDS mailing list
[email protected]
http://lists.ibiblio.org/mailman/listinfo/freetds