Re: Unicode and Sybase univarchar

Sorry, forgot to CC to the list ...

-------- Original Message --------
Subject: 	Re: Unicode and Sybase univarchar
Date: 	Thu, 03 Jun 2010 22:20:37 +0200
From: 	Alexander Foken <[email protected]>
To: 	Dave Rolsky <[email protected]>
References: 	<[email protected]>



On 03.06.2010 17:22, Dave Rolsky wrote:
>
> What's really bizarre is that when I select the value back I get 
> something like "0065006d00200064006100730068003a002000e200800094".
>
> Yes, that's a literal string containing a series of 2-digit hex numbers!
>
> I can translate this back to Perl unicode with this madness:
>
>     my $chars = do {
>         use bytes;
>
>         join q{}, map { chr( eval '0x' . $_ ) } $fromdb =~ /(....)/g;
>     };
>
>     my $unicode = decode( 'utf8', $chars );
>
Really strange way to avoid pack()/unpack(). At least, you can get rid 
of the evil string-eval, simply use hex($_).  The combination of "use 
bytes", chr() with an argument larger than 0x00FF, and decode() also 
looks very strange. $fromdb contains only hex digits, so there should be 
no need to "use bytes". Unless you force byte mode, chr() should already 
return perfect Unicode, flagged as such. So ...

my $unicode=join('',map { chr hex $_ } $fromdb=~/([0-9a-fA-F]{4})/g);

... should just work. It doesn't quite, because the hex string is not 
just a dump of a 16 Bit Unicode encoding, but it is a UTF-8 byte stream 
written with a 16 Bit Hex Format for each byte. Each and every 
16-Bit-Word has its most significant byte set to 0.

If it was a dump of a 16 Bit Unicode encoding, it should read
"0065006d00200064006100730068003a00202014"
and not
"0065006d00200064006100730068003a002000e200800094"

Your call to decode() compensates that, probably because you encoded 
once too much before writing the data into the database.

Appart from that, have a look at the tests in DBD::Oracle, there are a 
few tests for Unicode round trips in 40UnicodeRoundTrip.t and 
41Unicode.t, try to run them on DBD::Sybase.

Also consider using DBD::ODBC, a Unicode capable ODBC manager (the 
one(s) on Windows is/are fine) and an Unicode capable ODBC driver for 
Sybase. It may cost you a few CPU cycles for the extra layers, but 
DBD::ODBC supports Unicode quite well (on non-Windows, you need to 
explicitly enable Unicode).

Alexander

-- 

Alexander Foken
mailto:[email protected]  http://www.foken.de/alexander/