Re: Unicode and Sybase univarchar

>> ... should just work. It doesn't quite, because the hex string is not 
>> just a dump of a 16 Bit Unicode encoding, but it is a UTF-8 byte 
>> stream written with a 16 Bit Hex Format for each byte. Each and every 
>> 16-Bit-Word has its most significant byte set to 0.
>>
>> If it was a dump of a 16 Bit Unicode encoding, it should read
>> "0065006d00200064006100730068003a00202014"
>> and not
>> "0065006d00200064006100730068003a002000e200800094"
>>
>> Your call to decode() compensates that, probably because you encoded 
>> once too much before writing the data into the database.
>
> I did not encode at all. I simply created a utf8 string in Perl land 
> and inserted it into Sybase.
And Sybase / DBD::Sybase happily ignored Perl's UTF8 flag and stored the 
byte stream as if it were characters.
>
> Really, DBD::Sybase needs to handle any character set translation, not 
> the end user.
>
Right. (But remember that DBI was there before Unicode support was added 
to Perl, and also most DBDs are older that the Unicode support. Before 
Unicode was there, you just passed bytes around and everything just worked.)

For the same reason, but with a different DBD, I hacked the first 
Unicode patch for DBD::ODBC, just enough code to have proper Unicode 
support in bind parameters from Perl to the database, and in returned 
fetch values. DBD::ODBC has been improved since the patch was merged by 
Martin J. Evans into v1.14, and it now supports Unicode in many other 
places, too. The DBI API was also improved during that process, allowing 
to have Unicode SQL query strings.

That's why I proposed to switch to DBD::ODBC: It is well tested and 
supports Unicode as good as the ODBC driver does.

The raw patch is still available at 
<http://www.alexander-foken.de/unicode-patch.txt.gz>. DBD::Sybase 
obviously lacks such a patch. Sybase may have a Unicode API, but no part 
of DBD::Sybase uses it (properly). In DBD::ODBC, the patch "just" tests 
the UTF8 flag for all relevant data coming from Perl and converts UTF8 
to the UCS2 encoding required by the ODBC API, and converts UCS2 encoded 
data back to UTF8, settings the UTF8 flag when needed. Most of the new 
test just tests that all conversions work, even inside the database. The 
length check is very primitive, but a good indicator to test if the 
database saw bytes (length too large) or characters (length just right). 
The original patch has some minor problems in the test code with ancient 
perls that were fixed during DBD::ODBC development, see DBD::ODBC Changes.

Now you "just" need to find someone who is willing and has the time to 
patch DBD::Sybase ... ;-)
The "new" tests should work nearly unmodified with a properly patched 
DBD::Sybase.

Alexander