Re: Unicode and Sybase univarchar

On Jun 3, 2010, at 8:14 PM, Dave Rolsky wrote:

> On Thu, 3 Jun 2010, Michael Peppler wrote:
> 
>> Which version of Sybase, which version of Sybase OpenClient, and which version of DBD::Sybase?
> 
> Ah, I was using the old libs (11.0), which may have been the problem. I was also using DBD::Sybase 1.07.
> 
> I switch to Sybase 15.0 (OCS 15.0 if that makes sense), OpenClient 15.0 libs, and DBD::Sybase 1.10.
> 
> Now it's closer to working. If I set "charset=utf8" in the dsn, I get
> 
> 2010/06/03 14:08:11 unicode CRITICAL: FATAL: DBD::Sybase::db do failed: Server message number=2402 severity=16 state=1 line=1 server=HDATADEV1 text=Error converting characters into server's character set. Some character(s) could not be converted.
> 
> I'm not sure what that means.

Hmmm - is that on a query, or on an insert operation?


> If I _don't_ set that, the data goes in and comes out as bytes, rather than the bizarro hex string. However, the data does have the utf8 flag set when it comes back from Sybase, so I have to run it through Encode::decode.
> 
> I really don't think I can realistically tell the bazillion developers here "just run all the data through Encode".
> 
> I'd really like see an end-to-end solution.

I agree - I've just not had much opportunity (or requests) to ensure that this works 100%.

Ideally if you could send me some sample code, and a simple table structure and data that reproduces the problem for you I could try to look at it and see if I can fix it.


> Also, it's not clear to me that the data is actually being stored as characters at the Sybase level. I'm not even sure how I'd figure this out. When I do a select from sqsh, I see the wacky hex string, but I can't tell if that's Sybase trying to present data to me in a format it thinks my environment can handle.

When in doubt - use the Sybase tools (i.e. isql, and use -Jutf8 to force conversion to/from utf8 when reading/writing the data).

You can also do a convert(varbinary, the_univarchar_col) to see the hex representation of the data in the database.

> 
> Basically, what I need is to be able to take Perl native unicode strings, store them in Sybase in Sybase's native format (utf16, I believe), and then retrieve them as Perl native unicode strings again.

If you store data in univarchar, etc columns, then it's utf16. You can also set up the dataserver to use utf8 throughout - but that's something you'd have to discuss with your DBAs.

Michael
--
Michael Peppler
Sybase on Linux FAQ: http://www.peppler.org/FAQ/linux.html

"A successful [software] tool is one that was used to do something undreamed of by its author." -- S. C. Johnson