Re: [pysqlite] more Unicode grief

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Eric S. Johansson wrote:
> Gerhard Häring wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>>
>> The SQLite API returns UTF-8 encoded C strings. In order to produce
>> Unicode objects, pysqlite has to do the Python C-API equivalent of calling
>>
>> unicode(..., "utf-8")
>>
>> So, the problem is still that you've got non UTF-8 data in your database.
>>
>> I'll change pysqlite in the next release so that it will be impossible
>> to create such scenarios at least by using pysqlite to enter data.
> 
> not intending to sound too pissy but that doesn't help me a whole lot right now 
>   because I have a broken application and a cranky customer.  :-)

Ok, I'll try to help :-)

> The fact remains that I have this non-ASCII text I have to deal with.  I have 
> normalized all of this text into Unicode.  What do form do I need to convert it 
> into in order to have something that can be stored, searched, and retrieved?

- From what you're telling, you're doing it all correctly.

insert unicode objects into the database, and you'll get unicode objects
back. And everything is fine. Apparently, not everything is fine, so it
appears that at one point in time, non-utf8-decodable text got inserted
into the database.

> Like I said earlier, the documentation says I can just convert everything to 
> Unicode and it will be converted appropriately into and out of pysqlite. 

That's how it is. The code in question has been in place pretty much
unchanged since pysqlite 2.0 beta. If there really was a problem there,
it would have been discovered earlier ;-)

> If that is not the case, please, I ask you to fix the documentation so that other 
> folks are not led astray.
> 
> I apologize for my crankiness but I am currently receiving a fair amount of 
> heat and I'm trying to figure out a reasonable solution for this problem.

Can you switch your application to the Mercurial-tip version of
pysqlite? I've just committed a change a few minutes ago that makes
pysqlite barf on you if you *insert* the problematic data into the
database instead of freaking out later when it fetches the sh*t.

http://oss.itsystementwicklung.de/hg/pysqlite/rev/0ec0f228658e

Get it at: http://oss.itsystementwicklung.de/hg/pysqlite/archive/tip.tar.gz


Very ugly, but something like

con.text_factory = lambda x: unicode(x, "utf-8", "ignore")

can perhaps buy you time to find the real problem.

- -- Gerhard
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH5up1dIO4ozGCH14RAtX/AKCKq+TztqKA8zRKwFjgeoxIhVjdogCggFE3
1MyNEk2wEcrs3H6j9UI3xmo=
=fW05
-----END PGP SIGNATURE-----