Re: [pysqlite] Explanation of 2.5.0 behavioral change

Hugh Gibson wrote:
>> pysqlite now helps to not shoot yourself in the foot with Unicode 
>> vs. bytestrings. And it's quite strict about it. If you do this:
>>
>> ...snip...
>>
>> So it's now either:
>>
>> - - Unicode in, Unicode out
>>
>> or
>>
>> - - bytestring in, bytestring out
>>
>> This change is not entirely backwards compatible, but at the time I
>> believed it was worth it. This way, errors are raised early, at 
>> insert time into the database, instead of failing days/weeks/months
>> later when trying to fetch them from the database.
> 
> We got the warning message. I simply changed the connection text_factory
> to OptimisedUnicode. Is this a reasonable thing to do or are we going to
> have further problems?

It only works because of a bug (see below).

> For reference, see
> http://oss.itsystementwicklung.de/download/pysqlite/doc/sqlite3.html#sqlit
> e3.Connection.text_factory
> 
> I see that the changes were made as part of checkin 307 at
> http://oss.itsystementwicklung.de/trac/pysqlite/changeset/307%3A0ec0f22865
> 8e
> 
> The code to set allow_8bit_chars is the same in the most recent version
> of cursor.c. Is there an error in the boolean statement? It is:
>  
> /* Make shooting yourself in the foot with not utf-8 decodable
> 8-bit-strings harder */ 
> allow_8bit_chars = ((self->connection->text_factory !=
> (PyObject*)&PyUnicode_Type) && 
> (self->connection->text_factory != (PyObject*)&PyUnicode_Type &&
> pysqlite_OptimizedUnicode)); [...]

You're right. There's a bug here, I was AND-ing instead of comparing ti 
with pysqlite_OptimizedUnicode.

If I fix this bug, setting switching from Unicode to OptimizedUnicode 
won't work any longer.

I'll need to think about this a bit more. And maybe switch to emit a 
warning instead of raising a ProgrammingError.

-- Gerhard