Re: Problem with UTF-8

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Nicolas,

Brisset, Nicolas wrote:
> Hi,
> 
> I've discovered that it is possible to create in-memory databases with
> pysqlite, and I wanted to use that feature to work with tabulated files
> (containing accentuated characters). I wrote the attached test script,
> which almost works. The remaining problem is that I get errors like:
> 
>>>> print cur.fetchall()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> pysqlite2.dbapi2.OperationalError: Could not decode to UTF-8 column 'Col1' with text '  texte où il y a des éàè '

That's because you inserted bytestrings in the SQLite database which are
not encoded in UTF-8. This is invalid for SQLite. Unfortunately, you only
get bitten *later* when you try to fetch these strings.

The solution is to only insert valid strings in the SQLite database. You
can ensure this by only using unicode strings with the pysqlite interface.
Often this means you have to convert to unicode first.

Suppose you have a bytestring 's' that you read from a file, and the file
is encoded in latin-1 aka iso-8859-1:

s = ...
# now convert to unicode
us = unicode(s, "latin1")  # if it's not the correct encoding, you will
# get an exception here!
cur.execute("insert into ... values (?)", (us,))

> [...]

HTH,

- -- Gerhard
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFZzoDdIO4ozGCH14RAkRLAJ0SeTe6rs001W6Z3ExDw/KqRmzH1QCggxkw
6MfaXImna/XFJ4YXZmjLB+g=
=/s3V
-----END PGP SIGNATURE-----