Re: [pysqlite] From PySqLite to ASPW - hoping to solve encoding problem

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pierre-Yves Delens wrote:
> The file is supposed to be in _Unicode_.

That statement is not true.  The file is a collection of bytes and to
turn it into Unicode you need to know the encoding of the bytes.  The
two most ways of encoding a Unicode string into bytes are UTF-8 and
UTF-16 (although older software is actually usually doing UCS-2).

> After many tests in UTF-8 db files,  I came back to UTF-16 dbfiles

If you are talking about the encoding of the SQLite database then that
makes no difference whatsoever to behaviour.  Other than file size you
cannot even tell what the encoding used is.

> Accentuated characters come OK in the SqLite DB:
>     Ar20----__-$PAREMENT FA=C7ADES ET =C9TANCH=C9IT=C9

Which API are you using against SQLite? How are you then getting that
string above since it has gone from SQLite into your code and then back
out again and then into an email.

> =

> When processed by ASPW (idem PySqLite)
> =

> I get this.
>     (u'Ar20----__-$PAREMENT FA\xc7ADES ET \xc9TANCH\xc9IT\xc9', ....)

Your email seems to be implying that something is wrong with that?
Looks perfectly good to me.  Python's behaviour may appear a little
strange, but it does make sense.

  >>> print repr(u"abcd")
  u'abcd'
  >>> print repr(u"abcd\u00c7")
  u'abcd\xc7'
  >>> print repr(u"abcd\U000001c7")
  u'abcd\u01c7'
  >>> print repr(u"abcd\U001001c7")
  u'abcd\U001001c7'

As you can see it outputs each code point is given the shortest
representation possible.  It doesn't know how that code point was
specified in the first place.

> If I use 'decode(UTF-16) (and idem if UTF-8), Python claims  codecs
> can't decode .

What exactly is it you are decoding?  If it is the string above then it
is already a string.  (Decoding converts bytes to a string).  If you
want to encode it then pick a byte based representation such as utf8.

> Isn't there a way to process this kind of string as to get the
> characters properly displayed?

What do you mean by properly displayed? Once you have a Unicode string,
getting it shown in a text console or graphical user interface involves
doing whatever their APIs require.  Typically (on Unix) text consoles
have to be fed bytes in whatever encoding the console is set to.  GUIs
usually have APIs that take Unicode directly if using Python or require
to be in UTF8/UTF16 if using a lower level language like C.

You can see what SQLite does with strings by just selecting the value.
This demonstrates using APSW to feed SQLite various strings and shows
what comes back:

 >>> import apsw
 >>> con=3Dapsw.Connection(":memory:")
 >>> cur=3Dcon.cursor()
 >>> for t in ("abcd", u"abcd\u00c7", u"abcd\u01c7", u"abcd\U001001c7"):
 ...   print repr(t), repr(cur.execute("select ?", (t,)).fetchall())
 ...
 'abcd' [(u'abcd',)]
 u'abcd\xc7' [(u'abcd\xc7',)]
 u'abcd\u01c7' [(u'abcd\u01c7',)]
 u'abcd\U001001c7' [(u'abcd\U001001c7',)]

Roger

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkpjvfQACgkQmOOfHg372QRoZACgqrRCS1pVPP2liEizubMspN/C
y2IAoJ3yKdSW137OC3IuxEBrxyTEVcRW
=3DgEd9
-----END PGP SIGNATURE-----