Re: [pysqlite] Encoding for connect to SqLite3
"Dinesh B Vadhia" <[email protected]> Fri, 26 Jun 2009 09:32:42 -0700
| Newsgroups | gmane.comp.python.db.pysqlite.user |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --===============1272235865== Content-Type: multipart/alternative; boundary="----=_NextPart_000_002D_01C9F641.0E4CEF80" This is a multi-part message in MIME format. ------=_NextPart_000_002D_01C9F641.0E4CEF80 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I'm still confused about sqlite and unicode, utf-8, text_factory=3Dstr = etc. etc. All my python programs have been written to be utf-8 compatible ie. data = is read-in and written-out as utf-8 and hopefully ( :) ) python = processes the data as utf-8 within the program. Wrt sqlite, when the db is initialized and opened, text-factory=3Dstr is = set in order to stop "sqlite3.OperationalError: Could not decode to = UTF-8 column 'j' with text" errors. But, I'm still not confident that I fully understand what is going on = wrt python, sqlite and utf-8. All explanations from basics would be = really appreciated. Dinesh ---------------------------------------------------------------------- Message: 1 Date: Thu, 25 Jun 2009 19:54:36 +0200 From: Gerhard H?ring <[email protected]> Subject: Re: [pysqlite] Encoding for connect to SqLite3 To: PY-Delens <[email protected]> Cc: "About pysqlite and APSW." <list-pysqlite-FR6EJeJVuqdwc357pe9rcyQmJico6nz3epZhswDD4dQ@public.gmane.org> Message-ID: <[email protected]> Content-Type: text/plain; charset=3DISO-8859-1 Hello, I'll cc the mailing list. Please direct future questions there. PY-Delens wrote: > Bonjour, >=20 > I fail to display correctly accentuated characters from aSqLite 3 = table > (stored in UTF-8 after abandoned UTF-16 scenario). > I try to add parameter _/encoding /_to thre PySqlite connect function, > but this is refused: [...] No surprise. pysqlite doesn't have an encoding parameter anywhere. To store text that is not UTF8 encoded, you can achieve that using your custom text_factory (See http://oss.itsystementwicklung.de/download/pysqlite/doc/sqlite3.html). > I'm alo confused on the libs supposed to be called in PySqlite; = probably > one is nested into the other: > pysqlite2 > Sqlite3 [...] pysqlite2 is the Python wrapper for the SQLite 3 library, which is the embedded database engine. If you use Python 2.5 or later, a version of pysqlite2 is available under the module name "sqlite3" as well. > How should I do? How to handle the encoding problem at: > - Python module level > - SQL string level > - DATA in SqLite file (I was advised to go back to UTF-8, by = people > of DbManager tools) > ? I'd recommend to use UTF8 only. If you don't, you'll need to use bytestrings in your encoding as parameters to execute(). And set a text_factory that converts them back. If you just want the same bytestrings back (instead of, say, unicode string objects), you can just use text_factory=3Dstr. -- Gerhard ------=_NextPart_000_002D_01C9F641.0E4CEF80 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META content=3Dtext/html;charset=3Diso-8859-1 = http-equiv=3DContent-Type> <META name=3DGENERATOR content=3D"MSHTML 8.00.6001.18783"></HEAD> <BODY style=3D"PADDING-LEFT: 10px; PADDING-RIGHT: 10px; PADDING-TOP: = 15px"=20 id=3DMailContainerBody leftMargin=3D0 topMargin=3D0 = CanvasTabStop=3D"true"=20 name=3D"Compose message area"> <DIV><FONT color=3D#000080 face=3DGaramond>I'm still confused about = sqlite and=20 unicode, utf-8, text_factory=3Dstr etc. etc.</FONT></DIV> <DIV><FONT color=3D#000080 face=3DGaramond></FONT> </DIV> <DIV><FONT color=3D#000080 face=3DGaramond>All my python programs have = been=20 written to be utf-8 compatible ie. data is read-in and written-out = as utf-8=20 and hopefully ( :) ) python processes the data as utf-8 within the=20 program.</FONT></DIV> <DIV><FONT color=3D#000080 face=3DGaramond></FONT> </DIV> <DIV><FONT color=3D#000080 face=3DGaramond>Wrt sqlite, when the db is = initialized=20 and opened, text-factory=3Dstr is set in order to stop = "sqlite3.OperationalError:=20 Could not decode to UTF-8 column 'j' with text" errors.</FONT></DIV> <DIV><FONT color=3D#000080 face=3DGaramond></FONT> </DIV> <DIV><FONT color=3D#000080 face=3DGaramond>But, I'm still not confident = that I fully=20 understand what is going on wrt python, sqlite and utf-8. All = explanations=20 from basics would be really appreciated.</DIV> <DIV><BR></DIV></FONT> <DIV><FONT color=3D#000080 face=3DGaramond>Dinesh</FONT></DIV> <DIV><FONT color=3D#000080 face=3DGaramond></FONT> </DIV> <DIV><FONT color=3D#000080 face=3DGaramond></FONT> </DIV> <DIV><FONT color=3D#000080 face=3DGaramond><FONT=20 color=3D#000000>---------------------------------------------------------= -------------<BR>Message:=20 1<BR>Date: Thu, 25 Jun 2009 19:54:36 +0200<BR>From: Gerhard H?ring = <</FONT><A=20 href=3D"">[email protected]</A><FONT color=3D#000000>><BR>Subject: Re: = [pysqlite]=20 Encoding for connect to SqLite3<BR>To: PY-Delens <</FONT><A=20 href=3D"">[email protected]</A><FONT color=3D#000000>><BR>Cc: "About = pysqlite and=20 APSW."<BR><</FONT><A=20 href=3D"">list-pysqlite-FR6EJeJVuqdwc357pe9rcyQmJico6nz3epZhswDD4dQ@public.gmane.org</A><FONT=20 color=3D#000000>><BR>Message-ID: <</FONT><A=20 href=3D"">[email protected]</A><FONT = color=3D#000000>><BR>Content-Type:=20 text/plain; charset=3DISO-8859-1<BR><BR>Hello,<BR><BR>I'll cc the = mailing list.=20 Please direct future questions there.<BR><BR>PY-Delens wrote:<BR>>=20 Bonjour,<BR>> <BR>> I fail to display correctly accentuated = characters=20 from aSqLite 3 table<BR>> (stored in UTF-8 after abandoned UTF-16=20 scenario).<BR>> I try to add parameter _/encoding /_to thre PySqlite = connect=20 function,<BR>> but this is refused: [...]<BR><BR>No surprise. = pysqlite=20 doesn't have an encoding parameter anywhere. To<BR>store text that is = not UTF8=20 encoded, you can achieve that using your<BR>custom text_factory=20 (See<BR></FONT><A=20 href=3D"">http://oss.itsystementwicklung.de/download/pysqlite/doc/sqlite3= .html</A><FONT=20 color=3D#000000>).<BR><BR>> I'm alo confused on the libs supposed to = be called=20 in PySqlite; probably<BR>> one is nested into the other:<BR>>=20 pysqlite2<BR>> Sqlite3 [...]<BR><BR>pysqlite2 is the Python wrapper = for the=20 SQLite 3 library, which is the<BR>embedded database engine. If you use = Python=20 2.5 or later, a version of<BR>pysqlite2 is available under the module = name=20 "sqlite3" as well.<BR><BR>> How should I do? How to handle the = encoding=20 problem at:<BR>> - Python module=20 level<BR>> - SQL string=20 level<BR>> - DATA in SqLite file (I was = advised to go=20 back to UTF-8, by people<BR>> of DbManager =20 tools)<BR>> ?<BR><BR>I'd recommend to use = UTF8 only.=20 If you don't, you'll need to use<BR>bytestrings in your encoding as = parameters=20 to execute(). And set a<BR>text_factory that converts them back. If you = just=20 want the same<BR>bytestrings back (instead of, say, unicode string = objects), you=20 can just<BR>use text_factory=3Dstr.<BR><BR>--=20 Gerhard</FONT><BR><BR></FONT></DIV></BODY></HTML> ------=_NextPart_000_002D_01C9F641.0E4CEF80-- --===============1272235865== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ list-pysqlite mailing list list-pysqlite-FR6EJeJVuqdwc357pe9rcyQmJico6nz3epZhswDD4dQ@public.gmane.org http://itsystementwicklung.de/cgi-bin/mailman/listinfo/list-pysqlite --===============1272235865==--