Re: [pysqlite] Encoding for connect to SqLite3
P-Y Delens <[email protected]> Fri, 26 Jun 2009 17:13:45 +0200
| Newsgroups | gmane.comp.python.db.pysqlite.user |
|---|---|
| Message-ID | <[email protected]> |
--===============1330443959== Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#ffffff" text="#000000"> <font face="Helvetica, Arial, sans-serif">Dear </font>Mr Häring,<br> <br> Thanks for your answer.<br> <font face="Helvetica, Arial, sans-serif"><br> Please see answer/further questioning in your text:<br> </font><b>1. Users list</b><br> <b>2. documentation<br> </b><b>3. 'encode' parameter in connect function<br> </b><b>4 Libs<br> </b><b>5. Global comprehension of coding <u>in global processing flow</u>: u'XXX'</b><br> <font face="Helvetica, Arial, sans-serif"><br> Thank you on forward.<br> <br> Yours,</font><br> <pre class="moz-signature" cols="50">__________________________________________________________________ Pierre-Yves Delens __________________________________________________________________</pre> Gerhard Häring wrote: <blockquote cite="mid:[email protected]" type="cite"> <pre wrap="">Hello, I'll cc the mailing list. Please direct future questions there. </pre> </blockquote> <b>1. Users list</b><br> I cc to <a class="moz-txt-link-abbreviated" href="mailto:[email protected]">[email protected]</a><br> I see <a class="moz-txt-link-rfc2396E" href="mailto:list-pysqlite-FR6EJeJVuqdwc357pe9rcyQmJico6nz3epZhswDD4dQ@public.gmane.org">"list-pysqlite-FR6EJeJVuqdwc357pe9rcyQmJico6nz3epZhswDD4dQ@public.gmane.org"</a> on your mail. Is it the same list, should I subscribe to this second list<br> <blockquote cite="mid:[email protected]" type="cite"> <pre wrap="">PY-Delens wrote: </pre> <blockquote type="cite"> <pre wrap="">Bonjour, I fail to display correctly accentuated characters from aSqLite 3 table (stored in UTF-8 after abandoned UTF-16 scenario). I try to add parameter _/encoding /_to thre PySqlite connect function, but this is refused: [...] </pre> </blockquote> <pre wrap=""><!----> No surprise. pysqlite doesn't have an encoding parameter anywhere. To store text that is not UTF8 encoded, you can achieve that using your custom text_factory (See <a class="moz-txt-link-freetext" href="http://oss.itsystementwicklung.de/download/pysqlite/doc/sqlite3.html">http://oss.itsystementwicklung.de/download/pysqlite/doc/sqlite3.html</a>). </pre> </blockquote> <b>2. documentation</b><br> I was mislead by the 'pysqlite2 documentation: ' on page <a class="moz-txt-link-freetext" href="http://oss.itsystementwicklung.de/trac/pysqlite/">http://oss.itsystementwicklung.de/trac/pysqlite/</a><br> (out of date)<br> Is there more doc available, than the one pointed by you, here above ?<br> <b><br> 3. 'encode' parameter in connect function</b><br> (sorry: not 'encoding')<br> I found this in documentation, perhaps out of date (pySqlite Usage manual for pySqLite 2.4.0). I discover via your mail a fresher documentation.<br> But I was confused because when asking for<br> >>> dir (pysqlite2.dbapi2.connect.__module__)<br> I get well 'encode' attribute in the dir.<br> But test is refusing the keyword<br> <blockquote>>>> from pysqlite2 import dbapi2 as Sqlite<br> >>> cx = Sqlite.connect('Z:\\72_BasesDeDonnees_GestionBureau\\SqLite-CAO\\SqLite-CAO-1b.db', encode="UTF-8")<br> Traceback (most recent call last):<br> File "<pyshell#16>", line 1, in <module><br> cx = Sqlite.connect('Z:\\72_BasesDeDonnees_GestionBureau\\SqLite-CAO\\SqLite-CAO-1b.db', encode="UTF-8")<br> TypeError: 'encode' is an invalid keyword argument for this function<br> >>> <br> </blockquote> As indeed the doc you pointed to doesn't mention 'encode' any more, where is this feature gone up? Or why wasit decided not necessary any more? (because SqLite 3 supposed alqways serving UTF-8?)<br> <blockquote cite="mid:[email protected]" type="cite"> <pre wrap=""> </pre> <blockquote type="cite"> <pre wrap="">I'm alo confused on the libs supposed to be called in PySqlite; probably one is nested into the other: pysqlite2 Sqlite3 [...] </pre> </blockquote> <pre wrap=""><!----> pysqlite2 is the Python wrapper for the SQLite 3 library, which is the embedded database engine. If you use Python 2.5 or later, a version of pysqlite2 is available under the module name "sqlite3" as well. </pre> </blockquote> <b>4 Libs</b><br> <u><b>4a.</b></u> Do you mean that C:\Python25\Lib\sqlite3 is in the Python distribution (not added at pySQLite istall), self-sufficient as engine?<br> and that C:\Python25\Lib\site-packages\pysqlite2 is making use of sqLiter3 lib ?<br> <br> <b><u>4b.</u></b> ? Is there a difference between<br> import sqlite3<br> from pysqlite2 import dbapi2 as Sqlite <br> from pysqlite2 import _sqlite as Sqlite <br> What is the best way of coding? <br> <blockquote cite="mid:[email protected]" type="cite"> <pre wrap=""> </pre> <blockquote type="cite"> <pre wrap="">How should I do? How to handle the encoding problem at: - Python module level - SQL string level - DATA in SqLite file (I was advised to go back to UTF-8, by people of DbManager tools) ? </pre> </blockquote> <pre wrap=""><!----> I'd recommend to use UTF8 only. If you don't, you'll need to use bytestrings in your encoding as parameters to execute(). And set a text_factory that converts them back. If you just want the same bytestrings back (instead of, say, unicode string objects), you can just use text_factory=str. </pre> </blockquote> <b>5. Global comprehension of coding in global processing flow</b><br> This is my flow:<br> see STEP 4 : there is the main question, I think.<br> I give all my steps as to make sure you get the context/goal.<br> So that I won't miss interferences (or no interference ...) between steps.<br> <br> <table border="1" cellpadding="2" cellspacing="0" width="100%"> <tbody> <tr> <td align="center" valign="top"><b><br> </b></td> <td align="center" valign="top"><b>Step- Context - File concerned<br> </b></td> <td align="center" valign="top"><b>coding <br> </b></td> <td align="center" valign="top"><b>How controlled<br> </b></td> <td align="center" valign="top"><b>RMQ, question to support<br> </b></td> </tr> <tr> <td>1 <br> </td> <td valign="top">SQLite 3<br> </td> <td align="center" valign="top">UTF-8</td> <td valign="top">Option at Db creation time</td> <td valign="top"><br> </td> </tr> <tr> <td>2<br> </td> <td valign="top"><u>Python code</u> for accessing Db,<br> creating a Python 'table' as a 2Dim List<br> (code in my <i>FileLib.py</i>)<br> </td> <td align="center" valign="top">cp1252 </td> <td valign="top">header =<br> #-*- coding: cp1252 -*-<br> </td> <td valign="top">cp1252 is oftrn dised for workng on Windows Europe.<br> Should I change this here?<br> </td> </tr> <tr> <td>3 <br> </td> <td valign="top">SqlStatementString in Python code<br> </td> <td align="center" valign="top">? cp1252 </td> <td valign="top">? inherits from step 2 ?<br> </td> <td valign="top">Should I encode u'SqlString', or use text-Factory?<br> I think I may just leave palin stering?<br> Or should I encode the SqlStatementString to UTF-8?<br> </td> </tr> <tr> <td valign="top"><b>4<br> </b></td> <td valign="top"><b>Datas as accessed/retrieved by pySqlite<br> </b></td> <td align="center" valign="top"><b>? UTF-8<br> or<br> Unicode<br> </b></td> <td valign="top"><b>Controlled by step1 above<br> </b></td> <td valign="top"><b>The list /tuple items are displayed with the u'XXX' so I understand at this stage I got a Unicode encoded string but from a wrongly interpreted character.<br> I didn't explicitely ask for u'XXX'.<br> ==><big> What to do?<small><br> . ask for u'XXX'<small><big>, or not?<br> . ask for u'XXX', but after decoding the Slite3 data in a proper way?<br> <br> </big></small></small></big>Please keep in mind the final format ISO-8859-1 (see step 8 below)<br> </b></td> </tr> <tr> <td valign="top">5<br> </td> <td valign="top">Datas as stored in lists by pySqlite</td> <td align="center" valign="top">? UTF-8</td> <td valign="top">Influenced by ?<br> </td> <td valign="top">- Influenced by step1 (the Db)?<br> - Influenced by step2 (the Python header)?<br> - Influenced by ... ?<br> result is with such problems: <br> <i>u'Ar100810EV_$G-O_Ma<font color="#ff0000">?</font>nPrmnt' </i> <small>instead of </small><br> <i>u'Ar100810EV_$G-O_Ma<u><font color="#009900">çon</font></u>nPrmnt'</i><br> <i>u'Ar20----__-$PAREMENT FA?DES ET ?ANCH?T</i><br> <small>instead of</small> <i>...FADES ET éANCHéiTés...</i></td> </tr> <tr> <td valign="top">6<br> </td> <td valign="top">Cheetah template : <br> <div align="center"><i>FileTPL.tmpl</i></div> </td> <td align="center" valign="top">cp1252 </td> <td valign="top"># -*- coding: cp1252 -*- <br> </td> <td valign="top">coherent with step 2 (Python moduled called by Template)<br> </td> </tr> <tr> <td valign="top">7<br> </td> <td align="center" valign="top"> <div align="left">Cheetah compiled/filled servlet or generator :<br> </div> <i>FileTPL.py</i><br> <div align="left">this code is calling upon <i> FileLib.py</i> <br> (step 2 above)<br> </div> </td> <td align="center" valign="top"><br> </td> <td valign="top"><br> </td> <td valign="top">N.B : <br> </td> </tr> <tr> <td valign="top">8<br> </td> <td valign="top">XHTML generated page<br> </td> <td align="center" valign="top">ISO-8859-1<br> </td> <td valign="top">- xml declaration<br> - meta content<br> </td> <td valign="top">- global poption for a big static WebSite<br> - I would hardly imagine to move from this option.<br> - Should I?<br> </td> </tr> </tbody> </table> <br> <br> <blockquote cite="mid:[email protected]" type="cite"> <pre wrap="">-- Gerhard </pre> </blockquote> </body> </html> --===============1330443959== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ list-pysqlite mailing list list-pysqlite-FR6EJeJVuqdwc357pe9rcyQmJico6nz3epZhswDD4dQ@public.gmane.org http://itsystementwicklung.de/cgi-bin/mailman/listinfo/list-pysqlite --===============1330443959==--