Re: [pysqlite] Avoid conversion to unicode for APSW
Roger Binns <[email protected]>
| Newsgroups | gmane.comp.python.db.pysqlite.user |
|---|---|
| Message-ID | <[email protected]> |
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nemanja Čorlija wrote: > I don't think SQLite says that strings *are* utf-8. The APIs do! See http://www.sqlite.org/c3ref/bind_blob.html with the list at the bottom which talks about characters. If you don't know the encoding then they are bytes not characters. The API for sqlite3_result_text also explicitly says UTF-8, while sqlite3_column_text strongly suggests it. > I think behavior is intentional, Behind the scenes DRH decided not to check the input data so it was accepting invalid data. And then it became too late to change behaviour. Unfortunately this happens a lot in SQLite such as http://www.sqlite.org/cvstrac/tktview?tn=3037 or the embedded zeroes issue I just posted about. > SQLite is not particular about the text it receives and is more than > happy to process text strings that are not normalized or even > well-formed UTF-8 or UTF-16. Thus, programmers who want to store > IS08859 data can do so using the UTF-8 interfaces. As long as no > attempts are made to use a UTF-16 collating sequence or SQL function, > the byte sequence of the text will not be modified in any way. Basically it is saying you can treat blobs as strings providing you never try to treat them as strings. Ironically in the embedded zeroes issue DRH says to treat strings as blobs! This raises an issue for wrapper authors. Since SQLite doesn't do much validation, should we do validation or let people fall into traps because they haven't read all the fine print. ie should the wrapper be proactive or reactive? I have generally taken the proactive approach. For example APSW enforced the threading issues while SQLite required them. Similarly APSW won't let you put non-unicode data into the database in the first place. Gerhard has been more lenient letting you do whatever you want, but also with optional "protective" functionality you can turn on. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFIBTKXmOOfHg372QQRAhrUAKCgiq8nrWQy9KJs6W/wOGCw4PcP5QCg1ZdJ iPbrJIrr6aeocFerBQvpjxk= =u0OC -----END PGP SIGNATURE----- _______________________________________________ list-pysqlite mailing list [email protected] http://itsystementwicklung.de/cgi-bin/mailman/listinfo/list-pysqlite