Re: [pysqlite] Trouble with create_function interface to sqlite

Roger Binns wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Eric S. Johansson wrote:
>> err um, utf-8 *is* unicode.
>>
>> http://en.wikipedia.org/wiki/UTF-8
> 
> UTF-8 is an encoding of Unicode.  It is a way of expressing Unicode as
> bytes.  The Unicode standard itself merely defines codepoints.  They
> look like U+0041 for the letter A.  In order to express the codepoints
> in byte form you need an encoding such as UTF-8, UCS-2, UTF-16, UCS-4 etc.
> 
> People do use Unicode and UTF-8 interchangeably since in many cases the
> distinction doesn't matter, but in the case of SQLite 3 it does.  SQLite
> 3 operates on Unicode strings and will accept and provide them in UTF-8
> and UTF-16 byte encodings in the various apis.  That is all transparent
> and independent of what encoding SQLite uses for the file.  But if you
> do provide invalid data, one of those behind the scenes conversions can
> later easily end up biting you as well as functionality such as
> functions and collations.

I would have said it is an encoding scheme expressing the Unicode concept but no 
matter.  As old fable goes "the elephant is soft and mushy"[1].  Personally, I 
have serious scar tissue with 8-bit text strings and converting them to any of 
the Unicode encodings.   At the beginning of the effort, all I cared about was 
shoving the text into storage with a type of UTF-8 and thinking I wouldn't care 
about the side effects.  Boy was I wrong.  I've learned probably way too much 
about encoding sequences, search order etc. etc.

for me the biggest problem was how the system would sometimes accept text as I 
declared it to be but throw up a hairball on trying to return the same text. 
This may have been more a problem with Python and my misunderstandings than SQL 
light but still, it was pretty obnoxious.


---eric

[1]http://en.wikipedia.org/wiki/Blind_Men_and_an_Elephant
look down at the miscellaneous section  to get the joke