Re: [pysqlite] more Unicode grief

[email protected] wrote:
>     Eric> The fact remains that I have this non-ASCII text I have to deal
>     Eric> with.  I have normalized all of this text into Unicode.  What do
>     Eric> form do I need to convert it into in order to have something that
>     Eric> can be stored, searched, and retrieved?
> 
> Was all the content in the SQLite database created through your application?
> It seems like you will have to go back and scrub the content in the database
> in some other fashion.
> 
the contents of the data was derived from external sources (e-mail messages). 
Since I don't trust databases any farther than I can throw them, I've preserved 
the metadata (json+unicode) separately.  This makes it relatively easy to 
reconstruct the database with different data formats.

I guess I could try something like converting all my unicode to utf-16 or worse 
comes to worse, as hexadecimal strings.  whatever I choose, I need a format 
where I can do substring searches.

---eric


-- 
Speech-recognition in use.  It makes mistakes, I correct some.