Re: mysqlIndexerTest.test_unicode fails for me

On Mon, May 04, 2020 at 12:21:48PM +0200, Ralf Schlatterbeck wrote:
> OK, looks like mysql returns a unicode object if you specify a charset
> during database connect. I've now changed the String entry in
> sql_to_hyperdb_value in backends/rdbms_common.py from 'str' to 'us2s'.
> 
> This converts a unicode value returned from the db to an utf-8 encoded
> string for py2.
> 
> I'm in the process of running *all* tests before pushing that fix. Don't
> hold your breath.

I've now pushed the change after all tests have passed.

Note that the new behaviour works for the following cases (in python2):
- If the new mysql_charset option is set to utf8 (the default)
  everything, including generation of new databases works (at least
  that's what the existing tests tell us)
- When setting the value to "default" no explicit encoding option is
  passed to mysql. So mysql happily accepts and returns python2 strings.
  This also means that if the backend happens to be latin1, we will
  write utf-8 to it and retrieve it back as utf-8. Such legacy
  databases should be converted as soon as possible
- Using python2 it is probably *not* possible to use an *explicit*
  setting of the mysql_charset option other than the two values above.
  If this would be set, e.g., to latin1, we would still write utf-8 to
  it (using python2 strings encoded as utf-8). The database would return
  unicode decoded from the db with the encoding specified for the DB,
  latin1 in this example. We would decode it as utf-8. So we would get
  garbage out if mysql really returns unicode objects in that case (not
  tested). This is a classic double-encoding case.

Since the DB interface for python3 is unicode (python3 str), everything
is fine with python3.

Now that I understand the whole thing a little better: let me note that
legacy databases can probably converted by dumping them to sql text
format, editing the dump to change the charset to utf-8 and
re-importing.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: [email protected]