Re: mysqlIndexerTest.test_unicode fails for me

Hi Ralf:

Thanks for your work on this. I am sorry I can't really help. It's
well out of my depth.

In message <[email protected]>,
Ralf Schlatterbeck writes:
>I've now pushed the change after all tests have passed.

I just pushed your changes to travis. Christof can you push to circleci.

>Note that the new behaviour works for the following cases (in python2):
>- If the new mysql_charset option is set to utf8 (the default)
>  everything, including generation of new databases works (at least
>  that's what the existing tests tell us)
>- When setting the value to "default" no explicit encoding option is
>  passed to mysql. So mysql happily accepts and returns python2 strings.
>  This also means that if the backend happens to be latin1, we will
>  write utf-8 to it and retrieve it back as utf-8. Such legacy
>  databases should be converted as soon as possible
>- Using python2 it is probably *not* possible to use an *explicit*
>  setting of the mysql_charset option other than the two values above.
>  If this would be set, e.g., to latin1, we would still write utf-8 to
>  it (using python2 strings encoded as utf-8). The database would return
>  unicode decoded from the db with the encoding specified for the DB,
>  latin1 in this example. We would decode it as utf-8. So we would get
>  garbage out if mysql really returns unicode objects in that case (not
>  tested). This is a classic double-encoding case.

Earlier you said:

   I've now changed the String entry in sql_to_hyperdb_value in
   backends/rdbms_common.py from 'str' to 'us2s'.

is this what caused the double encoding? I assume not since it's
converting 's' to 's' for python2 right?

It would be better to allow an upgrade to 2.0.0 using python2 to not
require a dump/restore. We really need an 2.0.0 upgrade mechanism that
doesn't require somebody taking the tracker offline possibly for hours
to do a dump/restore. When converting to python 3.x it's not great to
need this change, but justifyable.

>Since the DB interface for python3 is unicode (python3 str), everything
>is fine with python3.
>
>Now that I understand the whole thing a little better: let me note that
>legacy databases can probably converted by dumping them to sql text
>format, editing the dump to change the charset to utf-8 and
>re-importing.

Does that work only because utf-8 is a superset of latin1? I assume
this will only work for latin1 encodings.

How does dump/restore differ from using:

  ALTER DATABASE databasename CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
  ALTER TABLE tablename CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Also any idea what this means if somebody is supporting asian
languages in mysql? I know we have had a few bugs reported handling
Chinese for example.

If we can't use the database schema mutability/upgrade code to do this
conversion (grep for _version_ in roundup/backends/back_mysql.py), can
you write up the sequence required to do the conversion. We can review
it here or in upgrading.txt.

Maybe we should post a notice of what is hapening on the roundup-users
list and see what effect this will have on uptake of 2.0.0.

Thoughts?

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.