Re: mysqlIndexerTest.test_unicode fails for me

On Thu, Apr 30, 2020 at 10:30:06AM -0400, John P. Rouillard wrote:
> >The key thing to test is: create a database with older Roundup (e.g. 1.6) 
> >with Python 2 and MySQL (on a default MySQL installation, so with the 
> >latin1 default).  Create items in the database with string fields stored 
> >in the database (issue title, etc.) that contain Unicode characters that 
> >cannot be represented in latin1 (this works fine - the individual bytes of 
> >the UTF-8 representation are stored as if they were latin1 characters).  
> >Then try using that database with Roundup with your change, still with 
> >Python 2, and make sure that the Unicode characters previously put in 
> >those string fields still appear correctly (my expectation is that they 
> >won't, because MySQL will try to translate from latin1 to UTF-8).
> 
> This makes sense and it probably should remain latin1 for backwards
> compatibility. Maybe it's something that could be configured using
> config.ini [rdbms] key. But given the need to dump/restore when moving
> to python3 maybe it doesn't need to be configurable unless it needs to
> match the server's config.

I see, it probably involves more than just dump/restore because the
encoding is messed up.

You know that T-Shirt:
https://www.amazon.de/getshirts-Best-T-Shirt-Scheiss-Encoding/dp/B00T0WG0CG

But having the default for new mysql databases be whatever the user has
configured (and not utf-8) is imho not the right thing to do. So I vote
for a config-item in the rdbms-section (mysql_charset?)

> However Joseph I am confused by Ralf's issue. The test should create a
> new database and database connection. So latin1 should work for
> python2 and does work in the CI platforms. Yet it doesn't.

I somehow (and I've not yet found *where*) set the default for mysql to
latin1. The database that is created is *still* utf-8. The failing test
(which deals with unicode characters in the indexer) explicitly passes
utf-8 to the database in python2 (in py3 it passes unicode) it's using
u2s(w) for the word to be added in indexer_rdbms.py add_text. The
*query* using the find method relies on the database to do the correct
thing and passes unicode. In my case the conversion defaults to latin1.
Therefore for the german test nothing is found (because the utf-8
version in the db doesn't match the latin1 version created by mysql's
default encoding). For the russian test I'm getting a Coding exception
because the russian text is not representable in latin1.

About the latin1 configured in my system: As stated all the config-files
in /etc/mysql have utf-8. I don't have a local mysql config file for my
user. So I don't know yet where the latin1 setting comes from.

> Could my.cnf be overriding the connection encoding server side?
No, the server is utf-8, too. See one of my previous mails concerning
this where I've verified with sql commands that the database and the
tables really use utf-8 encoding.

> Forcing utf8 on the server side while the client uses
> latin1. Everything would work as long as the representations in latin1
> and ut8 were the same right?
I guess this was only possible with very old versions of mysql and the
python libs for it.

> So you could have a mysql instance using utf8 even under python2. It
> just depends on how the mysql server is configured.
Yes, all my dbs are using utf-8. And when I change the default to utf-8
(see my latest pushed commit) everything starts working. The difference
in roundups code is that we had forced utf-8 for py3 and not set a
charset parameter when using py2. When I removed the if for the python
version everything started working.

My proposed way forward would be to add a config-item for the database
encoding to the rdbms section. And come up with some text why you would
want to use it. I'm not sure, however that such a broken config would
continue to work with later versions of roundup and/or the mysql python
libs. At least for the indexer we *know* that it fails.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: [email protected]