Re: mysqlIndexerTest.test_unicode fails for me

I've debugged this a little further:

I *do* have a non-standard configuration in
/etc/mysql/mariadb.conf.d/50-server.cnf:
#character-set-server  = utf8mb4
#collation-server      = utf8mb4_general_ci
character-set-server  = utf8
collation-server      = utf8_general_ci

and in /etc/mysql/mariadb.conf.d/50-client.cnf:
#default-character-set = utf8mb4
default-character-set = utf8

The commented lines are debian default.

With both, python2 and python3 I'm getting for the database encoding and
the relevant tables (I'm stopping in the debugger *after* the
self.dex.add_text calls in test_unicode)

(Pdb) p self.db.sql ('SELECT default_character_set_name FROM information_schema.SCHEMATA WHERE schema_name = "rounduptest"')
None
(Pdb) p self.db.sql_fetchall()
(('utf8',),)

(Pdb) p self.db.sql ('show tables')
None
(Pdb) p self.db.sql_fetchall()
(('__textids',), ('__words',), ('ids',), ('otks',), ('schema',), ('sessions',))

(Pdb) p self.db.sql ('SELECT CCSA.character_set_name FROM information_schema.`TABLES` T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA WHERE CCSA.collation_name = T.table_collation AND T.table_schema = "rounduptest" AND T.table_name = "__words"')
None
(Pdb) p self.db.sql_fetchall()
(('utf8',),)

(Pdb) p self.db.sql ('SELECT CCSA.character_set_name FROM information_schema.`TABLES` T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA WHERE CCSA.collation_name = T.table_collation AND T.table_schema = "rounduptest" AND T.table_name = "__textids"')
(Pdb) p self.db.sql_fetchall()
(('utf8',),)

Python2 and python3 differ in what is in the __words table but the
difference may be due to different representation returned by the two
implementations? It seems in case of python3, unicode is returned
directly, but in case of python2 we see utf-8 returned from the
database, this may be due to different backend implementations in
python.

Python2:
(Pdb) p self.db.sql('select * from __words')
None
(Pdb) p self.db.sql_fetchall()
(('SPR\xc3\x9cNGE', 1L), ('\xd0\xa2\xd0\x95\xd0\x9a\xd0\xa1\xd0\xa2SPR\xc3\x9cNGE', 2L), ('SPR\xc3\x9cNGE', 2L), ('\xd0\xa0\xd0\xa3\xd0\xa1\xd0\xa1\xd0\x9a\xd0\x98\xd0\x99', 2L))

Python3:
(Pdb) p self.db.sql('select * from __words')
None
(Pdb) p self.db.sql_fetchall()
(('SPRÜNGE', 1), ('SPRÜNGE', 2), ('РУССКИЙ', 2), ('ТЕКСТSPRÜNGE', 2))

When decoding the results from the Python2 version (in py3) the result
is at least valid utf-8 and decodes to the same values as in Python3:

>>> b'SPR\xc3\x9cNGE'.decode('utf-8')
'SPRÜNGE'
>>> b'\xd0\xa2\xd0\x95\xd0\x9a\xd0\xa1\xd0\xa2SPR\xc3\x9cNGE'.decode('utf-8')
'ТЕКСТSPRÜNGE'
>>> b'\xd0\xa0\xd0\xa3\xd0\xa1\xd0\xa1\xd0\x9a\xd0\x98\xd0\x99'.decode('utf-8')
'РУССКИЙ'

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: [email protected]


_______________________________________________
Roundup-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/roundup-devel