Inconsistent pylupdate5 behaviour on UTF8 data

Giuseppe Corbelli <[email protected]> Wed, 12 Feb 2020 16:27:09 +0100

Newsgroups	gmane.comp.python.pyqt-pykde
Message-ID	<[email protected]>

Hi all
I found a puzzling pylupdate5 behaviour inconsistency between Linux and 
Windows versions.
Scenario: I am extracting translatable strings from python modules. The 
files are saved as UTF8, I run pylupdate and get different 
representations in the XML output.

pylupdate5 v5.14.1 as Debian package on Linux and fresh pip install in a 
venv on Windows 10.

As you can find in the attached test data:

- on windows the 'ç' character (U+00E7	ç	c3 a7	LATIN SMALL LETTER C WITH 
CEDILLA) is converted to <source>this needs UTF8 encoding: 
&#xc3;&#xa7;&#xc2;&#xb0;&#xc2;&#xa7;</source>

- on linux the same 'ç' correctly converts to <source>this needs UTF8 
encoding: &#xe7;&#xb0;&#xa7;</source>

So it seems that on windows each byte of the utf8 string is replaced 
with its unicode point in xml numeric character format, while on linux 
the same applies (correctly) to the character itself (formed by two 
bytes in UTF8).

Am I doing something wrong?

Thanks
-- 
Giuseppe Corbelli

_______________________________________________
PyQt mailing list    [email protected]
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

it_IT.ts.linux (text/xml, 504 B)

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE TS><TS version="2.0">
<context>
    <name>@default</name>
    <message>
        <location filename="module.py" line="4"/>
        <source>this is pure ASCII</source>
        <translation type="unfinished"></translation>
    </message>
    <message>
        <location filename="module.py" line="5"/>
        <source>this needs UTF8 encoding: &#xe7;&#xb0;&#xa7;</source>
        <translation type="unfinished"></translation>
    </message>
</context>
</TS>

it_IT.ts.win32 (text/xml, 522 B)

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE TS><TS version="2.0">
<context>
    <name>@default</name>
    <message>
        <location filename="module.py" line="4"/>
        <source>this is pure ASCII</source>
        <translation type="unfinished"></translation>
    </message>
    <message>
        <location filename="module.py" line="5"/>
        <source>this needs UTF8 encoding: &#xc3;&#xa7;&#xc2;&#xb0;&#xc2;&#xa7;</source>
        <translation type="unfinished"></translation>
    </message>
</context>
</TS>

module.py (text/x-python, 107 B)

#!/usr/bin/python
# -*- coding: utf-8 -*-

tr("this is pure ASCII")
tr("this needs UTF8 encoding: ç°§")

test.pro (text/plain, 67 B)

CODECFORSRC = UTF-8

TRANSLATIONS = it_IT.ts

SOURCES = module.py