Re: createrepo/utils.py

Toshio Kuratomi <[email protected]> Wed, 16 Apr 2008 13:23:36 -0700

Newsgroups	gmane.linux.rpm.metadata
Message-ID	<[email protected]>

James Antill wrote:
> On Wed, 2008-04-16 at 12:16 -0400, Luke Macken wrote:
> 
>> Ok, so it looks like we're losing here.
>>
>> This utf8String method seems to be a bit misleading, and full of pain.  I assume we
>> want to give it a utf-8 encoded string, and get back a unicode object, right?
> 
>  See my later patch, that is probably less mis-leading?
> 
>  In the caller, that we are having problems with, we want to give it a
> str() from RPM (which may or may not be utf8) and get a valid utf8 str()
> object back _that is also valid inside an XML document_ (excepting
> random < > & bytes, which get converted).
> 
I took a look and libxml2 has a bug:  When creating an xml document, it 
should be removing control characters as they are not valid xml.  Does 
Daniel Veillard know about this (I know you talked to him about reading 
which he's right about but I don't know about writing.)

Doing this in utf8String() instead of libxml2 is certainly a valid 
workaround.

>  The big problem here being that a bunch of the "small bytes" like 0x01
> are valid utf8 but aren't valid XML data. Hence the patches.
> 
>  After looking again, it's now obvious that we still screw up if we pass
> a unicode() object in that has 0x01 bytes in it ... so we should
> probably fix that too (although I'm not sure if that's possible).
> 
Yeah.  Forgive me for saying this, but utf8String() is a bit crazy :-)

I'm attaching a saner version.

One note: This function should be split in two.  When you read from the 
rpm information, you should pass it to a unicodeString() function that 
performs the conversion into a unicode string.  When you output the 
value to libxml2 or another place that doesn't understand python unicode 
strings you pass that unicode string to the much smaller utf8String() 
which strips the control codes and changes it to a utf-8 encoded byte 
string.  That way everything you operate on within your program is a 
unicode string.  You only encode it to a byte representation when you 
output.

P.S. Mike Bonnet notes that iso-8859-15 covers every one byte value so 
no encoding mentioned after it will ever get called.  I didn't change 
that as I don't know how you want to address that.

-Toshio

_______________________________________________
Rpm-metadata mailing list
[email protected]
https://lists.dulug.duke.edu/mailman/listinfo/rpm-metadata

createrepo-utf8string-fix.patch (text/x-patch, 2.5 KB)

diff --git a/createrepo/utils.py b/createrepo/utils.py
index 1dc3b0c..1a50985 100644
--- a/createrepo/utils.py
+++ b/createrepo/utils.py
@@ -74,39 +74,34 @@ def returnFD(filename):
     return fdno
 
 def utf8String(string):
-    """hands back a unicoded string"""
-    if string is None:
+    """hands back a utf8 encoded byte string"""
+    if not string:
+        # Small optimization
         return ''
-    elif isinstance(string, unicode):    
-        return string
-    du = False
-    try:
-        x = unicode(string, 'ascii')
-        du = True
-    except UnicodeError:
-        encodings = ['utf-8', 'iso-8859-1', 'iso-8859-15', 'iso-8859-2']
-        for enc in encodings:
-            try:
-                x = unicode(string, enc)
-            except UnicodeError:
-                pass
-            else:
-                if x.encode(enc) == string:
-                    return x.encode('utf-8')
-    newstring = ''
-    # Kill bytes (or libxml will die) not in the small byte portion of:
-    #  http://www.w3.org/TR/REC-xml/#NT-Char
-    # we allow high bytes, if it passed the utf8 check above. Eg.
-    # good chars = #x9 | #xA | #xD | [#x20-...]
-    bad_small_bytes = range(0, 8) + [11, 12] + range(14, 32)
-    for char in string:
-        if ord(char) in bad_small_bytes:
-            pass # Just ignore these bytes...
-        elif not du and ord(char) > 127:
-            newstring = newstring + '?'
-        else:
-            newstring = newstring + char
-    return newstring
+    elif not isinstance(string, unicode):
+        # By the end of this block, have a unicode string
+        try:
+            string = unicode(string, 'utf-8')
+        except UnicodeError:
+            # Try several encodings
+            encodings = ('iso-8859-1', 'iso-8859-15', 'iso-8859-2')
+            for enc in encodings:
+                try:
+                    string = unicode(string, enc)
+                    break
+                except UnicodeError:
+                    continue
+            if not isinstance(string, unicode):
+                string = unicode(string, 'utf-8', 'replace')
+
+    # Replace control characters with '?'.  These are valid unicode and
+    # ASCII but not valid xml.
+    control_codes = range(0, 8) + [11, 12] + range(14, 32)
+    control_table = dict(zip(control_codes, u'?' * len(control_codes)))
+    string = string.translate(control_table)
+
+    # Finally return a utf-8 encoded byte string representation of the unicode
+    return string.encode('utf-8')
 
 def checkAndMakeDir(dir):
     """

signature.asc (application/pgp-signature, 189 B)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFIBmBWX6yAic2E7kgRAhQzAJ9ReoKZvhxyzaaTW9UXDoVKM7WZawCgpV+1
ElX0a5G7dO3knQPbKtU/ql4=
=fRAI
-----END PGP SIGNATURE-----