Re: Running tracker import with Python3 crashes with utf-8 error

Hi Joseph:

In message <[email protected]>,
Joseph Myers writes:
>On Wed, 13 Mar 2019, John P. Rouillard wrote:

Thanks for clearing up my first two questions.

>Presumably you've found a different path to indexing which also needs a 
>similar fix.
>
>>   3) Should the call to add_text be in a try block that ignores text
>>      conversion errors so attempts to index the un-indexable won't
>>      make a tracker unrestorable?
>
>I used
>
>index_content = content
>if bytes != str and isinstance(content, bytes):
>    index_content = content.decode('utf-8', errors='ignore')
>
>in that commit, with the errors='ignore' to avoid errors at that point.

I did a similar modification in hyperdb.py:FileClass::import_files():

         if not mime_type:
             mime_type = self.default_mime_type
         if props['content'].indexme:
+            index_content = self.get(nodeid, 'content')  (** 1 **)
+            if bytes != str and isinstance(index_content, bytes):
+                index_content = index_content.decode('utf-8', errors='ignore')
             self.db.indexer.add_text((self.classname, nodeid, 'content'),
-                self.get(nodeid, 'content'), mime_type)
+                index_content, mime_type)

As you see, content has to come from the object using self.get and
that's where it's failing.

Traceback (most recent call last):
  File "roundup/admin.py", line 1537, in run_command
    ret = function(args[1:])
  File "roundup/admin.py", line 1309, in do_import
    cl.import_files(dir, nodeid)
  File "roundup/hyperdb.py", line 1685, in import_files
    index_content = self.get(nodeid, 'content')
  File "roundup/backends/rdbms_common.py", line 3077, in get
    return b2s(self.db.getfile(self.classname, nodeid, None))
  File "roundup/anypy/strings.py", line 19, in b2s
    return b.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

The code in question is:

        if propname == 'content':
            try:
		#       v---- blamo here --- 
                return b2s(self.db.getfile(self.classname, nodeid, None))
            except IOError as strerror:
                # BUG: by catching this we donot see an error in the log.
                return 'ERROR reading file: %s%s\n%s\n%s'%(
                        self.classname, nodeid, poss_msg, strerror)
        elif propname == 'binary_content':
            return self.db.getfile(self.classname, nodeid, None)

should I be getting 'binary_content' for every file rather than
'content' at ** 1 ** and handle the conversion in hyperdb? Is the
whole purpose of binary_content to allow passing through the data in a
file unmolested?

Comments?

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.