Running tracker import with Python3 crashes with utf-8 error

Hi everybody:

I set up my original trackers with dbm under python 2. Sadly they will
not work with python 3 as dbhash support was apparently dropped in
python 3.0.

So I set up to convert (and move to sqlite as the back end) with:

    PYTHONPATH="" python2 roundup/scripts/roundup_admin.py -i demo export export

this completed without errors. Then I ran:

   PYTHONPATH="" python3 -mpdb roundup/scripts/roundup_admin.py -i demo import export

It got to where it was loading files (after loading messages, users
...). Then it threw a UTF-8 decoding error. Adding some print
statements to hyperdb.py in the file import path I found out it was a
jpeg image that was causing the issue. I set a breakpoint to start
when the file was loaded.

It looks like this code roundup/hyperdb.py:1669 FileClass::import_files

    mime_type = None
    props = self.getprops()
    if 'type' in props:
        mime_type = self.get(nodeid, 'type')
    if not mime_type:
        mime_type = self.default_mime_type
    if props['content'].indexme:
        self.db.indexer.add_text((self.classname, nodeid, 'content'),
            self.get(nodeid, 'content'), mime_type)

the mime type was set to image/jpeg but props.['content'].indexme was
set to true. So it tried to index the image. That's where the
string/UTF conversion and crash happened. So I have three questions:

  1) How does props['content'].indexme become True for an image?
  
  2) Shouldn't indexing depend on the mime type? No sense in indexing
    an image/jpeg. Are we missing a filter on mime-type? Should
    .indexme have been set to False in self.getprops().
    
  3) Should the call to add_text be in a try block that ignores text
     conversion errors so attempts to index the un-indexable won't
     make a tracker unrestorable?

For the record I got past this by invoking the debugger and setting
props['content'].indexme to False interactively. That seemed to be the
only file (despite a lot of other binary data) causing an issue.

I wonder what will happen if I try to reindex?

Quips, comments, evasions, questions and answers welcome.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.