Re: Running tracker import with Python3 crashes with utf-8 error

On Wed, 13 Mar 2019, John P. Rouillard wrote:

>   1) How does props['content'].indexme become True for an image?

Content properties are set to be indexed by default (that's a function of 
the property, not of the file type for that property in a particular 
item).

If the property in question is meant for binary uploads and not for any 
kind of text it might make sense to index, setting indexme='no' on it 
makes sense (content=String(indexme='no') in a FileClass call in the 
schema), but that ought just to be an optimization; uploading a binary 
file to an indexed property should not cause an error.

>   2) Shouldn't indexing depend on the mime type? No sense in indexing
>     an image/jpeg. Are we missing a filter on mime-type? Should
>     .indexme have been set to False in self.getprops().

See my commit 1a0498c1ed90, "Avoid errors indexing binary uploads with 
Python 3." (which dealt with such issues for normal uploads, I didn't try 
this case for imports).  To quote the commit message (in part), "The 
indexing itself only happens for text/plain files, but that check is in 
the indexers themselves, after this error occurs".

Presumably you've found a different path to indexing which also needs a 
similar fix.

>   3) Should the call to add_text be in a try block that ignores text
>      conversion errors so attempts to index the un-indexable won't
>      make a tracker unrestorable?

I used

index_content = content
if bytes != str and isinstance(content, bytes):
    index_content = content.decode('utf-8', errors='ignore')

in that commit, with the errors='ignore' to avoid errors at that point.

-- 
Joseph S. Myers
[email protected]