Re: UnicodeDecodeError

Hi Frank, hi Toby,
>>I'm using dirstorage with zope3. Sometimes it fails with an error like this
>>one:
>>
>>File "/SCR/zope3-svn/instance/lib/python/DirectoryStorage/Full.py", line
>>187, in _vote_impl
>>temp=struct.pack("!HHHIH",len(u),len(d),len(e),len(ob),0) + u + d + e + ob
>>UnicodeDecodeError: 'ascii' codec can't decode byte 0x99 in position 5:
>>ordinal not in range(128)
>>
>>What can I do to fix that?
> 
> Im not sure. You may have to do some debugging here.

Yes. It would especially be helpful to see the full tracebacks whenever
this error occurs; basically, we'd need to see when and by what this
error is triggered. Maybe you can provide us with some more info here,
Frank.

> One of the variables referenced on that line must be a unicode object.
> 
> 'ob' seems unlikely to be the cause, because it calculated from a list of oids 
> a little earlier. 'e' seems unlikely too, because it is generated by pickling 
> extension data in BaseStorage.py
> 
> 'u' is the user name and 'd' is the transaction description. These seem likely 
> to be the cause. Im not sure exactly where these come from in Zope3, but I am 
> sure they shouldnt be unicode.

Unfortunately, that is not correct. Principal IDs (what used to be known
as user names in Zope 2) can very well be unicode strings, in fact, they
actually *should* be, as specified by the
zope.security.interfaces.IPrincipal interface (there 'id' is a TextLine
field, which means a unicode string w/o newline characters).

I'm not sure about transaction descriptions, but there's some code in
Zope 3 that would lead me to believe that there could also be unicode
values passed in as the transaction note.

In order to fix this, we can't just use the 'latin-1' encoding or even
'ascii' which is even more restrictive to turn the unicode into a
string. If at all, we should use UTF-8 to encode the unicode; this must
of course be done before we compute the len() of those objects. And then
we must also properly decode when we read back the body (I think in
lines 229 and 431).

Of course, we could also argue that the ZODB perhaps doesn't allow
unicode for either the transaction's user name or description and that
therefore Zope 3 should be fixed accordingly. There are two strong
reasons against that line of thinking: a) FileStorage doesn't seem to
have a problem with it and FileStorage is a certain benchmark when it
comes to functionality, b) if Zope 3 is changed to store encoded strings
instead of unicode, it would also have to deal with old transaction
pickles which complicates the whole issue even further.

Season's greetings all the way from China,

Philipp


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click