Re: UnicodeDecodeError

On Wednesday 07 Dec 2005 15:16, Frank Burkhardt wrote:
> Hi,
>
> On Wed, Dec 07, 2005 at 01:23:06PM +0000, Toby Dickenson wrote:
> > On Tuesday 06 Dec 2005 14:39, Frank Burkhardt wrote:
> > > Hi,
> > >
> > > I'm using dirstorage with zope3. Sometimes it fails with an error like
> > > this one:
> > >
> > > File "/SCR/zope3-svn/instance/lib/python/DirectoryStorage/Full.py",
> > > line 187, in _vote_impl
> > > temp=struct.pack("!HHHIH",len(u),len(d),len(e),len(ob),0) + u + d + e +
> > > ob UnicodeDecodeError: 'ascii' codec can't decode byte 0x99 in position
> > > 5: ordinal not in range(128)
> > >
> > > What can I do to fix that?
> >
> > Im not sure. You may have to do some debugging here.
> >
> >
> > One of the variables referenced on that line must be a unicode object.
> >
> > 'ob' seems unlikely to be the cause, because it calculated from a list of
> > oids a little earlier. 'e' seems unlikely too, because it is generated by
> > pickling extension data in BaseStorage.py
> >
> > 'u' is the user name and 'd' is the transaction description. These seem
> > likely to be the cause. Im not sure exactly where these come from in
> > Zope3, but I am sure they shouldnt be unicode.
>
> I'm not really a python expert. How do I determine, if a variable is
> unicode or not? I used the debugger but type(ob,u,...) just told me, that
> those objects are of type "str".

I suggest something like:

    if type(u)!=type(''):
	raise ValueError('u is not a plain string. it is %r' % (u,) )
    if type(d)!=type(''):
	raise ValueError('d is not a plain string. it is %r' % (d,) )

> What do I do with a variable that is unicode but shouldn't - is there a way
> to convert it to ascii?

somestring.encode('latin1')

or, if you want unconvertable characters to be replaced with a question mark 
rather than raising an exception:

somestring.encode('latin1','replace')

you could also replace 'latin1' with 'ascii' if you *really* only want 
characters in the range 0..127.

But, I dont think thats what you want to be doing in this case. If you can 
identify where these unicode strings are coming from then I suggest fixing 
the problem at its origin.


ps. Please let me know if that type checking code turns out to be helpful or 
not. I am tempted to add that into the next release of DirectoryStorage
 








-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click