Re: Migrating to BTrees 4.x

> On Nov 19, 2016, at 17:58, Jim Fulton <[email protected]> wrote:
> 
> 
> 
> On Sat, Nov 19, 2016 at 3:21 PM, David Glick <[email protected]> wrote:
>> I'm working on updating Plone to support ZODB 4 and 5.
>> 
>> So far most of the blockers are because BTrees 4.x no longer allows using keys whose ordering is not well-defined, including None, and there are various places where Plone was using None as a key. I understand the reason for the restriction and am making progress on making Plone no longer do this.
>> 
> Hm. IMO, the restriction should be in adding items, not on unpickling.  We should go to great lengths, to avoid unpickling errors.
> 
> I think we should avoid raising this error in __setstate__. It should be easy to fix.

Interestingly, the Python implementation currently *doesn't* raise an error in this case, at least in small trees.

I pickled an OOBTree from ZODB3 3.10.7 that looked like {None: 42}. 

When I use the pure Python implementation, it loads:

  In [9]: import pickle

  In [10]: pickle.loads('ccopy_reg\n__newobj__\np0\n(cBTrees.OOBTree\nOOBTree\np1\ntp2\nRp3\n((((NI42\ntp4\ntp5\ntp6\ntp7\nb.')
  Out[10]: <BTrees.OOBTree.OOBTree at 0x10849eed0>

  In [11]: bt2 = pickle.loads('ccopy_reg\n__newobj__\np0\n(cBTrees.OOBTree\nOOBTree\np1\ntp2\nRp3\n((((NI42\ntp4\ntp5\ntp6\ntp7\nb.')

  In [12]: bt2
  Out[12]: <BTrees.OOBTree.OOBTree at 0x10863fa50>

  In [13]: type(bt2)
  Out[13]: BTrees.OOBTree.OOBTreePy
  
  In [14]: bt2[None]
  Out[14]: 42

  In [15]: None in bt2
  Out[15]: True

I can't set None to a new value:

  In [17]: bt2[None] = 43
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)

But I can delete it:

  In [30]: len(bt2)
  Out[30]: 1

  In [31]: del bt2[None]

  In [32]: len(bt2)
  Out[32]: 0


Iterating keys and values appear to work as expected, too.

Are these the semantics you would expect? If there are unorderable values in the tree, is that going to cause corruption on adding new keys or removing other keys? 

Some such keys may not be removable, at least in extreme cases.

Here I am back in ZODB3 again:

  In [6]: class Bad(object):
   ...:     def __eq__(self, other):
   ...:         return False # Corner case extreme example of broken object
   ...:


  In [8]: bt[Bad()] = 42

And here's loading that pickle in BTrees 4 Python. Note that I can load it successfully, but I can't delete it, even though I'm using the identical instance:

  In [35]: bt2 = pickle.loads('ccopy_reg\n__newobj__\np0\n(cBTrees.OOBTree\nOOBTree\np1\ntp2\nRp3\n((((ccopy_reg\n_reconstructor\np4\n(c__main__\nBad\np5\nc__builtin__\nobject\
    ...: np6\nNtp7\nRp8\nI42\ntp9\ntp10\ntp11\ntp12\nb.')

  In [36]: bt2
  Out[36]: <BTrees.OOBTree.OOBTree at 0x1087d72d0>

  In [37]: len(bt2)
  Out[37]: 1

  In [39]: list(bt2.keys())
  Out[39]: [<__main__.Bad at 0x108964b90>]

  In [40]: bad = list(bt2.keys())[0]

  In [41]: bad in bt2
  Out[41]: False

  In [42]: del bt2[bad]
  ---------------------------------------------------------------------------
  KeyError                                  Traceback (most recent call last)

Hmm, then I tried that same thing again with the C implementation, and it behaves very differently (correctly?):

  In [2]: class Bad(object):
     ...:     def __eq__(self, other): return False
     ...:


  In [4]: bt = pickle.loads('ccopy_reg\n__newobj__\np0\n(cBTrees.OOBTree\nOOBTree\np1\ntp2\nRp3\n((((ccopy_reg\n_reconstructor\np4\n(c__main__\nBad\np5\nc__builtin__\nobject\np
   ...: 6\nNtp7\nRp8\nI42\ntp9\ntp10\ntp11\ntp12\nb.')

  In [5]: bt
  Out[5]: <BTrees.OOBTree.OOBTree at 0x1053d3e60>

  In [6]: list(bt.keys())
  Out[6]: [<__main__.Bad at 0x1055fc390>]

  In [7]: bad = list(bt.keys())[0]

  In [9]: bad in bt
  Out[9]: True

  In [10]: del bt[bad]

But wait! Checking even further, I find that even in BTrees 4, I can still insert Bad into instances of the C implementation, but I can't into the Python implementation:

  In [10]: bt = BTrees.OOBTree.OOBTree()

  In [11]: bt[Bad()] = 42

  In [14]: bt2 = BTrees.OOBTree.OOBTreePy()

  In [15]: bt2[Bad()] = 42
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)

  TypeError: Can't use default __cmp__

Maybe this is an extreme and useless example of a broken object when it comes to ordering and testing what happens if we allow such objects into a BTree on unpickling. But it seems safe to say there are at least some discrepancies and some clear bugs in the handling of unorderable objects still.

Jason

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.