Re: Zeopack fails raising KeyError

On Tue, Mar 7, 2017 at 10:55 AM, Héctor Velarde <[email protected]>
wrote:

> thank you, very much, Jim; this is what I see in the ZEO server log now:
>
> 2017-03-07T12:24:58 (::ffff:127.0.0.1:32690) pack(time=1488813898.118094)
> started...
> 2017-03-07T12:40:32 pack failed
> Traceback (most recent call last):
>   File "/opt/plone/example.com/eggs/ZODB3-3.10.5-py2.7-linux-x86_64
> .egg/ZEO/StorageServer.py", line 344, in _pack_impl
>     self.storage.pack(time, referencesf)
>   File "/opt/plone/example.com/eggs/ZODB3-3.10.5-py2.7-linux-x86_64
> .egg/ZODB/FileStorage/FileStorage.py", line 1078, in pack
>     pack_result = self.packer(self, referencesf, stop, gc)
>   File "/opt/plone/example.com/eggs/ZODB3-3.10.5-py2.7-linux-x86_64
> .egg/ZODB/FileStorage/FileStorage.py", line 1034, in packer
>     opos = p.pack()
>   File "/opt/plone/example.com/eggs/ZODB3-3.10.5-py2.7-linux-x86_64
> .egg/ZODB/FileStorage/fspack.py", line 397, in pack
>     self.gc.findReachable()
>   File "/opt/plone/example.com/eggs/ZODB3-3.10.5-py2.7-linux-x86_64
> .egg/ZODB/FileStorage/fspack.py", line 190, in findReachable
>     self.findReachableAtPacktime([z64])
>   File "/opt/plone/example.com/eggs/ZODB3-3.10.5-py2.7-linux-x86_64
> .egg/ZODB/FileStorage/fspack.py", line 267, in findReachableAtPacktime
>     pos = oid2curpos[oid]
>   File "/opt/plone/example.com/eggs/ZODB3-3.10.5-py2.7-linux-x86_64
> .egg/ZODB/fsIndex.py", line 96, in __getitem__
>     return str2num(self._data[key[:6]][key[6:]])
> KeyError: '\x1dx'
> 2017-03-07T12:40:32 (12831) Error raised in delayed method
>

OK, this confirms that an object has gone missing.  This shouldn't happen.
:)
It can due to certain rare application bugs.  Is this a single-database
configuration? Or multi-database?

In the short term, I suggest disabling garbage collection by adding:

  pack-gc false

in your <filestorage> element on the server.

This will allow you to pack, but you'll be leaking objects.

The traditional approach to dealing with this has been to chase down the
reference and removing it.  This is difficult, however, due to limitations
in existing tools, because we don't keep backpointers.

Gaaaaa, I just realized that it should be straightforward to create such a
tool, if we had OIDS. It's a shame that the error above doesn't give a
POSKeyError with a full OID.  I wonder if newer versions of ZODB give
POSKeyErrors in this situation.

I suggest trying to use zc.zodbdgc to perform the garbage collection.
(Maybe against a copy of the database, if you're feeling paranoid. :))
 That may give a more helpful error.

I also suggest using zc.FileStorage to do the non-gc pack, at it is much
faster.

Sometime, I need to find the time to roll the functionality from those two
packages back into FileStorage (unless I replace it with FileStorage2).

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.