Re: RFC: make ZODB.Exceptions.POSKeyError and ZEO.Exceptions.ClientDisconnected transient errors

Le 01/11/16 13:11, Jim Fulton a écrit :
> I propose to add transaction.TransientError as a base class of
> ZODB.Exceptions.POSKeyError and ZEO.Exceptions.ClientDisconnected.
> 
> Any objections?

Yes too, for at least the reasons given by Jason.

I think you try to find an optimistic solution, which is good for performance, but another issue reason is that race conditions (i.e. transaction started before receiving invalidations from another process) don't always result in POSKeyError.

In ERP5, we rather get KeyError: a zope is told to work on some newly-created object pointing to by some url, and if the Connection has a too old view of the DB, the container (BTree) has no entry for the object id.

Or no exception at all, but wrong calculation. Consider for example a big task that is spread on several transactions and several processes: when a process resumes the task, it may depend on the changes done by previous transactions. So the problem is not always a missing object, but simply a too old value.

So we have cases where we must be sure all invalidations are processed, in order to have a recent-enough view of the DB, i.e. more recent that the event that triggers the transaction. For this, we use a network barrier, with the 'ping' & 'newTransaction' monkey-patches on Connection:

http://git.erp5.org/gitweb/erp5.git/blob/HEAD:/product/ERP5Type/patches/ZODBConnection.py?js=1

1. CMFActivity

We have a separate MySQL DB to track tasks that are filled by any zope, and processed by background zopes. Here, we need 2 things:
- this MySQL DB must be committed after the ZODB (using TM.sortKey)
- call transaction.begin() whenever we get a task to process from MySQL:
  http://git.erp5.org/gitweb/erp5.git/blob/HEAD:/product/CMFActivity/Activity/SQLBase.py?js=1#l489

2. mechanize

Race conditions can easily be reproduced by doing performance tests (e.g. mechanize) on a group of zopes served by haproxy, with creation and modification of objects.

The aforementioned ZODBConnection.py also contains an optional patch to make sure Connection.newTransaction is called when a ZPublisher/Publish transaction begins.

3. NEO - https://pypi.python.org/pypi/neoppod

The 'ping' patch is only for ZEO. In the case of NEO, we do a round-trip in IStorage.sync():
  http://git.erp5.org/gitweb/neoppod.git/blob/HEAD:/neo/client/Storage.py?js=1#l168
And to avoid a useless one at the end of the transaction, we patch afterCompletion():
  http://git.erp5.org/gitweb/neoppod.git/blob/HEAD:/neo/client/__init__.py?js=1#l66

Anyway, NEO also needs this round-trip because it does not receive invalidations from the same TCP connection as those used to retrieve data:
  http://git.erp5.org/gitweb/neoppod.git/blob/HEAD:/neo/client/app.py?js=1#l353
(where load() is usually translated as a kind of loadBefore)


Julien

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.