RE: ZEO conflicts

Running the jobs in separate processes, with the multiprocessing module, seems to work fine, even with much less aggressive retries (3 attempts, with random delays from 0.2 to .04 s).
Ross
________________________________________
From: [email protected] [[email protected]] on behalf of Boylan, Ross [[email protected]]
Sent: Tuesday, October 11, 2016 1:13 PM
To: zodb ‎[[email protected]]‎
Subject: [ZODB] ZEO conflicts

With ZEO clients some of my transactions seem to get lost without causing any reported error report on the client, despite having 10 retries and a long and variable delay between retries.  The clients are in separate threads, not separate processes, and I'm unclear exactly how this situation should be handled.  Some of the docs say
"Any number of processes can create a ClientStorage instance, and any number of threads in each process can be using that instance. ClientStorage aggressively caches objects locally, so in order to avoid using stale data the ZEO server sends an invalidation message to all the connected ClientStorage instances on every write operation." (http://www.zodb.org/en/latest/articles/old-guide/zeo.html).  I tried to do almost everything within threads.  I'm also not sure if using transaction.commit() and related calls is safe in this context.

The function executed in each thread is
def tickle(i):
    db = ZEO.DB(dbname)
    conn = db.open()
    if i==0:
        @transact_retry()
        def f1():
            aTree = IOBTree()
            conn.root.foo = aTree
            return aTree
        aTree = f1()
        event.set()
    else:
        event.wait()
        @transact_retry()
        def f2():
            return conn.root.foo
        aTree = f2()
    for j in range(2):
        aRun = OneRun(i+j*nThread)
        @transact_retry()
        def f3(aRun):
            aTree[aRun.num] = aRun
        f3(aRun)
    conn.close()

dbname is a named pipe.  After previous discussion, and difficulties getting my munged AST to compile, the wrapper is
def transact_retry(atm=transaction, attempts=10):
    """Decorator to run a function in a transaction.
    If a ConflictError occurs when committing, it will retry up to the
    specified number of times.

    atm can be the transaction module (default) or a transaction manager
    """
    def _transact_retry(f):
        @wraps(f)
        def wrapper(*args, **kw):
            for _fail in range(attempts):
                try:
                    atm.begin()
                    res = f(*args, **kw)
                    atm.commit()
                    return res
                except transaction.interfaces.TransientError:
                    atm.abort()
                    if _fail >= attempts-1:
                        raise
                    delay =random.uniform(0.4, 1.2)
                    print(delay, end=" ")
                    sleep(delay)
        return wrapper
    return _transact_retry

With 50 threads I get 82 to 100 of the expected 100 inserted objects.

zodb 5.0.0, zeo 5.0.1, python 3.4.

Ross Boylan

--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.