lots of ReadConflictErrors
"Boylan, Ross" <[email protected]>
| Newsgroups | gmane.comp.web.zope.zodb |
|---|---|
| Message-ID | <[email protected]> |
Running 50 threads with minimal synchronization produces lots of read conflict errors. I thought the transaction manager.attempts() idiom was supposed to catch such errors; in fact, it seems it sometimes does because one run had lots of errors but all items ended up in the IOBTree. But on another try only 87 of 100 did.
What am I missing?
The core operation each thread does is to add a couple of unique objects with unique integer keys to an IBTree. It looks like this:
aRun = OneRun(i+nThread, start, datetime.now(), 0)
for attempt in aTransactionManager.attempts():
with attempt:
# aVariant has an IOBTree into which aRun is inserted
aVariant.add(aRun)
aTransactionManager is thread-local, and the database was opened with
nThread = 50
db= ZODB.DB(dbname, pool_size=nThread+1)
I suspect that maybe every error is getting reported, even ones that are caught, But the problem is not just cosmetic, since some information is sometimes lost.
BTW, this seemed to be working fine when I used locks to assure that each thread wrote one item in round-robin fashion.
My goal for the round-robin was to produce maximum interleaving and maximum out-of-syncness between threads, but obviously
that also reduced contention since it serialized the operations.
Here's the first error:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/usr/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "./ztest.py", line 90, in tickle
aVariant.add(aRun)
File "/home/ross/.local/lib/python3.4/site-packages/transaction-1.6.1-py3.4.egg/transaction/_manager.py", line 132, in __exit__
self.commit()
File "/home/ross/.local/lib/python3.4/site-packages/transaction-1.6.1-py3.4.egg/transaction/_manager.py", line 123, in commit
return self.get().commit()
File "/home/ross/.local/lib/python3.4/site-packages/transaction-1.6.1-py3.4.egg/transaction/_transaction.py", line 280, in commit
reraise(t, v, tb)
File "/home/ross/.local/lib/python3.4/site-packages/transaction-1.6.1-py3.4.egg/transaction/_compat.py", line 56, in reraise
raise value
File "/home/ross/.local/lib/python3.4/site-packages/transaction-1.6.1-py3.4.egg/transaction/_transaction.py", line 271, in commit
self._commitResources()
File "/home/ross/.local/lib/python3.4/site-packages/transaction-1.6.1-py3.4.egg/transaction/_transaction.py", line 416, in _commitResources
reraise(t, v, tb)
File "/home/ross/.local/lib/python3.4/site-packages/transaction-1.6.1-py3.4.egg/transaction/_compat.py", line 56, in reraise
raise value
File "/home/ross/.local/lib/python3.4/site-packages/transaction-1.6.1-py3.4.egg/transaction/_transaction.py", line 390, in _commitResources
rm.commit(self)
File "/home/ross/.local/lib/python3.4/site-packages/ZODB-5.0.0-py3.4.egg/ZODB/Connection.py", line 472, in commit
oid, serial, transaction)
File "/home/ross/.local/lib/python3.4/site-packages/ZODB-5.0.0-py3.4.egg/ZODB/BaseStorage.py", line 357, in checkCurrentSerialInTransaction
oid=oid, serials=(committed_tid, serial))
ZODB.POSException.ReadConflictError: database read conflict error (oid 0x046b, serial this txn started with 0x03ba67e3a8300000, serial currently committed 0x03ba67e3a8df9955)
Sometimes, toward the end, some other errors occur:
ZODB.POSException.ReadConflictError: database read conflict error (oid 0x03f4, serial this txn started with 0x03ba67df5005eedd, serial currently committed 0x03ba67df5012a5dd)
Couldn't load state for __main__.OneRun 0x0453
Traceback (most recent call last):
File "/home/ross/.local/lib/python3.4/site-packages/ZODB-5.0.0-py3.4.egg/ZODB/FileStorage/FileStorage.py", line 440, in _lookup_pos
return self._index[oid]
File "/home/ross/.local/lib/python3.4/site-packages/ZODB-5.0.0-py3.4.egg/ZODB/fsIndex.py", line 109, in __getitem__
return str2num(self._data[key[:6]][key[6:]])
KeyError: b'\x04S'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ross/.local/lib/python3.4/site-packages/ZODB-5.0.0-py3.4.egg/ZODB/Connection.py", line 753, in setstate
p, serial = self._storage.load(oid)
File "/home/ross/.local/lib/python3.4/site-packages/ZODB-5.0.0-py3.4.egg/ZODB/mvccadapter.py", line 144, in load
r = self._storage.loadBefore(oid, self._start)
File "/home/ross/.local/lib/python3.4/site-packages/ZODB-5.0.0-py3.4.egg/ZODB/FileStorage/FileStorage.py", line 483, in loadBefore
pos = self._lookup_pos(oid)
File "/home/ross/.local/lib/python3.4/site-packages/ZODB-5.0.0-py3.4.egg/ZODB/FileStorage/FileStorage.py", line 442, in _lookup_pos
raise POSKeyError(oid)
ZODB.POSException.POSKeyError: 0x0453
--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.