Re: Multithreaded connection pooling support for ZODB databases?

> 
>>> 
>>> On Apr 15, 2018, at 8:17 PM, Etienne Robillard <[email protected]> wrote:
>>> 
>>> I would like to define a `ThreadedConnectionPoll` class to allow multithreaded caching of ZODB databases into memory.


The ZODB.DB object is *already* thread safe for connection management. You shouldn't need to do anything other than use ZODB.DB().open() and conn.close() from all the threads you're using. (That is, create exactly one ZODB.DB object for each database you are using and share that object amongst your threads, using db.open()/conn.close() as needed. For storages like RelStorage that have caches at the ZODB.DB level---which are also thread safe---this is important.)

If you are managing multiple databases and want to be able to make references between them (a multiple-database) it is critical that they share the same `databases` object. But that is an advanced usage.


> Here's a updated version of my code so far using threading.local :  
> 
> from threading import local
> from notmm.dbapi.orm import ClientStorageProxy
> 
> class ThreadedConnectionPool(object): 
> 
>     def __init__(self, d, debug=True): 
>         if debug: 
>             assert isinstance(d, dict) == True 
>         local.pool = d 
>         for key,value in local.pool.iteritems(): 

Unless you left out part of the code, it appears that you're trying to set (and read) a class attribute on the threading.local class. This does not create thread-local state. It is also not portable and doesn't work when local is an extension type:

py> from threading import local
py> import sys
py> sys.version_info # CPython 3.7; same result in 2.7
sys.version_info(major=3, minor=7, micro=0, releaselevel='beta', serial=3)
py> local
<class '_thread._local'>
py> local.foo = 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't set attributes of built-in/extension type '_thread._local'

pypy> sys.version_info # pypy 2
(major=2, minor=7, micro=13, releaselevel='final', serial=42)
pypy> from threading import local
pypy> local.foo = 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't set attributes on type object '_local'

It generally also shouldn't be necessary as the parameters to __init__ are preserved and used again in each thread; you can use this to share information if those parameters are mutable:

py> from threading import get_ident
py> from threading import Thread
py> class MyLocal(local):
...     def __init__(self, l):
...         print("Creating in thread", get_ident(), "param", l)
...         l.append(get_ident())
...
py> l = []
py> mylocal = MyLocal(l)
Creating in thread 140736147411840 param []
py> l
[140736147411840]
py> def target():
...     mylocal.foo = 1
...
py> t = Thread(target=target)
py> t.start()
Creating in thread 123145538318336 param [140736147411840]
None
py> l
[140736147411840, 123145538318336]

Jason

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.