Re: Multithreaded connection pooling support for ZODB databases?

Hi Jason,

I just made some more changes to my ThreadedConnectionPool class here: 
https://bitbucket.org/tkadm30/django-hotsauce/commits/81d2e8f30019840d9c8bbdf7f82df6de2be024fc

In summary, the `ThreadedConnectionPool` class is now a subclass of 
`threading.Thread` and is extending the `run` method to populate a 
thread local dictionary with ClientStorage connections.

Kind regards,

Etienne
Le 2018-04-16 à 09:14, Jason Madden a écrit :
>>>> On Apr 15, 2018, at 8:17 PM, Etienne Robillard <[email protected]> wrote:
>>>>
>>>> I would like to define a `ThreadedConnectionPoll` class to allow multithreaded caching of ZODB databases into memory.
>
> The ZODB.DB object is *already* thread safe for connection management. You shouldn't need to do anything other than use ZODB.DB().open() and conn.close() from all the threads you're using. (That is, create exactly one ZODB.DB object for each database you are using and share that object amongst your threads, using db.open()/conn.close() as needed. For storages like RelStorage that have caches at the ZODB.DB level---which are also thread safe---this is important.)
>
> If you are managing multiple databases and want to be able to make references between them (a multiple-database) it is critical that they share the same `databases` object. But that is an advanced usage.
>
>
>> Here's a updated version of my code so far using threading.local :
>>
>> from threading import local
>> from notmm.dbapi.orm import ClientStorageProxy
>>
>> class ThreadedConnectionPool(object):
>>
>>      def __init__(self, d, debug=True):
>>          if debug:
>>              assert isinstance(d, dict) == True
>>          local.pool = d
>>          for key,value in local.pool.iteritems():
> Unless you left out part of the code, it appears that you're trying to set (and read) a class attribute on the threading.local class. This does not create thread-local state. It is also not portable and doesn't work when local is an extension type:
>
> py> from threading import local
> py> import sys
> py> sys.version_info # CPython 3.7; same result in 2.7
> sys.version_info(major=3, minor=7, micro=0, releaselevel='beta', serial=3)
> py> local
> <class '_thread._local'>
> py> local.foo = 1
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: can't set attributes of built-in/extension type '_thread._local'
>
> pypy> sys.version_info # pypy 2
> (major=2, minor=7, micro=13, releaselevel='final', serial=42)
> pypy> from threading import local
> pypy> local.foo = 1
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: can't set attributes on type object '_local'
>
> It generally also shouldn't be necessary as the parameters to __init__ are preserved and used again in each thread; you can use this to share information if those parameters are mutable:
>
> py> from threading import get_ident
> py> from threading import Thread
> py> class MyLocal(local):
> ...     def __init__(self, l):
> ...         print("Creating in thread", get_ident(), "param", l)
> ...         l.append(get_ident())
> ...
> py> l = []
> py> mylocal = MyLocal(l)
> Creating in thread 140736147411840 param []
> py> l
> [140736147411840]
> py> def target():
> ...     mylocal.foo = 1
> ...
> py> t = Thread(target=target)
> py> t.start()
> Creating in thread 123145538318336 param [140736147411840]
> None
> py> l
> [140736147411840, 123145538318336]
>
> Jason
>

-- 
Etienne Robillard
[email protected]
https://www.isotopesoftware.ca/

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.