Re: Databases and threading

> On Nov 19, 2019, at 04:44, =C3=89loi Rivard <[email protected]> wrote:
>=20
> Hello, There are a few things I wonder about ZODB and concurrency, and I =
am not sure of the answers the documentation gives.
> The documentation states tha database connections, transactions managers =
and transactions are not thread safe. It also states that we cannot share d=
atabases between processes. But can we share databases an storages between =
threads?=20

A Connection instance, and all the persistent objects accessed through it, =
may be used from exactly one thread (or greenlet!) at a time; the same goes=
 for the transaction/manager associated with the connection.=20

A DB instance is there to vend Connection instances, and it can do so from =
any number of threads concurrently. How the DB, Connection, and storage coo=
perate to make this work are implementation details.

So this example should be fine:

> import ZODB, threading
> db =3D ZODB.DB(None)
> def work():
>     with db.transaction() as conn:
>         do_stuff()
>=20
> threading.Thread(target=3Dwork).start()
> threading.Thread(target=3Dwork).start()


But this would be broken:

    def process_chunk(container, chunk_number):
        chunk =3D container[chunk_number]
        # Do stuff with chunk
   =20
    db =3D get_the_db()
    conn =3D db.open()
    root =3D conn.root()

    threading.Thread(target=3Dprocess_chunk, args=3D(1,)).start()
    threading.Thread(target=3Dprocess_chunk, args=3D(2,)).start()

That's broken because it's sharing the same persistent object (the root) be=
tween threads without any sort of locking to ensure that the underlying Con=
nection (hidden in _p_jar), is only used from one thread at a time.

The cross-thread sharing is pretty blatant here. A more subtle trap is shar=
ed caches of persistent objects.=20

    cache =3D {}
    db =3D get_the_db()
    def process_chunk(chunk_number):
         result =3D []
         conn =3D db.open()
         root =3D conn.root()
         chunk =3D root[chunk_number]
         for part in chunk:
             answer =3D cache.get(part)
             if answer is None
                 answer =3D part.find_or_compute_answer()
                 if answer._p_jar is None:=20
                     conn.add(answer) # could be new, could be previously s=
aved
                 cache[part] =3D answer
             result.append(answer)
         return result

    threading.Thread(target=3Dprocess_chunk, args=3D(1,)).start()
    threading.Thread(target=3Dprocess_chunk, args=3D(2,)).start()

If any of the chunks use the same part, we could be accessing objects from =
different connections in different threads simultaneously.

Sometimes a shared cache like that can be found at a module level. That's d=
estined to eventually result in a ConnectionStateError, but because of ZODB=
's caching, it may not be until the application is under significant load i=
n a production setting. (Don't do that.)

Speaking of ZODB's caching, very often many of the cross-threading issues c=
an be hidden by it: if an object is already in memory, there will be no nee=
d to use the _p_jar at all and ZODB is taken out of the equation entirely. =
Then you're just left with the normal concerns about using objects from mul=
tiple threads. (All of the examples above are probably that way.) That's al=
l fine and good until an object gets ghosted (usually at the most inopportu=
ne time) or your access pattern changes just enough that an object that was=
 usually in memory before, or was never accessed at all before, suddenly ha=
s to be loaded. I suggest having your tests frequently make use of DB.cache=
Minimize() (e.g., between requests/transactions) to help smoke out issues l=
ike that; you can even make adjustments to the DB's connection pool so that=
 sequential requests don't get the same Connection object.

Is all of that to say there's no way to parallelize work using a single Con=
nection and transaction? Not at all. It can be done, but (like anything inv=
olving concurrency) it takes careful design. It's certainly easier to stick=
 to the one-thread-per-connection/one-connection-per-thread/nothing-shared-=
except-the-DB model that's encouraged by default.

HTH,
Jason


--=20
You received this message because you are subscribed to the Google Groups "=
zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/=
zodb/236EF6D7-AE04-4014-A937-41F53C37CB6D%40nextthought.com.