How cache invalidations work (and hack to listen in) (was Re: Applause for ZODB! And a question: How do you get pending change info from a transaction object?)

A wildly useful, but subtle feature of ZODB is that it has a *transactionally
consistent cache*.  Applications don't have to worry about cache
invalidation. Data in the cache is always consistent with the database
snapshot (https://en.wikipedia.org/wiki/Snapshot_isolation) they see, based
on transaction start time.

The way that ZODB does this is via object invalidations.  Clients get
invalidations for objects modified in transactions.  At the start of a
transaction, invalidations are used to remove state for any objects in the
cache that were modified.  Subsequent reads inside the transaction get
object data as of the transaction start time.

The way this works differs for RelStorage and other storages:

   - For RelStorage, at the start of a transaction, RelStorage queries for
   new invalidations (since the last poll).  This *pull* strategy likely
   allows RelStorage to scale to more clients, because multiple invalidations
   aren't sent for frequently-changing objects and because invalidations
   aren't sent to idle clients.  A disadvantage of this approach is that every
   transaction makes at least one round-trip to the server.

   - For other storages, invalidations are pushed to clients as
   transactions are committed.  These invalidations are buffered and applied
   at transaction starts.  Theoretically, this can cause a lot of network
   transaction, although, to my knowledge, no one has ever demonstrated this
   to be significant. I suspect that this is not significant. Because this is
   asynchronous. Many read transactions require no server round trips.

On Tue, May 1, 2018 at 8:30 AM, David Pesta <[email protected]> wrote:

> Yes, the hack you mentioned will be helpful if it's not too much trouble
> to share it. So that I can get started on something.
>
So, to get notified when objects change:

For non-RelStorage storahe, a storage adapter,
https://github.com/zopefoundation/ZODB/blob/c75a197deebc3a65aa521bda11bdaa83dadd8b48/src/ZODB/mvccadapter.py,
provides, among other things, invalidation buffering.

When a storage (including a local storage like a file storage) sends
invalidations to the client, the invalidate method,
https://github.com/zopefoundation/ZODB/blob/c75a197deebc3a65aa521bda11bdaa83dadd8b48/src/ZODB/mvccadapter.py#L77,
is called that broadcasts invalidations to instances for each local
database connection, which cache the invalidations for the next transaction.

On very rare occasions, an invalidateCache method is called to indicate
that the entire cache should be invalidated.  I suggest ignoring this for
now.

Soon, I'm going to provide a public API to subscribe to these
invalidations. In the meantime, you can monkey patch this method to call
your code in addition to what it's doing now. Something like (untested):

invalidate_orig = MVCCAdapter.invalidate
def invalidate(self, transaction_id, oids):
    my_subscriber(oids)
    invalidate_orig(self, transaction_id, oids)
MVCCAdapter.invalidate = invalidate

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.