Re: How cache invalidations work (and hack to listen in) (was Re: Applause for ZODB! And a question: How do you get pending change info from a transaction object?)

> ???
> Why doesn't the toy color in Terminal 3 change to green?
> ???


Because a ZODB connection (roughly) exposes a consistent view (version) of the database as-of the point in time at which it was opened, regardless of what other changes are happening and being committed by other connections, even if they change objects this connection has already looked at. That's what MVCC is about: Multi-Version Concurrency Control. In a traditional RDBMS, it's something like a SQL connection operating in REPEATABLE READ isolation mode. That's the only way that a connection's views of all the objects it may access is self-consistent.

The invalidation is to inform *other* connections that are not yet open (sitting in the DB pool) or haven't accessed 'root.toy2' that if they have cached data for 'root.toy2', they need to discard it because it's stale. 

[snip]

> As you can see, the custom invalidate function does trigger automatically when transaction.commit() happens in the other terminal. But why does invalidate get called twice in Terminal 3?

Terminal 3 has two active connections (one at the repl, one in script.py), neither of which caused the invalidation, so both of them need to know to invalidate data in their cache. Hence two messages. 

(Invalidations are sent at the level of the storage server, 1-1 with a ZODB.DB. A typical application process would have a single DB object and use it to open and pool multiple ZODB connections, so in that case you would see only one invalidation message. But ZEO.connection() is a convenience API that opens a unique storage server and ZODB.DB object and returns a single connection, so you have two storage servers opened.)

> Also, on both terminals, it hangs right after "print(connection)" and never finishes executing "toy = connection.get(oid)".
> 
> It's like the connection cannot get the oid and it stalls.

You're deep into implementation details here (and I may get some details wrong too, but I think I have the gist correct; Jim can correct me if I mess up salient details), but basically I think you're deadlocked. MVCCAdapter.invalidate is called in response to an (asynchronous) message from the server...that monkey-patched method turns around and tries to use a connection to get an object from the server (because it's not in cache) which requires sending a message to the server (asynchronously) and (synchronously) waiting for an (asynchronous) response, but we're already handling one asynchronous message. I believe threads and locks are involved (and possibly asyncio event loops)---presumably something along that chain is not re-entrant.

What does a ^C show? 

MVCCAdapter uses 'invalidate' to simply queue the list of pending invalidations for a connection. The *next* time the connection is opened and begins a transaction, it uses that queue to (in local memory only) invalidate the cache. Thus neatly sidestepping any question of crossing asynchronous barriers.

(I know this is toy code, but I'll note it's not generally safe to open a connection at module-scope and expect it to work long term, especially without committing the transaction it's joined to. A better practice is to explicitly open and close connections (DB.open(), conn.close()) within the smallest scope that they are needed, e.g., at the beginning and end of an HTTP request.)

HTH,
Jason (A RelStorage maintainer)

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.