Re: Application change notification

Hi all,

We maintain such a real-time ZODB/RelStorage-based web application so I can share my thoughts from that perspective.

It's not clear to me that the method described here would be useful for us at an application level.

The primary issue I see is that it's too low-level. Getting an OID of an object that's changed is not necessarily useful from an application perspective without further content. For example, a user may be editing a document that has an attachment that's a Blob. If the contents of the blob change, we'd get the OID notification for the Blob, but what we really want to show as changed---the containing document---didn't change from the ZODB perspective, so we wouldn't get notified about that. 

Now, if every possible object always contained a __parent__ pointer it might be possible to backtrack and divine what the actual user-facing object was, but not every object has such a pointer. Alternatively, somewhere globally we might maintain a reverse mapping from user objects to all the OIDs they depend on and use that to look for the corresponding object, but that seems overly complex. But sometimes a change doesn't even involve a Persistent object; an example would be adding or removing a string to a list of messages---the list is persistent, but the message string isn't, so even though you get a change notice for the list, unless you *already* had the old state in memory, you don't know what changed and how to inform the user (that is, we'd prefer to just inform the UI that "this message was added" or "this message was removed" instead of "oh, here's your entire message list again").

A related problem is that there are lots of object changes that we don't really care about presenting to a user, things due to internal bookkeeping like sessions and so forth---plus we use catalog-like structures, so this would be exacerbated. The callback would be getting called lots of times just to have to filter out the noise.

Fundamentally, with the RelStorage implementation being delayed, the "real-time" aspect goes missing, which makes this a bit self-defeating :)

So what do we do now when we want to deal with presenting users with changed information in real time? The application is more or less responsible for this: Using a system based on zope.event (and zope.lifecycleevent) the application can decide what changes are important to the users. Such important events are distributed across the server cluster in real-time using Redis.

If a hypothetical ZODB-based change notification system were opt-in (such that the application specifies what OIDs it wants to have subscribers called for), that would solve part of the low-level problem---although still only for Persistent objects. The real-time issue still remains.

I can certainly see the value of a low-level OID-based change system for certain classes of asynchronous problems---a job queue type system for indexing, say (for which we also use Redis)---and I think something like that is already implemented in Newt DB. But as far as real-time UIs, that might be too application-specific to be served at this level.

Those are just my thoughts, though. Others might have a different perspective.

~Jason

> On Mar 11, 2017, at 10:27, Jim Fulton <[email protected]> wrote:
> 
> In implementing "real-time" UIs, where you want to update a user's view of information due to actions by other users, you need some way of notifying users when objects have changed.
> This is a feature that I've wanted to add to ZODB for some time.  Until ZODB 5, it would have been extremely easy to add. So easy, in fact, that when I needed it I added it via monkey patching. :-|
> 
> With the switch to MVCC, however, it's become a bit trickier.  Here are some thoughts:
> 	• At the application level, the ZODB.DB class grows a subscribe method:
> 
>   subscribe(callback)
> 
> The callback will be called with a sequence of object ids that have changed.
> 
> The subscribe method may be called more than once to register multiple subscribers.
> 
> Subscribers may be called more frequently than there are actual changes.  (This is to make a potential RelStorage implementation a little simpler.)
> 
> 	• A new storage interface, IMVCCNotify will provide a subscribe method like the one above, except that it will never be called more than once, so the storage need not support multiple subscribers.
> 	• These interfaces will be implemented for DB and the built-in MVCC adapter used for traditional non-MVCC storages.
> 	• The subscribe method might be implemented for RelStorage, but notifications might be sent late, when a client polled for changes
> 	• I plan to implement this for Newt DB using a database trigger to get  notifications sent in real time.
> Thoughts?
> 
> Jim
> 
> -- 
> Jim Fulton
> http://jimfulton.info
> 
> -- 
> You received this message because you are subscribed to the Google Groups "zodb" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.