Re: Updating indexes asynchronously (was: Re: Zeo and two processes changing the zodb - not getting the up to date version of the object)
Jo Gilder <[email protected]>
| Newsgroups | gmane.comp.web.zope.zodb |
|---|---|
| Message-ID | <[email protected]> |
Jim, thanks for this. It all sounds very interesting and something I will keep a close eye on. On Friday, 10 February 2017 19:46:04 UTC, Jim Fulton wrote: > > I think your use case has a much better but very different solution. > > Your current approach, which is entirely reasonable and logical has a > number of pitfalls, in addition to the one you ran into: > > - Your application has to implement events and event handlers to implement > the handoff to celery. IMO, this adds a lot of annoying complexity to > applications. > > - Handoff to queueing systems is more brittle than it should be, because > it's not transactional. This is fixable, but remains unfixed, AFAIK. This > is because the handoff rarely fails, which is in some ways the worst kind > of failure. I fixed this for a specific project at a previous company. I > would love to come up with a general solution at some point. > > - Depending on your queueing system, if your indexer is down for some > reason for a long enough time, you may lose data. > > - If you need to re-do indexing, you have to use some different mechanism. > > Something I've wanted to do for some time is to provide a pub/sub ( > https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) > mechanism for ZODB data. The basic idea: > > - You have an application that wants to act on database changes, typically > by updating an index. > > - Your application calls an API that returns an iterator. This iterator > returns data that you can act on. By default the iterator runs forever, > until you interrupt it, returning new data as it is committed. > > - Optionally, you can supply start and end tids. You'd supply a start tid > to pick up where you previously left off. An end tid would typically be > used to redo computations, say after changing how you want to process data. > > I've contemplated implementing this for ZEO on top of ZRS for some time. > > I recently implemented this for RelStorage: > > http://www.newtdb.org/en/latest/topics/following.html > > (Jo, I realize you aren't using RelStorage, so this isn't something you > can use now.) > > Note that although this is part of Newt DB, you don't have to use Newt. > You could, for example, use it to update SOLR. One thing to note is that > there's an API to save your progress, so if your indexer stops for a while, > it can pick up where it left off. Also note the complication that packing > adds. > > Newt uses this for it's new asynchronous update, which provides an example > of using the "follow" API to update an separate data representation > asynchronously: > > https://github.com/newtdb/db/blob/master/src/newt/db/updater.py#L223 > > I plan to implement a similar "follow" API for ZEO based on ZRS. > > Jim > > > > I have posted this on substnced as well. >> >> So I am using celery to maintain a solr index of my zodb database. I am >> using substanced cms and zeo. >> >> Celery is running under supervisor and I am using deferred indexing as >> described in the substanced documentation. >> >> Once I set celery going the first batch of tasks will execute beautifully >> but any of the next batch will say they are successful but don't show any >> of the changes I have made in zodb. I had assumed this was something to do >> with my celery setup but I think it may be zeo returning the old unchanged >> object and not the new one. >> >> I read this post and it seems very similar. >> >> >> https://groups.google.com/forum/#!searchin/zodb/zeo|sort:relevance/zodb/wJ34URX5zp0/ZXODMcEvEwAJ >> >> >> substanced object changed - event modified triggered which has two >> subscribers. >> 1 - index catalog of substanced >> >> 2- make updates to solr >> >> Process 1 is run via a script sd_drain_indexing controlled by supervisor >> >> Process 2 is run using celery which is passed the UUID of the substanced >> object that has changed. >> >> To allow the indexing to happen in 1 I am delaying 2 for two minutes. >> >> However 2 doesn't seem to pick up the latest version of the object. If I >> restart celery then I get the latest version but otherwise I am getting a >> 'stale' version of the object. >> >> How can I ensure that I am getting a current version of the object? I am >> using the following code in the __call__ method of my basetask. >> >> >> >> class TalbotTask(BaseTask): >> """Abstract base class for all tasks in my app.""" >> # >> >> def __call__(self, *args, **kwargs): >> if getattr(self, "registry", None) is None: >> registry = App.conf['PYRAMID_REGISTRY'] >> env = bootstrap(registry.settings[u'ini_file_celery']) >> root = env['root'] >> self.registry = registry >> self.root = root >> self.solrFeed = TalbotSolrFeeder(self.registry, TALBOT_LOG) >> self.objectmap = find_objectmap(self.root) >> super(BaseTask, self).__call__(*args, **kwargs) >> >> >> >> I then tried adding in an after commit hook but think this isn't relevant >> as the commit I want to pick up is the one from sd_drain_indexing. >> >> I also took out the if statement in the above code so I always recreated >> the root. This worked initially but falls over because I have 'too many >> open files'. >> >> Any help? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "zodb" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&[email protected]> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Jim Fulton > http://jimfulton.info > -- You received this message because you are subscribed to the Google Groups "zodb" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.