Re: tapestry background jobs in a clustered environment

Not sure I understand where those pessimistic locks came from.

In out case there's no locking at all, every machine in a cluster processes
jobs simultaneously, unless, of course, the jobs are not from the same
logical queue and must be executed in order.

By row-level locking I mean PostgreSQL's SELECT ... FOR UPDATE, i.e.:

UPDATE units_of_work
SET started_at = ?
WHERE id = (SELECT id
            FROM units_of_work
            WHERE started_at IS NULL
            LIMIT 1
            FOR UPDATE)
RETURNING id

This is a simplified version of what's actually happening, but illustrates
the idea: different coordinators don't lock each other.


On Wed, Jun 28, 2017 at 11:05 PM, Ilya Obshadko <[email protected]>
wrote:

> I was actually looking at Spring Batch (and a couple of other solutions). I
> don’t think Spring Batch could be of much help here.
>
> My conclusion is similar to what you are saying - implementing lightweight
> job coordinator is much easier.
>
> Row-level locking works well when you are dealing with a simple queue table
> - you do a pessimistic lock on N rows, process them and give a chance to
> another host in the cluster. Unfortunately only one of my background jobs
> is suitable for this type of refactoring.
>
> Other jobs process records that shouldn’t be locked for a considerable
> amount of time.
>
> So currently I’m thinking of the following scenario:
>
> - pass deployment ID via environment to all containers (ECS can do this
> quite easily)
> - use a simple table with records containing job name, current cluster
> deployment ID and state
> - first background executor that is able to lock an appropriate job row
> starts working, the other(s) are cancelled
>
>
>
> On Tue, Jun 27, 2017 at 10:16 PM, Dmitry Gusev <[email protected]>
> wrote:
>
> > Hi Ilya,
> >
> > If you have Spring in your classpath you may look at Spring Batch.
> >
> > For our projects we've built something similar -- a custom jobs framework
> > on top of PostgreSQL.
> >
> > The idea is that there a coordinator service (Tapestry service) that runs
> > in a thread pool and constantly polls special DB tables for new records.
> > For every new unit of work it creates instance of a worker (using
> > `ObjectLocator.autobuild()`) that's capable of processing the job.
> >
> > The polling can be optimised well for performance using row-level locks &
> > DB indexing.
> >
> > Coordinator runs in the same JVM as the rest of the app so there's no
> > dedicated process.
> > It integrates with tapestry's EntityManager so that you could create a
> job
> > in transaction.
> >
> > When running in a cluster every JVM has its own coordinator -- this it
> how
> > the jobs get distributed.
> >
> > But you're saying that row-level locking doesn't work for some of your
> > use-cases, can you be more concrete here?
> >
> >
> > On Tue, Jun 27, 2017 at 9:35 PM, Ilya Obshadko <[email protected]>
> > wrote:
> >
> > > I’ve recently expanded my Tapestry application to run multiple hosts.
> > While
> > > it’s quite OK for the web-faced part (sticky load balancer does most of
> > the
> > > job), it’s not very straightforward with background jobs.
> > >
> > > Some of them can be quite easily distributed using database row-level
> > > locks, but this doesn’t work for every use case I have.
> > >
> > > Are there any suggestions about this? I’d prefer not to have a
> dedicated
> > > process running background tasks. Ideally, I want to dynamically
> > distribute
> > > background jobs between hosts in cluster, based on current load status.
> > >
> > >
> > > --
> > > Ilya Obshadko
> > >
> >
> >
> >
> > --
> > Dmitry Gusev
> >
> > AnjLab Team
> > http://anjlab.com
> >
>
>
>
> --
> Ilya Obshadko
>



-- 
Dmitry Gusev

AnjLab Team
http://anjlab.com