Re: [werkflow-user] Clustered workflow

On Wed, 2004-02-25 at 23:45, Paul Russell wrote:
> 
> Firstly, a caveat: There's going to be a lot of teaching grandma to  
> suck eggs in this e-mail.

No problem! I've had similar thoughts floating around in my head for a
bit, but I've never got around to putting them down in writing, so
having something to slot my thoughts into is very useful.

> Helpfully, Werkflow and J2EE provides most of the gubbins we need to do  
> this:
> 
> * Werkflow already has support for pluggable persistence providers, so  
> connecting to some kind of cluster wide persistent store shouldn't be a  
> problem.

I'd place a bit of a caveat over that. The persistence interface is
there, but I don't know whether it's currently sufficient to persist the
full state of a case. As far as I know, no-one's actually done that yet
(although many of us want to). Architecturally I think the approach is
more-or-less there, but the interface is probably more simple than it
should be. I think bob has described the persistence interface and the
change-set stuff as work in progress.

One major question that probably needs answering straight off: at the
moment the werkflow core works with state held in memory, and the
persistence interface seems more like an in-flight backup mechanism.
Process execution causes changes to the in-memory state, and then it's
flushed to disc through the persistence interface.

In an environment like that you're describing, one node can't know about
all the processes in existence. Similarly, even if you've only got one
node you probably won't have enough resources to keep all your process
state in memory, so the core can't rely on having all the process cases
in memory.

To me this suggests one of two things:
      * The in-memory state held by the core should be considered a
        cache of what is in the persistent store, so cache misses need
        to go through the persistence API to check for cases that aren't
        currently loaded. There's a little support for this in the core
        at the moment (PersistenceManager has a getCorrelations() method
        that should probably be correlating incoming messages against
        cases that have been lost from the cache, but the method is
        never called). This solution would have implications for
        clustering - it would probably be more efficient if a process
        case was handled by a node that had that case cached in memory.
      * Alternatively, move the whole of the case state behind the
        persistence API. Fleeting persistence then becomes a simple
        'everything in memory' workflow manager for when you don't need
        resilience, and the EJB/DB persistence module uses a database to
        hold process state. Nothing is cached, which probably simplifies
        things.
It's really a question of which module looks after process state. With
reference to your werkflow-with-clustering diagram, at the moment the
state is in the Werkflow Core box and the question is how much of that
should remain there and how much should move out to the implementations
of the Persistence Service.

I'm not sure which of the two approaches above would be simpler to
implement, and there may be other options I've not mentioned here. I'd
probably go with the first option to start with, on the basis that it's
more of a migratory path than the second option.

> * J2EE provides JMS which can be used with the Command Message pattern  
> to dispatch commands. Helpfully, JMS can be clustered, which means that  
> each message can only be consumed by one host.
> * J2EE provides Message Driven Beans, which provide an easy way to  
> consume messages on queues, and more importantly provides automatic  
> thread pooling across a cluster, so it should automatically load  
> balance messages across all nodes in a cluster as well as providing  
> transparent failover.

Is it worth thinking about the scheduler as two sub-modules: there's a
part that holds information about which processes are sleeping and what
they're sleeping for, and a second part which deals with how werkflow
gets woken up to do things. From your explanation of the IBM product it
sounds like they have a similar distinction - the scheduler state held
in the database and the JMS queue being effectively a way to wake engine
and get it to do things.

The reason I suggest this is I can think of a few different combinations
that might be useful:
      * For the scheduler state we either have in memory (as we do now)
        or persisted to disc (probably similar to the persistence
        manager).
      * For scheduling werkflow itself, we've got the current thread
        pool mechanism, JMS MDBs and possibly the EJB Timer service.
Hmm; the scheduler state is kind-of a second persistence API. Perhaps we
should push that into the persistence service...

One question that crosses my mind about using JMS to schedule things. At
the moment werkflow doesn't do timed events, but there's prior art in
the petri-net world for transitions that fire a fixed amount of time
after they become enabled. Used in parallel with other transitions they
allow time-outs to be represented quite easily. Adding something like
this to werkflow would probably need the ability to wake a process up
either regularly, or at some fixed time in the future. Other than having
an external process feed a sequence of messages into a queue at a steady
rate, are there other ways to implement this kind of time-based
scheduling with JMS? Is this something the IBM product does?

> * Werkflow doesn't provide a mechanism for 'undoing' changes that have  
> been committed already.

This isn't something I'd looked at in detail. I was under the vague
impression that BPEL and such like allowed you to specify compensating
actions as part of the process. Can these compensating action be treated
as normal parts of the process, or do they need to be collated based on
which actions have been performed (hence needing the mechanism you talk
about)?

> What do you guys think about this? Am I talking cobblers?

You're definitely heading in the right direction, I think. If we can
sort out what we think needs doing I'll definitely be interested in
working on it.

-Mark.