Re: asynchronous execution, was Re: implementing a set of queue-processing servers

Stephen Adkins wrote:

> So what I think you are saying for option 2 is:
>
>    * Apache children (web server processes with mod_perl) have two
>      personalities:
>        - user request processors
>        - back-end work processors
>    * When a user submits work to the queue, the child is acting in a
>      "user request" role and it returns the response quickly.
>    * After detaching from the user, however, it checks to see if fewer
>      than four children are processing the queue and if so, it logs into
>      the mainframe and starts processing the queue.
>    * When it finishes the request, it continues to work the queue until
>      no more work is available, at which time, it quits its "back-end
>      processor" personality and returns to wait for another HTTP request.

>
>
> This just seems a bit odd (and unnecessarily complex).


It does when you put it like that, but it doesn't have to be that way. 
I would separate the input (user or queue) from the processing part. 
You'd have a module that runs in mod_perl which knows how to process 
requests.  You have a separate module which can provide a UI for placing 
  requests.  Synchronous ones go straight to processing, while asynch 
ones get added to the queue.

You'd also have a controlling process that polls the queue and if it 
finds anything it uses LWP to send it to mod_perl for handling.  I would 
make this a tiny script triggered from cron if possible, since cron is 
robust and can handle outages and error reporting nicely.

> Why not let there be web server processes and queue worker processes
> and they each do their own job?  Web servers seem to me to be for
> synchronous activity, where the user is waiting for the results.


When I think of queue processing, I think of a system for handling tasks 
in parallel that provides a simple API for plugging in logic, a 
well-defined control interface, logging, easy configuration... sounds 
like Apache to me.  You just need a tiny control process to trigger it 
via LWP.  Apache is already a system for handling a queue of HTTP 
requests in parallel, so you just have to make your requests look like HTTP.

You certainly could do this other ways, but you'd probably have to write 
a lot more code or else use something far less reliable than Apache.

> P.S. Another limitation of the "use Apache servers for all server 
> processing"
> philosophy seems to be scheduled events or system events (those not
> initiated by an HTTP request, which are user events).


Cron/at + LWP.

- Perrin