Re: asynchronous execution, was Re: implementing a set of queue-processing servers

Bas A.Schulte wrote:
> I don't want to use a database table for the sole purpose of sharing 
> data, I mean, I run the Apache/mod_perl servers to handle different 
> components of our system, some run on top of a database and some of them 
> don't.

The thing is, if you need to share data reliably on a cluster of 
machines your choices are basically a database or NFS, and I suspect 
MySQL will be more effective than NFS in many cases.

> I have been looking at some of the IPC::Share* modules, the one I think 
> I can use is (not sure here) IPC::ShareLite, but that darned thing won't 
> install on my dev. machine (iBook/OS X) so I've been postponing things a 
> bit ;)

That will be slow.  All of the shared-memory modules are slow except 
IPC::MM.  They're probably slower than using a database in most cases. 
File-based ones like MLDBM::Sync are faster.

> - what is a given child doing (to do things like: ok, I'm currently 
> pushing data to some client in 5 children, and I don't want to have 
> another child do this now so stuff this task in a queue somehere so I 
> can process it later);

To keep track of what other processes are doing, you need to have them 
make entries in a shared data structure.  IPC::MM would be good for this.

If you want to make a queue, and it needs to be robust (i.e. survive 
server crashes), I'd suggest using MLDBM::Sync or a RDBMS.

> - application state. This is domain-specific so it's a bit hard to 
> explain what I mean. I need serialized and *fast* access to this info so 
> I would prefer not having this in my database.

The modules I mentioned above are generally your best bet.

> NB: I posted a question on the first issue (look for "IPC suggestions 
> sought/talking between children?" somewhere in the mod_perl mailinglist, 
> I never seem to recall the proper archive site for it), didn't get any 
> feedback on it as it probably goes beyond what someone would normally 
> want from a web server.

That would be here:
http://marc.theaimsgroup.com/?l=apache-modperl&m=103538774616838&w=2

You could implement this with IPC::MM.  It supports a hash structure, so 
you could make a hash with keys for the active clients to track how many 
are active for each client.

> Again; I don't know exactly but when I read stuff about entity-, 
> session- and message beans, JMS etc., it has a lot of resemblance with 
> what I'm currently doing "by hand" i.e. implement functionality like 
> that on top of a "bare" Apache/mod_perl server.

You'd have to be more specific, but in my experience people generally 
should not use Enterprise Java Beans unless they are doing something 
foolish like transactions across two databases.  I think you're 
overestimating their value for common projects.  If you just want a good 
way to do object persistence in an RDBMS, this isn't it.

> A good example would be JMS: you get this "for free" (with JBoss anyway 
> ;)) in a J2EE app. server but there's no obvious choice for us perl 
> guys.

It kind of depends on what you want to use it for.  If you want very 
safe message-passing, I would look at MQ Series.  If you just want to 
have a queue, there's no need for messaging at all.  If you want to pass 
asynchronous messages reliably between parts of a distributed system, 
you might get everything you need from qmail and something like 
Mail::Audit that can hand off incoming messages to a Perl program.

> BTW: with the issue on data-sharing: the same thing: I have raw metal 
> (Apache/mod_perl and IPC:MM) and need to implement an API on top of them 
> before I have the needed functionality. Again I'm building stuff again 
> before I can solve my actual business problems.

I think you are vastly over-estimating how much effort JMS/EJB/etc. 
would save you.  The code that implements tracking what children are 
doing on top of IPC::MM is not much different from the code that would 
implement the same thing in Java threads.  General-purpose APIs like JMS 
can't solve specific problems for you.

- Perrin