Re: mod_perl for multi-process file processing?

Alan Raetz wrote:

> So I have a perl application that upon startup loads about ten perl
> hashes (some of them complex) from files. This takes up a few GB of
> memory and about 5 minutes. It then iterates through some cases and
> reads from (never writes) these perl hashes. To process all our cases,
> it takes about 3 hours (millions of cases). We would like to speed up
> this process. I am thinking this is an ideal application of mod_perl
> because it would allow multiple processes but share memory.

Sure you could use modperl for this.
I would also consider at least these alternatives:

- use Cache::FastMmap, https://metacpan.org/pod/Cache::FastMmap
   Load up your data with a loader script, and forget about it.
   Cache::FastMmap also works with modperl.

- use a network server, like memcached or redis to store your
   read-only data, and use a lightweight network protocol (on localhost)
   to get the data.

In both cases, reading from multiple processes will not be a problem.
The cheapest solution for the consumer part (the "cases" above)
would be to use a command like "parallel" to fire up as many copies
of your consumer script as you can afford.

Hope this helps,

-- 
Cosimo