mod_perl for multi-process file processing?

Alan Raetz <[email protected]>
Newsgroups gmane.comp.apache.mod-perl
Message-ID <CANJD0wKmBWkTh2ccoeS+vhoBWDcGwKRVLan+QVkPkhwXboBCcw@mail.gmail.com>
So I have a perl application that upon startup loads about ten perl hashes
(some of them complex) from files. This takes up a few GB of memory and
about 5 minutes. It then iterates through some cases and reads from (never
writes) these perl hashes. To process all our cases, it takes about 3 hours
(millions of cases). We would like to speed up this process. I am thinking
this is an ideal application of mod_perl because it would allow multiple
processes but share memory.

The scheme would be to load the hashes on apache startup and have a master
program send requests with each case and apache children will use the
shared hashes.

I just want to verify some of the details about variable sharing.  Would
the following setup work (oversimplified, but you get the idea…):

In a file Data.pm, which I would use() in my Apache startup.pl, I would
load the perl hashes and have hash references that would be retrieved with
class methods:

package Data;

my %big_hash;

open(FILE,"file.txt");

while ( <FILE> ) {

      … code ….

      $big_hash{ $key } = $value;
}

sub get_big_hashref {   return \%big_hash; }

<snip>

And so in the apache request handler, the code would be something like:

use Data.pm;

my $hashref = Data::get_big_hashref();

…. code to access $hashref data with request parameters…..

<snip>

The idea is the HTTP request/response will contain the relevant
input/output for each case… and the master client program will collect
these and concatentate the final output from all the requests.

So any issues/suggestions with this approach? I am facing a non-trivial
task of refactoring the existing code to work in this framework, so just
wanted to get some feedback before I invest more time into this...

I am planning on using mod_perl 2.07 on a linux machine.

Thanks in advance, Alan
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.