Re: mod_perl for multi-process file processing?

Cache::FastMmap is a great module for sharing read/write data, but it can't
compete with the speed of loading it all into memory before forking as Alan
said he plans to do.

- Perrin

On Tue, Feb 3, 2015 at 2:05 AM, Cosimo Streppone <[email protected]>
wrote:

> Alan Raetz wrote:
>
>  So I have a perl application that upon startup loads about ten perl
>> hashes (some of them complex) from files. This takes up a few GB of
>> memory and about 5 minutes. It then iterates through some cases and
>> reads from (never writes) these perl hashes. To process all our cases,
>> it takes about 3 hours (millions of cases). We would like to speed up
>> this process. I am thinking this is an ideal application of mod_perl
>> because it would allow multiple processes but share memory.
>>
>
> Sure you could use modperl for this.
> I would also consider at least these alternatives:
>
> - use Cache::FastMmap, https://metacpan.org/pod/Cache::FastMmap
>   Load up your data with a loader script, and forget about it.
>   Cache::FastMmap also works with modperl.
>
> - use a network server, like memcached or redis to store your
>   read-only data, and use a lightweight network protocol (on localhost)
>   to get the data.
>
> In both cases, reading from multiple processes will not be a problem.
> The cheapest solution for the consumer part (the "cases" above)
> would be to use a command like "parallel" to fire up as many copies
> of your consumer script as you can afford.
>
> Hope this helps,
>
> --
> Cosimo
>
>