Re: Forked Processes Won't Migrate, Perl & ForkManager

Interesting...  I re-worked your script Daniel to use ForkManager to 
verify that wasn't what was causing the problem.  I then started adding 
bits of my program, one by one, in place of the loop.  (First calling a 
subroutine, then passing data to it, then opening a file, then reading 
the file, then passing the file data to a routine, etc.)  I discovered a 
couple of very interesting things.

If I have a print statement in the loop anywhere, the process will not 
migrate.  OK, so I went in and commented out all the print statements in 
my app, but it still didn't migrate.  Hmmm....

When I tested whether I could open a text file and pass a line to a 
subroutine, I used the script itself - a few dozen lines.  I then opened 
the actual data file (comma separated, about 135 meg) and BAM!  No more 
migration.

Note that I'm not passing all the data into the subroutine, but rather 
just one line at a time.  As a matter of fact, the process won't migrate 
to another node even if I don't pass anything to the routine; simply 
having opened and read the file will prevent it.  Regardless of this 
fact, when I open a CSV data file (all of which are large), the process 
no longer migrates.  I'm guessing that it would need to copy the entire 
memory space over the network for each forked process, which it won't do 
given that it's so large.  This is a guess though - can anybody confirm 
or deny it?

Anybody have ideas as to how to get around this problem?  (Maybe a 
different method of reading the csv file?  This is the only method I've 
every learned or used in perl - possibly reading line by line instead of 
the entire file would prevent this trouble from occurring?)  I'm using 
something like the following:

[perl code]
open(THEFILE, "/data/datafile.csv");
my @datarows = <THEFILE>;
close(THEFILE);

my $procman = new Parallel::ForkManager(100); # Maximum number of 
processes to fork
foreach $dataline(@datarows)
  {
  $procman->start() and next;  # Fork child and proceed to next record 
in foreach loop
    ProcessRecord($dataline);  # Do subroutine to process record
    $procman->finish(); # Clean up child
  }
$procman->wait_all_children();  # Hold up until all children are done
[/perl code]

James Wagner wrote:

>Thanks Daniel for your reply.
>
>I tried your script and it indeed does spread the work among the 
>available nodes.  (I got the highest load reading I've ever seen using 
>it - very cool. :) )
>
>Using this method of forking processes, would I be better served 
>re-writing the script (again) to fork a separate process for each record 
>and pass the record in as a parameter or by including a call to the 
>processing routine in where you have the math functions?
>
>Are there any special commands I should not use that would possibly 
>prevent the migration of the spawned processes?
>
>Jim
>
>Daniel 'Fremen' Llewellyn wrote:
>
>  
>
>>I haven't used ForkManager, so cannot verify whether it works or not,
>>but I did write a simple perl benchmark tool for oM clusters. The
>>forking is managed quite simply. Check it out (I'll direct you to a
>>less complicated earlier version that just does the forking and
>>processing with no graphical dohickery) at
>>http://www.xyz-network.com/ombenchmark.html/downloads/2.00+/mosixtest-2.02.download 
>>
>>the latest version (3.00) has tk and curses user interfaces if you
>>have the relevent modules installed; though the curses version was
>>broken last time I looked at it.
>>    
>>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV