Re: Forked Processes Won't Migrate, Perl & ForkManager

I googled and found the answer to my previous question and tried it 
out.  Indeed the original migration problem must have something to do 
with the memory space of the forked processes because when I use the 
following code, the program starts to distribute.

[perl code]
my $procman = new Parallel::ForkManager(100);
open(CSVFILE, "/data/datafile.csv");
while( <CSVFILE> )
  {
  $dataline = $_;
  $procman->start() and next;
    ProcessRecord($dataline);
    $procman->finish();
  }
$procman->wait_all_children();  # Hold up until all children are done
[/perl code]

The only problem I have now is that the program runs *primarily* on the 
home node.  The other nodes will have a short burst of activity, then 
will go idle again.  The home node is doing 90% of the work; is there a 
way to get openMosix to migrate the processes more easily?  (I've read 
the man pages and have tried various operands to mosrun, none of which 
seem to have any effect.)

Jim

James Wagner wrote:

>Interesting...  I re-worked your script Daniel to use ForkManager to 
>verify that wasn't what was causing the problem.  I then started adding 
>bits of my program, one by one, in place of the loop.  (First calling a 
>subroutine, then passing data to it, then opening a file, then reading 
>the file, then passing the file data to a routine, etc.)  I discovered a 
>couple of very interesting things.
>
>If I have a print statement in the loop anywhere, the process will not 
>migrate.  OK, so I went in and commented out all the print statements in 
>my app, but it still didn't migrate.  Hmmm....
>
>When I tested whether I could open a text file and pass a line to a 
>subroutine, I used the script itself - a few dozen lines.  I then opened 
>the actual data file (comma separated, about 135 meg) and BAM!  No more 
>migration.
>
>Note that I'm not passing all the data into the subroutine, but rather 
>just one line at a time.  As a matter of fact, the process won't migrate 
>to another node even if I don't pass anything to the routine; simply 
>having opened and read the file will prevent it.  Regardless of this 
>fact, when I open a CSV data file (all of which are large), the process 
>no longer migrates.  I'm guessing that it would need to copy the entire 
>memory space over the network for each forked process, which it won't do 
>given that it's so large.  This is a guess though - can anybody confirm 
>or deny it?
>
>Anybody have ideas as to how to get around this problem?  (Maybe a 
>different method of reading the csv file?  This is the only method I've 
>every learned or used in perl - possibly reading line by line instead of 
>the entire file would prevent this trouble from occurring?)  I'm using 
>something like the following:
>
>[perl code]
>open(THEFILE, "/data/datafile.csv");
>my @datarows = <THEFILE>;
>close(THEFILE);
>
>my $procman = new Parallel::ForkManager(100); # Maximum number of 
>processes to fork
>foreach $dataline(@datarows)
>  {
>  $procman->start() and next;  # Fork child and proceed to next record 
>in foreach loop
>    ProcessRecord($dataline);  # Do subroutine to process record
>    $procman->finish(); # Clean up child
>  }
>$procman->wait_all_children();  # Hold up until all children are done
>[/perl code]
>  
>
>  
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV