Re: Forked Processes Won't Migrate, Perl & ForkManager

Which version of openMosix is that, James?

Moshe

 

-----Original Message-----
From: openmosix-general-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
[mailto:openmosix-general-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org] On Behalf Of James
Wagner
Sent: Monday, July 24, 2006 12:16 PM
To: openmosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
Subject: [openMosix-general] Forked Processes Won't Migrate,Perl &
ForkManager

Hi all

I've built an OpenMosix cluster on Slackware Linux.  From any node, I can
run mosmon and see the load for each node.  I built this to aid in the
processing of a large amount of data, but have run into a problem.

I wrote a perl script to read the file and use a foreach loop to process
each record.  I have the program structured to print a status indicater to
the screen, then call a subroutine to do the actual processing, passing it
the variable containing the data.  That process then chews on the data,
writes to a mysql database and a text file.

I tried running this with the cluster enabled and there was no migration
- the node the process was running on was maxxed out and the other ones were
idle.  "Hey, I bet I need to re-write it to use threads!" I think and
proceed to do so.  (When will I learn to read FAQs *first*??? ;) ) Anyways,
the threaded implementation of my program didn't distribute the work among
the nodes and after reading the FAQ, I re-wrote the program again to use the
ForkManager package referenced in the FAQ.  (MUCH easier than using threads,
plus I don't have to download and re-compile
Perl.)

After doing this and still obtaining the same results, I downloaded a stock
analyzer program I found on the 'net, one possible use of which is to test
openmosix clusters.  All the nodes went to 100% CPU usage and stayed there
until the program was done, so I have to assume that the cluster is set up
correctly.

After this, I figured that the routine responsible for chewing on the data
must be exiting too quickly for openmosix to recognize the high load and to
distribute (takes about 5 seconds on an AMD 2500+), so I incorporated a
crude blocking scheme and passed 1000 records a shot to be processed.  This
make no difference either.

I assumed that there must be a problem using a mysql connection on the local
machine, so I changed the program to connect to mysql on a separate machine.
No change.

I then thought that it must be writing to a file on the local machine.  
After trying to enable MFS and continually getting an 'unknown file system
type' error, I gave up and changed it to write to a text file on another
machine using a TCP connection.  I set up a daemon on the remote machine
that does nothing but listen for connections and spit whatever it receives
into a text file.  When I start the program, the only node working on the
data is still the whatever one I start the process on.

I'm frankly out of ideas of what to try.  Processing on the amount of data I
need to will take on the order of months and I would very much like to speed
this up, but don't know what else to try.  Any advice is appreciated.

Jim


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's
Techsay panel and you'll get the chance to share your opinions on IT &
business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
openMosix-general mailing list
openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/openmosix-general


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV