Wierd slowdowns

Hello list,

I'm having a very strange problem where some processes just execute much 
slower than they should.  Let me explain: sometimes, for no apparent 
reason, a certain process takes much more CPU time than normal.  For 
instance, a certain program normally finishes after about 1 CPU-hour, 
but sometimes it might take 30 hours or more.  With "CPU-hour", I really 
mean time spent on CPU (not real time), as reported by ps for example.

The even wierder thing is that this 'slowness' is constant during a 
process: if a certain process starts up at normal speed, it will remain 
so as long as it stays on the same node.  After migration, the process 
will again either run slow or normal on the new node.

To test all this, I wrote the following piece of shell-code:

node=32
while true
do
	mosrun -$node sh -c 'time echo "3 2 60 ^ 2 3700 ^ 1 - |" | dc'
done 2>&1 | sed -n 's/^user\s*//p'

This constantly runs the command 'echo "3 2 60 ^ 2 3700 ^ 1 - |" | dc'
on node 32, and prints the user time.  This gives the following results:

0m1.010s
0m38.280s
0m1.050s
0m1.240s
0m1.010s
0m1.010s
0m1.010s
0m1.000s
0m1.000s
0m1.010s
0m1.020s
0m1.000s
0m1.140s
0m1.010s
0m1.010s
0m1.010s
0m1.000s
0m1.010s
0m1.010s
0m1.010s
0m1.010s
0m1.000s
0m1.000s
0m1.010s
0m7.390s
0m1.000s
0m1.000s
0m1.010s
0m1.000s
0m38.270s
0m1.050s
0m1.050s
0m1.240s
0m1.020s
0m1.010s
0m1.000s
0m1.150s
0m1.010s
0m1.010s
0m1.000s
0m1.010s
0m1.010s
0m1.000s
0m1.010s
0m1.010s
0m1.000s
0m1.010s
0m1.040s
0m1.000s
0m1.020s
0m38.270s
0m1.040s
0m1.250s
0m1.010s
0m1.010s

As you can see, normally the computation takes 1 second, but sometimes a 
little more, sometimes 7, sometimes 12 (not on this screenshot) and 
sometimes 38.  A big problem to analyzing this issue is the fact that 
these slowdowns only start occuring after some time.  After I reboot 
this node 32, I might not have any slowdowns today or this week, but 
eventually they will start occuring.  Usually this is after a couple of 
days.  They will also become much more frequent if the node has been 
running long enough.

Also note that I don't see any particular correlation between this bug 
and openMosix, so it could be that openMosix doesn't have anything to do 
with it.  However, so far I have only seen this problem on the diskless 
'slave' nodes, and not on the master node (with uptime of 58 days), 
which runs the same kernel.

If anyone has ever seen anything similar to this, please say so.

Regards,

Jeroen Demeyer


<rant>I *really* hope this e-mail does not get forwarded to /dev/null as 
usual, because I'm getting more and more frustrated with openMosix' many 
bugs and annoyances...  The fact that I almost never get any response 
from this list does not help.</rant>


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642