Re: Wierd slowdowns
Tony Travis <[email protected]>
| Newsgroups | gmane.linux.cluster.openmosix.general |
|---|---|
| Message-ID | <[email protected]> |
Moshe Bar wrote:
> Maybe David Black or Tony Travis want to pick it up again for
openMosix 2.6?
Hello, Moshe.
I'm running 2.4.22-openmosix-2 with the local time() patch, which
I use to avoid processes that check the time() frequently from having to
migrate back to their home node to do it. The only issue with the patch,
as far as I know, is that the time must be correct on all nodes in the
cluster for it to work properly.
To achieve this, I run NTP on all the nodes in our cluster, using the
'head' node as a time reference. I've not had any time-related problems
until recently, when the 'head' node lost sync with an external time
reference and the entire cluster was out by three seconds: Something to
do with GPS time not adjusting for leap seconds I believe. However, the
relative time between nodes in the cluster was not affected.
I don't think these problems are anything to do with checking time
locally on cluster nodes because local elapsed time measurements on a
given node will not be affected by any time offsets from other nodes
unless the processes are migrated there. However, Jeroen used "mosrun"
to force his process onto node32. I agree with Bruce that I'd be very
suspicious about this node's hardware. First thing I would check is that
the node's RAM is not faulty. Have a look at:
http://howto.x-tend.be/openMosixWiki/index.php/Additions%20to%20the%20FAQ
I ran Jeroen's script on node32 in our cluster with the local time()
patch enabled, but I don't see the bug:
ajt@topcat:% cat > time-bug
node=32
while true
do
mosrun -$node sh -c 'time echo "3 2 60 ^ 2 3700 ^ 1 - |" | dc'
done 2>&1 | sed -n 's/^user\s*//p'
ajt@topcat:% chmod +x time-bug
ajt@topcat:% ./time-bug
0m0.970s
0m0.960s
0m0.950s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.950s
0m0.970s
0m0.960s
0m0.950s
0m0.960s
0m0.960s
0m0.950s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.950s
0m0.950s
0m0.960s
0m0.960s
0m0.960s
0m0.950s
0m0.970s
0m0.960s
0m0.960s
0m0.950s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.950s
0m0.970s
0m0.960s
0m0.950s
0m0.960s
0m0.960s
0m0.950s
0m0.960s
0m0.960s
0m0.960s
0m0.960s
0m0.950s
0m0.970s
0m0.960s
0m0.970s
0m0.970s
...
> On Feb 5, 2006, at 5:36 PM, Bruce Knox wrote:
>
>> Below is the reference to the sys_time() patch.
>> While the sys_time patch can dramatically affect the speed of some
>> programs, I would think you'd see this slowdown appear on different
>> nodes as the processes ran over an extended period as with your
>> application. With one node behaving this way, I would be very
>> suspicious of that node's hardware. This could very well be a heat
>> related problem. Bruce
>>
>>>> David Black and Tony Travis - sys_time() patch
>> http://sourceforge.net/mailarchive/message.php?msg_id=7184719
>> allows execution of the function on the current node
>> Programs that frequently execute the sys_time function, such as
>> SETI@home, show remarkable improvements by executing this clock function
>> on the current openMosix node.
>>
>> WARNING: It is critical to sync all the clocks in the cluster.
>>
>> (Typical of many openMosix contributions, this innovations benefits
>> from stepwise improvements. David Black's original patch was improved
>> by Vincent Hanquez (Tab), then by Tony Travis.)
>>
>>
>>>>> "Ian Latter" <[email protected]> 2/5/2006 3:32 PM >>>
>>
>> Hmm, I thought this was new a couple of days ago when I first
>> read it too .. but in my half awake state this morning, it sounds
>> a little like the time() bug .. the one that the patch was written
>> for (the local time patch). Didn't it end up with long on-cpu
>> delays after random run-times (of up to days)? Don't remember
>> who wrote the patch ... I did some experimenting with NTP about
>> 18-24months ago to work the patch into CHAOS tho .. not that
>> this worked.
>>
>>
>>
>>
>> ----- Original Message -----
>>> From: "Kris Buytaert" <mlkb-1zPVpAQ/[email protected]>
>>> To:
>>> Subject: Re: [openMosix-general] Wierd slowdowns
>>> Date: Sun, 05 Feb 2006 14:59:30 +0100
>>>
>>> On Thu, 2006-02-02 at 15:56 +0100, Jeroen Demeyer wrote:
>>>> Hello list,
>>>>
>>>> I'm having a very strange problem where some processes just
>> execute much
>>>> slower than they should. Let me explain: sometimes, for no
>> apparent
>>>> reason, a certain process takes much more CPU time than normal.
>> For
>>>> instance, a certain program normally finishes after about 1
>> CPU-hour,
>>>> but sometimes it might take 30 hours or more. With "CPU-hour", I
>> really
>>>> mean time spent on CPU (not real time), as reported by ps for
>> example.
>>>>
>>>
>>>
>>>> Also note that I don't see any particular correlation between this
>> bug
>>>> and openMosix, so it could be that openMosix doesn't have anything
>> to
>> do
>>>> with it. However, so far I have only seen this problem on the
>> diskless
>>>> 'slave' nodes, and not on the master node (with uptime of 58 days),
>>
>>>> which runs the same kernel.
>>>>
>>>> If anyone has ever seen anything similar to this, please say so.
>>> I've never seen anything similar to this.
>>>
>>> You mention "this node 32" , do you mean that it only happens on 1
>>> particular node or is it happening on more nodes ?
>>>
>>> Also, have you got an idea about the status of the network at that
>>> moment ? It it more loaded than otherwise ?
>>>
>>>
>>> greetings
>>>
>>> --Kris Buytaert <mlkb-1zPVpAQ/[email protected]>
>>>
>>>
>>> -------------------------------------------------------
>>> This SF.net email is sponsored by: Splunk Inc. Do you grep through
>> log
>> files
>>> for problems? Stop! Download the new AJAX search engine that makes
>>> searching your log files as easy as surfing the web. DOWNLOAD
>> SPLUNK!
>>> http://sel.as-us.falkag.net/sel?
>> cmd=lnk&kid=103432&bid=230486&dat=121642
>>> _______________________________________________
>>> openMosix-general mailing list
>>> openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>>> https://lists.sourceforge.net/lists/listinfo/openmosix-general
>>>
>>
>>
>> --
>>
>> Ian Latter
>> Late night coder ..
>> http://midnightcode.org/
>>
>>
>>
>> -------------------------------------------------------
>> This SF.net email is sponsored by: Splunk Inc. Do you grep through log
>> files
>> for problems? Stop! Download the new AJAX search engine that makes
>> searching your log files as easy as surfing the web. DOWNLOAD
>> SPLUNK!
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
>>
>> _______________________________________________
>> openMosix-general mailing list
>> openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>> https://lists.sourceforge.net/lists/listinfo/openmosix-general
>>
>>
>> -------------------------------------------------------
>> This SF.net email is sponsored by: Splunk Inc. Do you grep through
log files
>> for problems? Stop! Download the new AJAX search engine that makes
>> searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
>> _______________________________________________
>> openMosix-general mailing list
>> openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>> https://lists.sourceforge.net/lists/listinfo/openmosix-general
>>
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through
log files
> for problems? Stop! Download the new AJAX search engine that makes
> searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
> _______________________________________________
> openMosix-general mailing list
> openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/openmosix-general
>
>
--
Dr. A.J.Travis, | mailto:[email protected]
Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt
Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751
Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642