Re: Wierd slowdowns
"Ian Latter" <[email protected]>
| Newsgroups | gmane.linux.cluster.openmosix.general |
|---|---|
| Message-ID | <[email protected]> |
Hello Tony, His problem was reported as an MTRR problem .. he fixed it with manual MTRR entries. I originally suggested the local time patch as a solution, not as the problem - and I still think its a good idea (generally, obviously not now that the problem is known to be MTRR in this case). It was just unfortunate that I couldn't get the NTP time mesh that I wanted (automatically generated). I hadn't heard of the leap second problem with GPS .. interesting find. ----- Original Message ----- >From: "Tony Travis" <[email protected]> >To: "Moshe Bar" <moshe-ay74M1d3r6RWk0Htik3J/[email protected]> >Subject: Re: [openMosix-general] Wierd slowdowns >Date: Sun, 19 Feb 2006 21:42:50 +0000 > > Moshe Bar wrote: > > Maybe David Black or Tony Travis want to pick it up again for > openMosix 2.6? > > Hello, Moshe. > > I'm running 2.4.22-openmosix-2 with the local time() patch, which > I use to avoid processes that check the time() frequently from having to > migrate back to their home node to do it. The only issue with the patch, > as far as I know, is that the time must be correct on all nodes in the > cluster for it to work properly. > > To achieve this, I run NTP on all the nodes in our cluster, using the > 'head' node as a time reference. I've not had any time-related problems > until recently, when the 'head' node lost sync with an external time > reference and the entire cluster was out by three seconds: Something to > do with GPS time not adjusting for leap seconds I believe. However, the > relative time between nodes in the cluster was not affected. > > I don't think these problems are anything to do with checking time > locally on cluster nodes because local elapsed time measurements on a > given node will not be affected by any time offsets from other nodes > unless the processes are migrated there. However, Jeroen used "mosrun" > to force his process onto node32. I agree with Bruce that I'd be very > suspicious about this node's hardware. First thing I would check is that > the node's RAM is not faulty. Have a look at: > > http://howto.x-tend.be/openMosixWiki/index.php/Additions%20to%20the%20FAQ > > I ran Jeroen's script on node32 in our cluster with the local time() > patch enabled, but I don't see the bug: > > ajt@topcat:% cat > time-bug > node=32 > while true > do > mosrun -$node sh -c 'time echo "3 2 60 ^ 2 3700 ^ 1 - |" | dc' > done 2>&1 | sed -n 's/^users*//p' > ajt@topcat:% chmod +x time-bug > ajt@topcat:% ./time-bug > 0m0.970s > 0m0.960s > 0m0.950s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.950s > 0m0.970s > 0m0.960s > 0m0.950s > 0m0.960s > 0m0.960s > 0m0.950s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.950s > 0m0.950s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.950s > 0m0.970s > 0m0.960s > 0m0.960s > 0m0.950s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.950s > 0m0.970s > 0m0.960s > 0m0.950s > 0m0.960s > 0m0.960s > 0m0.950s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.960s > 0m0.950s > 0m0.970s > 0m0.960s > 0m0.970s > 0m0.970s > ... > > > On Feb 5, 2006, at 5:36 PM, Bruce Knox wrote: > > > >> Below is the reference to the sys_time() patch. > >> While the sys_time patch can dramatically affect the speed of some > >> programs, I would think you'd see this slowdown appear on different > >> nodes as the processes ran over an extended period as with your > >> application. With one node behaving this way, I would be very > >> suspicious of that node's hardware. This could very well be a heat > >> related problem. Bruce > >> > >>>> David Black and Tony Travis - sys_time() patch > >> http://sourceforge.net/mailarchive/message.php?msg_id=7184719 > >> allows execution of the function on the current node > >> Programs that frequently execute the sys_time function, such as > >> SETI@home, show remarkable improvements by executing this clock function > >> on the current openMosix node. > >> > >> WARNING: It is critical to sync all the clocks in the cluster. > >> > >> (Typical of many openMosix contributions, this innovations benefits > >> from stepwise improvements. David Black's original patch was improved > >> by Vincent Hanquez (Tab), then by Tony Travis.) > >> > >> > >>>>> "Ian Latter" <[email protected]> 2/5/2006 3:32 PM >>> > >> > >> Hmm, I thought this was new a couple of days ago when I first > >> read it too .. but in my half awake state this morning, it sounds > >> a little like the time() bug .. the one that the patch was written > >> for (the local time patch). Didn't it end up with long on-cpu > >> delays after random run-times (of up to days)? Don't remember > >> who wrote the patch ... I did some experimenting with NTP about > >> 18-24months ago to work the patch into CHAOS tho .. not that > >> this worked. > >> > >> > >> > >> > >> ----- Original Message ----- > >>> From: "Kris Buytaert" <mlkb-1zPVpAQ/[email protected]> > >>> To: > >>> Subject: Re: [openMosix-general] Wierd slowdowns > >>> Date: Sun, 05 Feb 2006 14:59:30 +0100 > >>> > >>> On Thu, 2006-02-02 at 15:56 +0100, Jeroen Demeyer wrote: > >>>> Hello list, > >>>> > >>>> I'm having a very strange problem where some processes just > >> execute much > >>>> slower than they should. Let me explain: sometimes, for no > >> apparent > >>>> reason, a certain process takes much more CPU time than normal. > >> For > >>>> instance, a certain program normally finishes after about 1 > >> CPU-hour, > >>>> but sometimes it might take 30 hours or more. With "CPU-hour", I > >> really > >>>> mean time spent on CPU (not real time), as reported by ps for > >> example. > >>>> > >>> > >>> > >>>> Also note that I don't see any particular correlation between this > >> bug > >>>> and openMosix, so it could be that openMosix doesn't have anything > >> to > >> do > >>>> with it. However, so far I have only seen this problem on the > >> diskless > >>>> 'slave' nodes, and not on the master node (with uptime of 58 days), > >> > >>>> which runs the same kernel. > >>>> > >>>> If anyone has ever seen anything similar to this, please say so. > >>> I've never seen anything similar to this. > >>> > >>> You mention "this node 32" , do you mean that it only happens on 1 > >>> particular node or is it happening on more nodes ? > >>> > >>> Also, have you got an idea about the status of the network at that > >>> moment ? It it more loaded than otherwise ? > >>> > >>> > >>> greetings > >>> > >>> --Kris Buytaert <mlkb-1zPVpAQ/[email protected]> > >>> > >>> > >>> ------------------------------------------------------- > >>> This SF.net email is sponsored by: Splunk Inc. Do you grep through > >> log > >> files > >>> for problems? Stop! Download the new AJAX search engine that makes > >>> searching your log files as easy as surfing the web. DOWNLOAD > >> SPLUNK! > >>> http://sel.as-us.falkag.net/sel? > >> cmd=lnk&kid=103432&bid=230486&dat=121642 > >>> _______________________________________________ > >>> openMosix-general mailing list > >>> openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org > >>> https://lists.sourceforge.net/lists/listinfo/openmosix-general > >>> > >> > >> > >> -- > >> > >> Ian Latter > >> Late night coder .. > >> http://midnightcode.org/ > >> > >> > >> > >> ------------------------------------------------------- > >> This SF.net email is sponsored by: Splunk Inc. Do you grep through log > >> files > >> for problems? Stop! Download the new AJAX search engine that makes > >> searching your log files as easy as surfing the web. DOWNLOAD > >> SPLUNK! > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > >> > >> _______________________________________________ > >> openMosix-general mailing list > >> openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org > >> https://lists.sourceforge.net/lists/listinfo/openmosix-general > >> > >> > >> ------------------------------------------------------- > >> This SF.net email is sponsored by: Splunk Inc. Do you grep through > log files > >> for problems? Stop! Download the new AJAX search engine that makes > >> searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > >> _______________________________________________ > >> openMosix-general mailing list > >> openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org > >> https://lists.sourceforge.net/lists/listinfo/openmosix-general > >> > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through > log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > > _______________________________________________ > > openMosix-general mailing list > > openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org > > https://lists.sourceforge.net/lists/listinfo/openmosix-general > > > > > > > -- > Dr. A.J.Travis, | mailto:[email protected] > Rowett Research Institute, | http://www.rri.sari.ac.uk/~ajt > Greenburn Road, Bucksburn, | phone:+44 (0)1224 712751 > Aberdeen AB21 9SB, Scotland, UK. | fax:+44 (0)1224 716687 > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > openMosix-general mailing list > openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org > https://lists.sourceforge.net/lists/listinfo/openmosix-general > -- Ian Latter Late night coder .. http://midnightcode.org/ ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642