[Helix-server-dev] CR: Fix related to PR 258373: RHEL5 edge has much higher CPU usage than other OSes

Synopsis
========
Fix related to PR 258373: RHEL5 edge has much higher CPU usage than other OSes

Branches: SERVER_14_0_RN, SERVER_CURRENT_RN (HEAD)
Suggested Reviewer: Anyone


Description
===========

Profiling the server with VTune on Linux revealed a problem causing it
to spin through the mainloop many hundreds of thousands of times per
second or more.  I determined that when calling epoll_wait() we were
often passing in a timeout of zero.  The Linux epoll is so wickedly fast
that it would immediately pop out most of the time, often with no FDs
to service.  We would wind the stack back up to the mainloop(), check
the FDs, scheduler, ischeculer and DispatchQ for callbacks to fire-off,
find there were few if any to handle, then call back into epoll_wait().
The overhead of this spinning was accounting for a significant amount
of user-space and kernel-space CPU time.

The solution is to check the timeout before calling epoll_wait().
If it's zero, make it 1 so there's at least a small delay each time.
This small change makes a huge difference.  The timeout is in milliseconds.

The interesting part is the code to implement this already existed,
but was commented out!  :-)  All the diff does is uncomment the code.
It had been commented-out since the code was originally checked-in.
I don't see it mentioned in the CVS commit log, but I seem to recall
discussing this and we were unsure whether it was the right thing to do so
left it alone until we had time to test it.  At the time, the server still
had significant mutex contention that may have made the change degrade
performance.  Testing has shown that with the v14 server, at least on the
two server-class systems it was tested on, this change is very necessary.
In many scenarios it makes as much as a 8x or 10x improvement.


Running an on-demand-only uptime test, based off the standard uptime
configuration, the improvements were reducing overall CPU load 
from about 40% to 5%.  MainLoopIterations were reduced from
~1,000,000/sec to 2,500/sec.  
Before: http://plots.coreqa.prognet.com/dcollins/archive/pr258373/servlab/defender/Logs.1-no-cloaking-demand-serveronly/server.html
After: http://plots.coreqa.prognet.com/dcollins/archive/pr258373/servlab/defender/Logs.18-no-cloaking-demand-serveronly-vtune-testfix1/server.html

A more isolated test easier to analyze that showed the problem more
clearly was to just use the 3.3Mbps mp4 clip from the uptime, with
36 players.  In this scenario, CPU usage was reduced from about 35% to 3%.
MainLoopIterations were reduced from ~1,000,000/sec to ~2,000/sec.
Mutex Collisions went from ~6,000/sec to ~1,000/sec.
Before: http://plots.coreqa.prognet.com/dcollins/archive/pr258373/servlab/defender/Logs.13-no-cloaking-demand-serveronly-mp4-3320k_hinted-udp-36players-vtune/server.html
After: http://plots.coreqa.prognet.com/dcollins/archive/pr258373/servlab/defender/Logs.17-no-cloaking-demand-serveronly-mp4-3320k_hinted-udp-36players-vtune-testfix1/server.html

Lots of other stats also show significant improvements, as well as my
VTune data.  Despite the MP4 test above, there's nothing MP4-specific
about the fix.  It should improve just about anything the server does
to varying degrees.


Files Affected
==============
server/engine/core/pub/platform/unix/servcallback.h


Testing Performed
=================

Unit Tests:
- N/A

Integration Tests:
- Uptime tested in servlab (on-demand only) and in a full uptime in the QA rig.

Leak Tests:
- N/A

Performance Tests:
- N/A

Platforms Tested: linux-rhel5-i686
Build verified: linux-rhel5-i686


QA Hints
========
* N/A

_______________________________________________
Helix-server-dev mailing list
[email protected]
http://lists.helixcommunity.org/mailman/listinfo/helix-server-dev

Index: server/engine/core/pub/platform/unix/servcallback.h
===================================================================
RCS file: /cvsroot/server/engine/core/pub/platform/unix/servcallback.h,v
retrieving revision 1.16
diff -u -r1.16 servcallback.h
--- server/engine/core/pub/platform/unix/servcallback.h	28 Apr 2009 16:33:42 -0000	1.16
+++ server/engine/core/pub/platform/unix/servcallback.h	4 Mar 2010 16:04:07 -0000
@@ -1207,8 +1207,11 @@
         return m_nReadyFDs;
     }
 
-    // if (!timeout)
-	// timeout = 1;
+    // If timeout is zero epoll_wait() returns so fast, whether we have any fds
+    // to handle or not, that we often end up spinning through the mainloop many
+    // thousands or millions of times per second.  epoll_wait() is quick!
+    if (!timeout)
+        timeout = 1;
 
     m_nReadyFDs = epoll_wait(m_nPollFD, m_pEPollEvents, EPOLL_EVENTS_MAX, timeout);
     if (m_nReadyFDs < 0 && errno != EINTR && errno != 0)