[Helix-server-dev] Re: CR: Fix related to PR 258373: RHEL5 edge has much higher CPU usage than other OSes

Looks good.

Thanks,
Chytanya

At 08:52 AM 3/4/2010, Dean Collins wrote:
>Synopsis
>========
>Fix related to PR 258373: RHEL5 edge has much higher CPU usage than other OSes
>
>Branches: SERVER_14_0_RN, SERVER_CURRENT_RN (HEAD)
>Suggested Reviewer: Anyone
>
>
>Description
>===========
>
>Profiling the server with VTune on Linux revealed a problem causing it
>to spin through the mainloop many hundreds of thousands of times per
>second or more.  I determined that when calling epoll_wait() we were
>often passing in a timeout of zero.  The Linux epoll is so wickedly fast
>that it would immediately pop out most of the time, often with no FDs
>to service.  We would wind the stack back up to the mainloop(), check
>the FDs, scheduler, ischeculer and DispatchQ for callbacks to fire-off,
>find there were few if any to handle, then call back into epoll_wait().
>The overhead of this spinning was accounting for a significant amount
>of user-space and kernel-space CPU time.
>
>The solution is to check the timeout before calling epoll_wait().
>If it's zero, make it 1 so there's at least a small delay each time.
>This small change makes a huge difference.  The timeout is in milliseconds.
>
>The interesting part is the code to implement this already existed,
>but was commented out!  :-)  All the diff does is uncomment the code.
>It had been commented-out since the code was originally checked-in.
>I don't see it mentioned in the CVS commit log, but I seem to recall
>discussing this and we were unsure whether it was the right thing to do so
>left it alone until we had time to test it.  At the time, the server still
>had significant mutex contention that may have made the change degrade
>performance.  Testing has shown that with the v14 server, at least on the
>two server-class systems it was tested on, this change is very necessary.
>In many scenarios it makes as much as a 8x or 10x improvement.
>
>
>Running an on-demand-only uptime test, based off the standard uptime
>configuration, the improvements were reducing overall CPU load
>from about 40% to 5%.  MainLoopIterations were reduced from
>~1,000,000/sec to 2,500/sec.
>Before: 
>http://plots.coreqa.prognet.com/dcollins/archive/pr258373/servlab/defender/Logs.1-no-cloaking-demand-serveronly/server.html
>After: 
>http://plots.coreqa.prognet.com/dcollins/archive/pr258373/servlab/defender/Logs.18-no-cloaking-demand-serveronly-vtune-testfix1/server.html
>
>A more isolated test easier to analyze that showed the problem more
>clearly was to just use the 3.3Mbps mp4 clip from the uptime, with
>36 players.  In this scenario, CPU usage was reduced from about 35% to 3%.
>MainLoopIterations were reduced from ~1,000,000/sec to ~2,000/sec.
>Mutex Collisions went from ~6,000/sec to ~1,000/sec.
>Before: 
>http://plots.coreqa.prognet.com/dcollins/archive/pr258373/servlab/defender/Logs.13-no-cloaking-demand-serveronly-mp4-3320k_hinted-udp-36players-vtune/server.html
>After: 
>http://plots.coreqa.prognet.com/dcollins/archive/pr258373/servlab/defender/Logs.17-no-cloaking-demand-serveronly-mp4-3320k_hinted-udp-36players-vtune-testfix1/server.html
>
>Lots of other stats also show significant improvements, as well as my
>VTune data.  Despite the MP4 test above, there's nothing MP4-specific
>about the fix.  It should improve just about anything the server does
>to varying degrees.
>
>
>Files Affected
>==============
>server/engine/core/pub/platform/unix/servcallback.h
>
>
>Testing Performed
>=================
>
>Unit Tests:
>- N/A
>
>Integration Tests:
>- Uptime tested in servlab (on-demand only) and in a full uptime in 
>the QA rig.
>
>Leak Tests:
>- N/A
>
>Performance Tests:
>- N/A
>
>Platforms Tested: linux-rhel5-i686
>Build verified: linux-rhel5-i686
>
>
>QA Hints
>========
>* N/A
>