Re: operf jit problem. No anon samples?

On 10/15/2014 04:25 PM, Maurice Marks wrote:
> I'm trying to profile  non Java Jit code  (using llvm's built in oprofile interface) on Ubuntu 14.04 using operf.
> I built the latest git version of oprofile just to be sure I'm up to date.
> 
> What I see is that there are lots of jit samples counted, but rather than being attributed to anon or to (hopefully) <pid>.jo they are
> counted against one of the .so files.
> 
> Like this:
> CPU: Intel Haswell microarchitecture, speed 3498 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
> CPU_CLK_UNHALT...|
>   samples|      %|
> ------------------
>     20792 100.000 lli
>     CPU_CLK_UNHALT...|
>       samples|      %|
>     ------------------
>         20634 99.2401 ld-2.19.so <http://ld-2.19.so>
>            86  0.4136 no-vmlinux
>            27  0.1299 libLLVMCore.so
>            16  0.0770 libLLVMCodeGen.so
>            10  0.0481 libLLVMSelectionDAG.so
>             4  0.0192 libLLVMSupport.so
>             3  0.0144 libLLVMJIT.so
>             3  0.0144 libLLVMX86Desc.so
>             2  0.0096 libLLVMScalarOpts.so
>             2  0.0096 libLLVMX86CodeGen.so
>             1  0.0048 lli
>             1  0.0048 libc-2.19.so <http://libc-2.19.so>
>             1  0.0048 libpthread-2.19.so <http://libpthread-2.19.so>
>             1  0.0048 libLLVMAnalysis.so
>             1  0.0048 libLLVMAsmParser.so
> 
> ~99% of the samples are actually in the Jit'd code, not in ld-2.19.so <http://ld-2.19.so>.
> 
> Debugging I see that the opjitagent calls are being made correctly for the jit'd routines, and jitdump files are being generated.
> And opjitconv runs and deletes the jitdump files at the end. But because there are no "anon" samples
> nothing is reported.
> 
> With-Vdebug in operf I see:
> ....
> profiled app ended normally.
> operf recording finished.
> Total bytes recorded from perf events: 841464
> operf-record process returned OK
> * * * * WARNING: Profiling rate was throttled back by the kernel * * * *
> The number of samples actually recorded is less than expected, but is
> probably still statistically valid.  Decreasing the sampling rate is the
> best option if you want to avoid throttling.
> operf_read: Total bytes received from operf_record process: 841280
> Calling _do_jitdump_convert
> start time/end time is 1413408126/1413408137
> opjitconv: Ending with rc = 2. This code is usually OK, but can be useful for debugging purposes.
> JIT dump processing complete.
> operf-read process returned OK
> 
> 
> I'm probably doing something wrong. But I'm not sure what.
> Any ideas?
Hi, Maurice.  Cool, I didn't know that LLVM had an interface to oprofile's JIT support. Googling, I see that this have been around for a while.  So, the first question is have you ever gotten this to work (yourself) in the past?  If so, then please provide details (distro, oprofile version, processor type, etc.). 

What really jumps out at me is that according to the opjitconv start time/end time debug message above, your profile run was 11 seconds, but you only got a total of 20792, while sampling at a rate of one sample per 100,000 CPU_CLK_UNHALTED events! I've never seen the kernel do throttling like that. There may be a kernel issue there, but the first thing I would try is back off on the sampling rate ... try something like one sample every 500,000 CPU_CLK_UNHALTED events.

-Maynard
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> 
> 
> 
> _______________________________________________
> oprofile-list mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/oprofile-list
> 


------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho