Re: operf jit problem. No anon samples?
Maynard Johnson <[email protected]>
| Newsgroups | gmane.linux.oprofile |
|---|---|
| Message-ID | <[email protected]> |
On 10/16/2014 11:31 AM, Maurice Marks wrote: > I don't think that was the problem. Just in case it was involved I tried changing the event count to 5000000. That reduced the sample count but the problem of missing anon samples was just the same. There was one other thing that worried me. Most of my testing was on a VM (VMware Player 6.0.3) which, although it makes the performance counters available, isn't perfect. However operf works just fine on it with non Jit applications. > [snip] > dad@piled:/media/dad/work$ opreport > Using /media/dad/work/oprofile_data/samples/ for samples directory. > CPU: AMD64 family15h, speed 4000 MHz (estimated) > Counted CPU_CLK_UNHALTED events (CPU Clocks not Halted) with a unit mask of 0x00 (No unit mask) count 1000000 > CPU_CLK_UNHALT...| > samples| %| > ------------------ > 19536 100.000 lli > CPU_CLK_UNHALT...| > samples| %| > ------------------ > 19339 98.9916 ld-2.19.so <http://ld-2.19.so> > 60 0.3071 no-vmlinux > 48 0.2457 libLLVMCore.so > 37 0.1894 libLLVMCodeGen.so > 15 0.0768 libLLVMSelectionDAG.so > 10 0.0512 libLLVMSupport.so > 6 0.0307 libLLVMX86CodeGen.so > 5 0.0256 libc-2.19.so <http://libc-2.19.so> > 5 0.0256 libpthread-2.19.so <http://libpthread-2.19.so> > 3 0.0154 libstdc++.so.6.0.20 > 2 0.0102 lli > 2 0.0102 libLLVMAsmParser.so > 1 0.0051 libLLVMAnalysis.so > ... > [snip] > Other possibilities? > > Is there an oprofile test for the jit functionality I can try? Maurice, Please try profiling the LLVM app without the oprofile JIT support. Not sure how you do that, but I presume you know. The results you get should look similar to what I get when I profile a java app without passing the -agentlib or -agentpath to the JVM; i.e., ------------------------------------------------------- [mpjohn@oc1757000783 myJavaStuff]$ opreport Using /home/mpjohn/myJavaStuff/oprofile_data/samples/ for samples directory. CPU: Intel Sandy Bridge microarchitecture, speed 2401 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000 CPU_CLK_UNHALT...| samples| %| ------------------ 140438 100.000 java CPU_CLK_UNHALT...| samples| %| ------------------ 133294 94.9131 anon (tgid:11436 range:0x7fc4ed000000-0x7fc4ed26ffff) 4387 3.1238 libjvm.so 2301 1.6384 no-vmlinux 211 0.1502 libc-2.12.so 90 0.0641 libpthread-2.12.so 66 0.0470 libzip.so 43 0.0306 ld-2.12.so 20 0.0142 [vdso] (tgid:11436 range:0x7fffae4ea000-0x7fffae4eafff) 18 0.0128 [vsyscall] (tgid:11436 range:0xffffffffff600000-0xffffffffff600fff) 5 0.0036 libjava.so 1 7.1e-04 libnss_files-2.12.so 1 7.1e-04 libjli.so 1 7.1e-04 libverify.so ------------------------------------------------------- Note the "anon" samples. The JVM's JIT compiler puts JITed code into anonymous memory mappings. The oprofile JIT support works on the assumption that samples from JITed code will be associated with those anonymous memory mappings. So, if after profiling LLVM without the oprofile JIT agent support, you are not seeing any "anon" samples, that tells me that LLVM is not putting JITed code in anonymous memory mappings, and oprofile's JIT support won't work for it. I have verified that oprofile's support for Java JITed code works fine on Ubuntu. Does oprofile's support for LLVM JITed code work on any other platform? -Maynard > > Oprofile seems to be getting the events OK, and counting them, but just not saving them as anon samples when they are outside the DSOs. > > Has anyone else on the list been using oprofile with a non-Java jit configuration? > > > > > On Wed, Oct 15, 2014 at 6:53 PM, Maynard Johnson <[email protected] <mailto:[email protected]>> wrote: > > On 10/15/2014 04:25 PM, Maurice Marks wrote: > > I'm trying to profile non Java Jit code (using llvm's built in oprofile interface) on Ubuntu 14.04 using operf. > > I built the latest git version of oprofile just to be sure I'm up to date. > > > > What I see is that there are lots of jit samples counted, but rather than being attributed to anon or to (hopefully) <pid>.jo they are > > counted against one of the .so files. > > > > Like this: > > CPU: Intel Haswell microarchitecture, speed 3498 MHz (estimated) > > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000 > > CPU_CLK_UNHALT...| > > samples| %| > > ------------------ > > 20792 100.000 lli > > CPU_CLK_UNHALT...| > > samples| %| > > ------------------ > > 20634 99.2401 ld-2.19.so <http://ld-2.19.so> <http://ld-2.19.so> > > 86 0.4136 no-vmlinux > > 27 0.1299 libLLVMCore.so > > 16 0.0770 libLLVMCodeGen.so > > 10 0.0481 libLLVMSelectionDAG.so > > 4 0.0192 libLLVMSupport.so > > 3 0.0144 libLLVMJIT.so > > 3 0.0144 libLLVMX86Desc.so > > 2 0.0096 libLLVMScalarOpts.so > > 2 0.0096 libLLVMX86CodeGen.so > > 1 0.0048 lli > > 1 0.0048 libc-2.19.so <http://libc-2.19.so> <http://libc-2.19.so> > > 1 0.0048 libpthread-2.19.so <http://libpthread-2.19.so> <http://libpthread-2.19.so> > > 1 0.0048 libLLVMAnalysis.so > > 1 0.0048 libLLVMAsmParser.so > > > > ~99% of the samples are actually in the Jit'd code, not in ld-2.19.so <http://ld-2.19.so> <http://ld-2.19.so>. > > > > Debugging I see that the opjitagent calls are being made correctly for the jit'd routines, and jitdump files are being generated. > > And opjitconv runs and deletes the jitdump files at the end. But because there are no "anon" samples > > nothing is reported. > > > > With-Vdebug in operf I see: > > .... > > profiled app ended normally. > > operf recording finished. > > Total bytes recorded from perf events: 841464 > > operf-record process returned OK > > * * * * WARNING: Profiling rate was throttled back by the kernel * * * * > > The number of samples actually recorded is less than expected, but is > > probably still statistically valid. Decreasing the sampling rate is the > > best option if you want to avoid throttling. > > operf_read: Total bytes received from operf_record process: 841280 > > Calling _do_jitdump_convert > > start time/end time is 1413408126/1413408137 > > opjitconv: Ending with rc = 2. This code is usually OK, but can be useful for debugging purposes. > > JIT dump processing complete. > > operf-read process returned OK > > > > > > I'm probably doing something wrong. But I'm not sure what. > > Any ideas? > Hi, Maurice. Cool, I didn't know that LLVM had an interface to oprofile's JIT support. Googling, I see that this have been around for a while. So, the first question is have you ever gotten this to work (yourself) in the past? If so, then please provide details (distro, oprofile version, processor type, etc.). > > What really jumps out at me is that according to the opjitconv start time/end time debug message above, your profile run was 11 seconds, but you only got a total of 20792, while sampling at a rate of one sample per 100,000 CPU_CLK_UNHALTED events! I've never seen the kernel do throttling like that. There may be a kernel issue there, but the first thing I would try is back off on the sampling rate ... try something like one sample every 500,000 CPU_CLK_UNHALTED events. > > -Maynard > > > > > > > > > > > > ------------------------------------------------------------------------------ > > Comprehensive Server Monitoring with Site24x7. > > Monitor 10 servers for $9/Month. > > Get alerted through email, SMS, voice calls or mobile push notifications. > > Take corrective actions from your mobile device. > > http://p.sf.net/sfu/Zoho > > > > > > > > _______________________________________________ > > oprofile-list mailing list > > [email protected] <mailto:[email protected]> > > https://lists.sourceforge.net/lists/listinfo/oprofile-list > > > > > > > -- > Not sent from my Blackberry, Raspberry or Gooseberry! > > > ------------------------------------------------------------------------------ > Comprehensive Server Monitoring with Site24x7. > Monitor 10 servers for $9/Month. > Get alerted through email, SMS, voice calls or mobile push notifications. > Take corrective actions from your mobile device. > http://p.sf.net/sfu/Zoho > > > > _______________________________________________ > oprofile-list mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/oprofile-list > ------------------------------------------------------------------------------ Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho