Re: Oprofile vs. perf vs. gperf

On 06/04/2015 07:01 AM, Michael Petlan wrote:
> On Thu, 2015-06-04 at 07:10 +0000, Mahmood N wrote:
>> Hi,
>> What is the difference among Oprofile, perf-tools and gperf-tools? I
>> didn't find an study comparing them. Which one is mostly accurate for
>> AMD processors?

Hi Mahmood,

It would be useful to describe what you are interested in measuring and then figure out what tools and techniques would best accomplish that.

> 
> Hi,
> 
> I don't know much about gprof, so maybe someone should add some more
> information...
> 
> The aim of OProfile is counting, profiling (sampling), code annotation
> and callgraph generation, all that based on PMU events generated by CPU.
> Historically, it supported some software TIMER interrupts, but now,
> since opcontrol legacy mode has been replaced by operf/ocount tools, the
> oprofile's source of events is PMU only. OProfile now uses kernel
> backend for it, it's the same backend as perf uses.
> 
> Perf tool came later as a part of kernel. Before that oprofile worked in
> the legacy mode (opcontrol), so some daemon and kernel module had to be
> enabled before the profiling may have started, etc. When perf came with
> the kernel backend, oprofile switched to using it as well.
> 
> Which one is more accurate? Hard to say, in my opinion they should be
> almost at the same level.

Yes, both operf and perf are using the same underlying mechanism to collect data.  A better question is what events should be used.  The performance monitoring hardware on particular processors may inaccurately count certain events.  For some work general order of magnitude numbers are accurate enough.  In other cases need to have number accurate to a couple of percent.  Also there are performance events for shared resources such as cache so other processes on the system may affect a process being monitored.  A good source of information about what events should be monitored are in the Intel and AMD software optimization manuals.  There are links to them toward the bottom of my Red Hat people page (http://people.redhat.com/wcohen/).

> 
> Perf can use PMU as well, but it is not the main aim of it. So it has
> some events predefined (see `perf list hw hwcache` command) and the rest
> has to be specified by raw numbers like "-e r410e". So that means, that
> perf can use all the events oprofile can, but many of them are
> unfriendly to use and you just have to go through your CPU vendor's
> documentation to see what numbers to put there.

The kernel perf significantly subsets the pmu events and maps those to generic event names such as cycles and llc-misses (last level cache misses).  The performance monitoring hardware is very machine specific, so you need to be careful when comparing data across machines.  The llc-misses on one processor may be very different to the llc-misses on another processor due to one having only l2 cache and another have a l3 cache.  In some cases you may need to resort to using the events outside the perf defined ones and those are bit of a pain to configure with the raw (-e rNNNN) options.  OProfile provides names for all the events "ophelp" will list out the available events on the machine.

> 
> Perf can also use so called "tracepoint" events (see `perf list
> tracepoint` (you have to be root to run it)), which are mapped to some
> kernel trace points. You can also probe functions (similar to
> systemtap), trace syscalls (like strace), do some NUMA benchmarking and
> many other things.
> 
> Compare the following:
> 
> ocount ls
> perf stat -e cycles ls
> 
> The numbers should be similar. If you see differences there, it's not
> because perf or oprofile are imprecise compared to the other tool, but
> the reason is rather that in each run is a bit different and CPU can
> pre-cache something, etc.
> 
> As I understand, gprof is a bit different tool which needs to be used
> also at the compilation time, so the application is instrumented by it
> before it is built.
> 
> When you are profiling with `operf` or `perf record` you don't need
> anything special, just if you have the debuginfo, you can see the
> symbols in the report, otherwise you will see addresses...

If possible, code should be compile with debugging information ('-g' option in gcc) to provide that mapping information from addresses to source code.  The debuginfo doesn't affect the execution of the code.  Linux distributions such as Fedora pull that information out and put it into separate debuginfo rpms to make it easy to do analysis on the official executables.  Things are a little more complicated with java code.  perf doesn't have machinery to map addresses to methods.  OProfile does have the ability to map samples to java methods, but you will need to start up the java program with an agentlib (http://oprofile.sourceforge.net/doc/setup-jit.html).

-Will

> 
> I hope I have explained it a bit.
> 
> Regards,
> Michael
> 
>>  
>> Regards,
>> Mahmood
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> oprofile-list mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/oprofile-list
> 
> 
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> oprofile-list mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/oprofile-list
> 


------------------------------------------------------------------------------