Re: ERROR: operf_read::convertPerfData

On Mon, 2015-06-01 at 15:17 -0700, Daniel Wilkerson wrote:
> On Mon, Jun 1, 2015 at 3:28 AM, Michael Petlan <[email protected]> wrote:
> >
> > 1) Command `opreport --symbols` without the "oprofile_data" argument.
> > The "oprofile_data" dir is used by default. If you want to specify some
> > other dir, you'd need to use the proper option: "--session-dir=dir". By
> > putting "oprofile_data" there without any switch, opreport thinks, that
> > "./oprofile_data" is the name of the executed binary. Since there are no
> > samples of the binary "./oprofile_data", you get the "no samples"
> > message.
> 
> Ok, so that worked for the single-threaded toy program I wrote;
> thanks.  However the large multithreaded program I ran it under
> crashes *while* it is running; see below.
> 

Yes, but we see, that oprofile works as it should and the issue could be
rather a bug in it than a wrong use case.

> > 2) Check something about your environment:
> >
> > * ophelp -r
> > --> You should see what how oprofile recognizes your HW (or HW support).
> > It is "Intel Ivy Bridge microarchitecture" in my case.
> 
> $ ophelp -r
> AMD64 family15h
> 
> $ perf stat ls
> The program 'perf' is currently not installed.  You can install it by typing:
> sudo apt-get install linux-tools-common

Yes, it is an another program, I recommend to install it in order to
check, whether the bug is in oprofile or in the backend.

`perf stat something` should produce similar output as `ocount
something`. But that is not necessary, since we know, that the counters
work correctly. We need rather `perf record` tool which works similar to
operf (see below).

> 
> $ ophelp -d
> CPU_CLK_UNHALTED:100000:0:1:1
> 
> 
> > The second value (100000) is the sample rate. It means that when ocount reports
> > circa 2200k of events, you should get around 2200k/100k = 22 samples.
> >
> > So even /bin/ls should generate enough samples to be detected by opreport.
> >
> > * operf ls
> > * opreport
> >
> > If the previous things work, try to run `ls` under operf, there should really be
> > samples.
> 
> > Hope it helps.
> 
> I appreciate all of the detail above, and it does seem to work as you
> suggest for ls, but below as you can see for my multithreaded program
> I am getting a failure while it is running.  Should I do something to
> "lower the sample rate" to prevent that crash?  Again, see "appendix
> A" below.
> 

Yes, that could be an option, I would try to do it by:

operf -e CPU_CLK_UNHALTED:4000000 ./your_program

I would try also perf, after installing the additional package:

perf record -e cycles ./your_program
perf report --stdio

Note that `ulimit -n` could be a problem here, since perf needs one FD
perf event per thread.

You might also try to use `operf --verbose=debug ./your_program` in
order to get more logs.

I am curious whether perf record is OK with your multithreaded
application.

In case that perf record works fine with it, could you please try to
create a reproducer for the possible oprofile bug?

Thanks.
Michael

> Daniel
> 
> >> ---- appendix A
> >>
> >> ERROR: operf_read::convertPerfData quitting. Bad data read from pipe.
> >> Closing read end of data pipe. operf-record process will stop with SIGPIPE (13).
> >> Try lowering the sample frequency to avoid this error; e.g., double the 'count'
> >> value in your event specification.
> >>
> >>
> >>  * * * ATTENTION: The kernel lost 4260291 samples. * * *
> >> Decrease the sampling rate to eliminate (or reduce) lost samples.
> >>
> >> WARNING: Lost samples detected! See
> >> /data2/dsw/wyd_runs/runE700/oprofile_data/samples/operf.log for
> >> details.
> >> operf-record process killed by signal 13
> >> Error running profiler
> >> make[1]: *** [0.align] Error 2
> >> make[1]: Leaving directory `/data2/dsw/wyd_runs/runE700'
> >> make: *** [nohup/go1] Error 2
> >> ^C
> >> [1]+  Exit 2                  make go



------------------------------------------------------------------------------