Re: Fwd: Potential deadlock in operf when using --pid

Thanks for reviewing the patches, Will,

I am trying to remember the code....  As far as I understand, the comm
pipes are used only for newly spawned processes.  I need to investigate
more to see if they can be removed.

It is less likely, but theoretically it is possible that the comm pipe gets
full first and as a result the sample output pipe gets full next.  In any
case, my patches should solve the issue.


Thanks,
Rei Odaira

2017-06-30 14:50 GMT-05:00 William Cohen <[email protected]>:

> On 06/24/2016 08:06 PM, 大平怜 wrote:
> > I forgot to include oprofile-list.
> >
> >
> > Regards,
> > Rei Odaira
> >
> > ---------- Forwarded message ----------
> > From: *大平怜* <[email protected] <mailto:[email protected]>>
> > Date: 2016-06-24 19:04 GMT-05:00
> > Subject: Re: Potential deadlock in operf when using --pid
> > To: William Cohen <[email protected] <mailto:[email protected]>>
> >
> >
> > Hi,
> >
> > Now I come back to this problem.
> >
> > I am attaching two patches to solve it.  They apply to the latest master
> in the git repository.  Please apply oprofile_avoid_deadlock_1.patch
> first and then oprofile_avoid_deadlock_2.patch. The first patch is just
> for refactoring the code to make the second patch possible, and the second
> patch is for the actual problem solving.
> >
> > So, the problem is that the operf-record process is blocked on a write
> to the sample output pipe when the pipe is full, while the operf-read
> process, which is the consumer of the pipe, cannot read from the pipe
> because it is blocked on a read from the comm pipe, to which the
> operf-record process is supposed to write.  Please refer to the previous
> emails for the detailed problem description.
> >
> >             |<--sample output pipe--|
> > operf-read  |-------comm pipe------>|  operf-record
> >             |<------comm pipe-------|
> >
> > My solution is to make the sample output pipe non-blocking on the
> operf-record side.  When the sample output pipe is full, instead of waiting
> on it, the operf-record process polls both the sample output pipe and the
> (read-end of) comm pipe.  If there is a message in the comm pipe from the
> operf-read process, the operf-record process handles it and writes a
> response to the comm pipe, so that the operf-read process can finish being
> blocked on the read from the comm pipe and eventually can consume the
> sample output pipe.
> >
> > There are some other ways to solve this problem, so any comments are
> welcome.  With my patches, you can finish profiling
> oprofile_multithread_test by a single ctrl-c. They passed all the tests in
> oprofile-tests.
> >
> >
> > Regards,
> > Rei Odaira
> >
>
>
>
>
> Hi Da Rei,
>
> Sorry for missing the patches to address rapid thread
> creation/deletion deadlock issue.  I have been taking a look at them.
> As a sanity check I ran the testsuite before and after the patches and
> things look the same.
>
> Test results before the patches:
>
>                 === oprofile Summary ===
>
> # of expected passes            149
>
> kernel version: 4.11.5-200.fc25.x86_64
> userspace version: oprofile 1.2.0git compiled on Jun 29 2017 13:59:35
>
>
> Test results after the patches:
>
>                 === oprofile Summary ===
>
> # of expected passes            149
>
> kernel version: 4.11.5-200.fc25.x86_64
> userspace version: oprofile 1.2.0git compiled on Jun 29 2017 14:10:33
>
>
> The patches does appear to help with the issue demonstrated with the
> reporducer oprofile_mutithread_test with operations per thread of
> 1000000.  However, with smaller number of operations per thread
> settings (100000) operf has issue getting connected:
>
> $ operf --pid 7555
> !!!! No samples collected !!!
> The target program/command ended before profiling was started.
> operf record init failed
> usage: operf [ options ] [ --system-wide | --pid <pid> | [ command [ args
> ] ] ]
> Error running profiler
>
>
> Or if it is able to connect (the process was paused with cntl-z and
> then put back in the foreground after oprofile had attached):
>
> $ operf --pid 7555
> ...
> Unable to collect samples for forked process 17240. Process may have ended
> before recording could be started.
> Unable to collect samples for forked process 17241. Process may have ended
> before recording could be started.
> Unable to collect samples for forked process 17242. Process may have ended
> before recording could be started.
> Unable to collect samples for forked process 17243. Process may have ended
> before recording could be started.
> Failed to mmap kernel profile data.
> This issue may be caused by a non-root user running multiple operf
> sessions simultaneously. Try running as root or increasing the value of
> /proc/sys/kernel/perf_event_mlock_kb to resolve the problem.
>
> Unable to collect samples for forked process 17245. Process may have ended
> before recording could be started.
>
>
> Is is notable that perf has similar problems it is unable to attach to
> quickly spawning processes
>
> $ perf record --pid 7555
> Error:
> The sys_perf_event_open() syscall returned with 3 (No such process) for
> event (cycles).
> /bin/dmesg may provide additional information.
> No CONFIG_PERF_EVENTS=y kernel support configured?
> [wcohen@santana oprofile]$ perf record --pid 7555
> ^C[ perf record: Woken up 81 times to write data ]
>
> However "perf record --pid 7555" is able to collect data without issue
> if the 7555 is paused when "perf record" starts up.  I have let the
> perf developers know about this issue.
>
>
> The portion of oprofile the patches affect is not easy to follow, so I
> am still trying to make sure I fully understand the issue and that the
> patches address the issue without introducing some other issue.
> Earlier email has the following diagram:
>
>
>             |<--sample output pipe--|
> operf-read  |-------comm pipe------>|  operf-record
>             |<------comm pipe-------|
>
>
> The data in the sample "sample output pipe" stream includes the
> PERF_RECORD_FORK events.  The operf-read processes the
> PERF_RECORD_FORK events in the stream and sends pid back up the comm
> pipe for operf-record add another thread.
>
> Is the comm pipe only being used to communicate the new processes to be
> monitored?
>
> Would it be possible to have operf-record just do that directly rather
> than having to make a round trip from operf-record to operf-read and
> back to operf-record?  Seems like there is a lot of overhead when
> starting to monitoring a new thread.
>
> The patches implement polling to avoid the opef-record getting hung
> and in turn causing operf-read getting stuck when sample output pipe
> gets full.  Is there any way that the comm pipe might similarly get
> full and cause a hang?
>
> -Will
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
oprofile-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/oprofile-list