RE: Anyone interested in having additional debugging support in NPTL via a tracing facility?

Hi,

Thanks for opening up this topic.
This is very much required while debugging
pthreads in multi-threaded environment. The
fact that now with 2.6 kernel, NPTL has improved
a lot compared to eariler versions, i think this 
is right time to have this feature which is required
for many applications debugging.

Meanwhile i have some clarifications w.r.t current versions
of NPTL debug support?

1. Is it possible to list all threads which are running in current 
process?

2. If breakpoint is set for one of the thread, will other threads
continue the execution even though break-point hit thread stops?

3. When one thread is stopped through breakpoint, is it possible to get 
stack trace of all threads?

4. When thread-break point is hit, is it possible to know the status of
synchronization objects like mutexes and CVs etc?
 

Thanks in advance
-MS



-----Original Message-----
From: Jerry Harrow [mailto:[email protected]]
Sent: Friday, January 30, 2004 7:43 AM
To: [email protected]
Subject: RE: Anyone interested in having additional debugging support in
NPTL via a tracing facility?


Sorry for the re-send, but appearently the mail system truncated the 
previous message after the first paragraph.  Hopefully the whole thing 
comes through this time.

-Jer
=============================================

I've had some on & off discussions with Ulrich about adding detailed
event tracing support to NPTL for quite a while.  While favoring the
idea in general, Ulrich isn't convinced that people (i.e. you all) have
much of a desire for it.  When I last posted a proposal in this regard
(https://listman.redhat.com/archives/phil-list/2002-December/msg00043.ht
ml)
 there was only one reply -- obviously not an overwhelming
endorsement.  I've done some updates on the original proposal and
published it to
ftp://ftp.compaq.com/pub/products/visualthreads/Linux/NPTLTracingV2.html


Is there any significant interest in having support for this kind of 
tracing/debug in NPTL?  


Let me summarize the concepts:

* Add call-outs at most significant state changes in NPTL.  This would
include attempt to lock a mutex, blocking while locking a mutex,
acquiring a mutex, thread creation, thread termination, etc.  

* The call-outs are directed to a replaceable shared library which
allows activity to be communicated, or analyzed in whatever manner is
desired.  This is basically an extension of the hooks libthread_db.so
uses to support GDB, but it can invoke locally loaded code to handle the
event.

* A simple text log writer would be provided with NPTL to make it useful
out-of-the-box and as an example of how to use the interfaces.  Another
relatively simple tracing engine that could be built is a shared-memory
ring buffer.  So by just defining an environment variable, you could get
a text file describing all the threading activity of an application.

Potential Concerns some people may have:

* Won't having all these event points in the code cause reduced
performance?  

When we built similar capability into the Tru64 UNIX pthreads library it
was done by having a bit test before each event, and it didn't result in
any significant impact -- although in that case our compilers knew how
to move the event code to "cold" sections of the library through
profile-directed optimization.  When HP-UX added similar tracing to its
thread library, it supplied a separate version of libpthread.so which
exactly matches the default library except with tracing support.  In
that case there is no runtime overhead unless LD_LIBRARY_PATH is used to
redirect to the tracing library.

* Is this really useful without writing a bunch of code?

My suggestion is to provide a simple tracing engine with NPTL that
writes out events to a file, so that could be edited/searched with
common tools like sed/awk/perl etc.  Using this it is very easy to find
pthread functions that are failing for unexpected reasons (how many of
you always check the status of pthread_mutex_lock?).  To write a tracing
engine it only takes 4 routines (basically init, fini, re-init on fork,
and event write) so it is very easy to build special-purpose event
handling to help debug a particular problem in your application
(especially when building from an example).   For more details and an
example see the updated proposal.


So if there is interest in this type of facility, I'd be glad to help
put it in place; either in the proposed form or perhaps a simplified
version.  The existing proposal is very detailed and almost completely
isolates the trace engine from the libpthread.so implementation details.
Though it may seem complex, the NPTL version of these changes is the
addition of about 81 lines of code plus one module to load the trace
engine, and format the events. 

-Jer

=====================================================

Jerry J. Harrow, Jr. 
Hewlett-Packard Company
110 Spit Brook Rd 
Nashua, NH 03062-2698 
USA
Jerry.Harrow at hp dot com
Telephone: 603.884.2193


-- 
Phil-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/phil-list