Re: Anyone interested in having additional debugging support in NPTL via a tracing facility?

I think that this would be very useful for anyone who needs to dig into 
multi-threaded performance/debugging.

Ric

Jerry Harrow wrote:

>Sorry for the re-send, but appearently the mail system truncated the 
>previous message after the first paragraph.  Hopefully the whole thing 
>comes through this time.
>
>-Jer
>=============================================
>
>I've had some on & off discussions with Ulrich about adding detailed
>event tracing support to NPTL for quite a while.  While favoring the
>idea in general, Ulrich isn't convinced that people (i.e. you all) have
>much of a desire for it.  When I last posted a proposal in this regard
>(https://listman.redhat.com/archives/phil-list/2002-December/msg00043.html)
> there was only one reply -- obviously not an overwhelming
>endorsement.  I've done some updates on the original proposal and
>published it to
>ftp://ftp.compaq.com/pub/products/visualthreads/Linux/NPTLTracingV2.html
>
>
>Is there any significant interest in having support for this kind of 
>tracing/debug in NPTL?  
>
>
>Let me summarize the concepts:
>
>* Add call-outs at most significant state changes in NPTL.  This would
>include attempt to lock a mutex, blocking while locking a mutex,
>acquiring a mutex, thread creation, thread termination, etc.  
>
>* The call-outs are directed to a replaceable shared library which
>allows activity to be communicated, or analyzed in whatever manner is
>desired.  This is basically an extension of the hooks libthread_db.so
>uses to support GDB, but it can invoke locally loaded code to handle the
>event.
>
>* A simple text log writer would be provided with NPTL to make it useful
>out-of-the-box and as an example of how to use the interfaces.  Another
>relatively simple tracing engine that could be built is a shared-memory
>ring buffer.  So by just defining an environment variable, you could get
>a text file describing all the threading activity of an application.
>
>Potential Concerns some people may have:
>
>* Won't having all these event points in the code cause reduced
>performance?  
>
>When we built similar capability into the Tru64 UNIX pthreads library it
>was done by having a bit test before each event, and it didn't result in
>any significant impact -- although in that case our compilers knew how
>to move the event code to "cold" sections of the library through
>profile-directed optimization.  When HP-UX added similar tracing to its
>thread library, it supplied a separate version of libpthread.so which
>exactly matches the default library except with tracing support.  In
>that case there is no runtime overhead unless LD_LIBRARY_PATH is used to
>redirect to the tracing library.
>
>* Is this really useful without writing a bunch of code?
>
>My suggestion is to provide a simple tracing engine with NPTL that
>writes out events to a file, so that could be edited/searched with
>common tools like sed/awk/perl etc.  Using this it is very easy to find
>pthread functions that are failing for unexpected reasons (how many of
>you always check the status of pthread_mutex_lock?).  To write a tracing
>engine it only takes 4 routines (basically init, fini, re-init on fork,
>and event write) so it is very easy to build special-purpose event
>handling to help debug a particular problem in your application
>(especially when building from an example).   For more details and an
>example see the updated proposal.
>
>
>So if there is interest in this type of facility, I'd be glad to help
>put it in place; either in the proposed form or perhaps a simplified
>version.  The existing proposal is very detailed and almost completely
>isolates the trace engine from the libpthread.so implementation details.
>Though it may seem complex, the NPTL version of these changes is the
>addition of about 81 lines of code plus one module to load the trace
>engine, and format the events. 
>
>-Jer
>
>=====================================================
>
>Jerry J. Harrow, Jr. 
>Hewlett-Packard Company
>110 Spit Brook Rd 
>Nashua, NH 03062-2698 
>USA
>Jerry.Harrow at hp dot com
>Telephone: 603.884.2193
>
>
>  
>