RE: Anyone interested in having additional debugging support in NPTL via a tracing facility?

Hello,

As I'm starting to write testing and stressing code for the NPTL, the
features you describe are very interresting to me, and it would be great
I think to merge this in the glibc directly, as it can be very usefull
to people wanting to use it, and those who don't can simply ignore it.

I have a few questions anyway:

-> what will happen if two threads running concurrently (on 2 processors
for example) will generate an event exactely on the same time? To avoid
any problem, is it possible, for instance, to have one logfile per
thread, let's say "app-nptl-%TID%.log"? or per processor?

-> as I am not a kernel guru, is it possible to have the same kind of
monitoring over the scheduler, to be able to manage the full activity of
the multithreaded application? (to construct a graph of execution, for
example).


Best regards,

Sebastien Decugis.





Le ven 30/01/2004 à 03:13, Jerry Harrow a écrit :
> Sorry for the re-send, but appearently the mail system truncated the 
> previous message after the first paragraph.  Hopefully the whole thing 
> comes through this time.
> 
> -Jer
> =============================================
> 
> I've had some on & off discussions with Ulrich about adding detailed
> event tracing support to NPTL for quite a while.  While favoring the
> idea in general, Ulrich isn't convinced that people (i.e. you all) have
> much of a desire for it.  When I last posted a proposal in this regard
> (https://listman.redhat.com/archives/phil-list/2002-December/msg00043.html)
>  there was only one reply -- obviously not an overwhelming
> endorsement.  I've done some updates on the original proposal and
> published it to
> ftp://ftp.compaq.com/pub/products/visualthreads/Linux/NPTLTracingV2.html
> 
> 
> Is there any significant interest in having support for this kind of 
> tracing/debug in NPTL?  
> 
> 
> Let me summarize the concepts:
> 
> * Add call-outs at most significant state changes in NPTL.  This would
> include attempt to lock a mutex, blocking while locking a mutex,
> acquiring a mutex, thread creation, thread termination, etc.  
> 
> * The call-outs are directed to a replaceable shared library which
> allows activity to be communicated, or analyzed in whatever manner is
> desired.  This is basically an extension of the hooks libthread_db.so
> uses to support GDB, but it can invoke locally loaded code to handle the
> event.
> 
> * A simple text log writer would be provided with NPTL to make it useful
> out-of-the-box and as an example of how to use the interfaces.  Another
> relatively simple tracing engine that could be built is a shared-memory
> ring buffer.  So by just defining an environment variable, you could get
> a text file describing all the threading activity of an application.
> 
> Potential Concerns some people may have:
> 
> * Won't having all these event points in the code cause reduced
> performance?  
> 
> When we built similar capability into the Tru64 UNIX pthreads library it
> was done by having a bit test before each event, and it didn't result in
> any significant impact -- although in that case our compilers knew how
> to move the event code to "cold" sections of the library through
> profile-directed optimization.  When HP-UX added similar tracing to its
> thread library, it supplied a separate version of libpthread.so which
> exactly matches the default library except with tracing support.  In
> that case there is no runtime overhead unless LD_LIBRARY_PATH is used to
> redirect to the tracing library.
> 
> * Is this really useful without writing a bunch of code?
> 
> My suggestion is to provide a simple tracing engine with NPTL that
> writes out events to a file, so that could be edited/searched with
> common tools like sed/awk/perl etc.  Using this it is very easy to find
> pthread functions that are failing for unexpected reasons (how many of
> you always check the status of pthread_mutex_lock?).  To write a tracing
> engine it only takes 4 routines (basically init, fini, re-init on fork,
> and event write) so it is very easy to build special-purpose event
> handling to help debug a particular problem in your application
> (especially when building from an example).   For more details and an
> example see the updated proposal.
> 
> 
> So if there is interest in this type of facility, I'd be glad to help
> put it in place; either in the proposed form or perhaps a simplified
> version.  The existing proposal is very detailed and almost completely
> isolates the trace engine from the libpthread.so implementation details.
> Though it may seem complex, the NPTL version of these changes is the
> addition of about 81 lines of code plus one module to load the trace
> engine, and format the events. 
> 
> -Jer
> 
> =====================================================
> 
> Jerry J. Harrow, Jr. 
> Hewlett-Packard Company
> 110 Spit Brook Rd 
> Nashua, NH 03062-2698 
> USA
> Jerry.Harrow at hp dot com
> Telephone: 603.884.2193
-- 
Sébastien DECUGIS
Bull S.A.