Re: LTTng flight recorder

* Maxim Uvarov ([email protected]) wrote:
> Hello Mathieu,
> 
Hi,

> Could you please tell me what is the situation with flight recorder? Is 
> it possible to get  flight recorder records after crash?
> Is  somebody working on that?
> 

The flight recorder mode (which is basically a normal trace started with
the -m flight lttctl command line option) is fully functionnal. However,
no work has been done to develop an application that would extract the
trace buffers from a kernel core dump. This is something I know many
people would be interested in, but we just did not have the time to do
it yet.

SystemTAP have a flight recorder functionnality with their own kernel
binary buffer extraction tool. Since LTTng uses multiple channels, it
could make things slightly more complex (more pointers to follow), but
I think it could be adapted relatively easily to our traces.

Once you have the trace buffers extracted into trace files, things
becomes easy.

Note that the "facilities" channel will always be in normal (non
overwrite) tracing mode. It extracts information vital to understand the
trace at trace start.

The flight recorder mode makes the other channels run in overwrite mode.

The hybrid mode makes the cpu channels run in overwrite mode, and all
other channels (interrupts, processes, ...) run in normal tracing mode.

Note that I always provide the consumed offset (to be used with the
proper mask) for each subbuffers so we can always tell where the good
data starts in the buffer, and which buffer is partially read. It will
be very useful to reconstruct the tracefiles from a combination of the
flight recorder buffers (which may have dumped to disk or not, that's up
to the user) and the crash dump.

Also see linux/ltt-tracer.h: ltt_write_commit_counter().

It is currently commented out, and it should be tweaked to support crash
dump correctly. Its goal is to provide a counter of the number of "good"
bytes in the last subbuffer. In order to get that, we use the commit
count instead of the write "offset". The space at the end of the
write "offset" may not be filled when the crash occurs (due to the non
sequential nature of the reserve/commits. However, the commit_count %
subbuffer size on the last subbuffer will tell you how must data has
been written in this last subbuffer. Note that this data is not
necessarily contiguous, because reserve/commits are unordered.

Therefore, the goal of ltt_write_commit_counter() is to update the
lost_size in the subbuffer header each time the commit count reaches the
write "offset"; in those situations, we are guaranteed that all the
reserved space has been filled with data. It will cause a slightly
higher performance impact (mostly, it will occupy a cache line to keep
the subbuffer header hot), but should not be so bad. I planned to only
activate this in flight recorder traces.

So, I guess that if someone tweaks ltt_write_commit_counter() as I
proposed and tune SystemTAP's crash dump extractor to LTTng's format, we
could get something good. :)

Regards,

Mathieu

> In the old LTT link 
> http://www.opersys.com/LTT/dox/ltt-online-help/ltt-daemon-commandline.html 
> said  that  " flight recorder' traces, which are traces that run 
> continuously and are only written to disk when requested (see the -j 
> option below) or are examined after a system crash has taken place (see 
> SGI's LKCD (Linux Kernel Crash Dump) for tools that allow the retrieval 
> of the flight recorder buffers in a post-mortem situation)."
> 

Oh, right, you might also want to look at the old LTT to see how they
extract the buffers.

> Is it possible now to get LTTng buffers from core dump?
> 
> Best regards,
> Maxim Uvarov.

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68