Re: Flight recorder mode

* Jan Kiszka ([email protected]) wrote:
> Jan Kiszka wrote:
> > Mathieu Desnoyers wrote:
> >> * Jan Kiszka ([email protected]) wrote:
> >>> Mathieu Desnoyers wrote:
> >>>> * Jan Kiszka ([email protected]) wrote:
> >>>>> Jan Kiszka wrote:
> >>>>>> Mathieu Desnoyers wrote:
> >>>>>>> * Jan Kiszka ([email protected]) wrote:
> >>>>>>>> Mathieu Desnoyers wrote:
> >>>>>>>>> * Jan Kiszka ([email protected]) wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> I'm trying to understand the flight recorder mode (including the hybrid
> >>>>>>>>>> one) of LTTng. I originally expected that this mode means:
> >>>>>>>>>>
> >>>>>>>>>>  1. Fixed set of buffers, older data will be overwritten on wrap-around
> >>>>>>>>>>  2. Limited size of trace files
> >>>>>>>>>>
> >>>>>>>>>> While 1. seems to be true, 2. is neither what I observe in the output
> >>>>>>>>>> directory (flight-cpu_* get larger and larger) nor what I can find in
> >>>>>>>>>> lttd's code. Intentionally?
> >>>>>>>>>>
> >>>>>>>>> Yep, see the other lttctl options and the quickstart guide to see how to
> >>>>>>>>> spawn one lttd for the "normal" channels and a different one for the
> >>>>>>>>> flight recorder channels (should be done after you stop tracing).
> >>>>>>>> Sorry, /me is too blind: The quickstart guide is not mentioning flight
> >>>>>>>> recorder or hybrid mode, and also lttctl --help gives me no hint what
> >>>>>>>> pattern to apply for achieving limited trace file sizes.
> >>>>>>>>
> >>>>>>> Hrm, or do I recall doing documentaion I haven't done, let's see :)
> >>>>>>>
> >>>>>>> Ah, it's in a mailing list, not in the documentation yet :
> >>>>>>>
> >>>>>>> http://osdir.com/ml/linux.kernel.tracing/2006-09/msg00015.html
> >>>>>>>
> >>>>>>> It explains how to use the hybrid mode.  I'll cut'n'paste it at the end
> >>>>>>> of the quickstart guide.
> >>>>>>>
> >>>>>>> There is not trace file size limit you can give to lttctl nor lttd. You
> >>>>>>> can only specify the buffer size. The idea is that you can decide at
> >>>>>>> which moment you want to take you flight recorder "snapshot" by running
> >>>>>>> lttctl -f at the correct moment.
> >>>>>> OK, -EBRAINSUSPENDED, you just woke it up again:
> >>>>>>
> >>>>>> lttctl -b -n mytrace -m flight -l <debugfs>/ltt
> >>>>>> <run application>
> >>>>>> <when something happened:>
> >>>>>> lttctl -f -n mytrace -t <trace-out> -l <debugfs>/ltt
> >>>>> Hmm, in fact this won't fly yet. It looks like I need the hybrid mode,
> >>>>> because flight recorder traces generated like above are lacking some
> >>>>> basic information (stuff under /control I guess) to make lttv happy.
> >>>>>
> >>>>> That leaves me with a (for now) acceptable tracing procedure via hybrid
> >>>>> traces. But I'm still lacking some howto for pure flight recorder
> >>>>> traces. Any hints still appreciated... :)
> >>>>>
> >>>> Hrm, LTTV should be able to cope with the missing control information.
> >>>> The only tracefiles that needs to be complete are the facilities
> >>>> tracefiles.
> >>>>
> >>>> What do you get when you start lttv on a flight recorder trace ?
> >>> What I did:
> >>> 1. armall
> >>> 2. lttctl -n mytrace -l /sys/kernel/debug/ltt -b -m flight
> >> Ah, I see.. try to change -b for -d : it will spawn an lttd daemon which
> >> will read the "normal" channels (in this case, only the facilities
> >> tracefiles). (also add -l /sys/kernel/debug/ltt -t /tmp/trace)
> >>
> >> It should then start working better.
> > 
> > Perfect! That's now the behavior we already had with an ancient version
> > and that I wanted to recreate with latest LTTng.
> 
> Wait, that was too easy. That's _not_ what we have so far here, that is
> now again the unfortunate scenario I described in my first posting:
> continuous saving of the flight channel to disk while I only want the
> facility infos + the entries from the flight buffers on stop. :-/
> 

Ah, am I dumb... sorry..

to take a flight recorder trace, making sure everything is read
correctly :

armall
lttctl -n trace -c -m flight
lttd -n -d -t /tmp/trace -c /sys/kernel/debug/ltt
lttctl -n trace -s
.. do stuff
lttctl -n trace -q
lttd -f -d -t /tmp/trace -c /sys/kernel/debug/ltt
lttctl -m trace -r

This should work also for hybrid traces if you replace -m flight with -m
hybrid.

Maybe I should adapt lttctl so we can use it in the same way for hybrid
and flight recorder traces without calling lttd explicitly.

Mathieu


> Jan
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68