Re: Flight recorder mode

* Jan Kiszka ([email protected]) wrote:
> Mathieu Desnoyers wrote:
> > Ah, am I dumb... sorry..
> > 
> > to take a flight recorder trace, making sure everything is read
> > correctly :
> > 
> > armall
> > lttctl -n trace -c -m flight
> > lttd -n -d -t /tmp/trace -c /sys/kernel/debug/ltt
> > lttctl -n trace -s
> > .. do stuff
> > lttctl -n trace -q
> > lttd -f -d -t /tmp/trace -c /sys/kernel/debug/ltt
> > lttctl -m trace -r
> > 
> > This should work also for hybrid traces if you replace -m flight with -m
> > hybrid.
> > 
> > Maybe I should adapt lttctl so we can use it in the same way for hybrid
> > and flight recorder traces without calling lttd explicitly.
> 
> What would be even nicer (considering our old, custom setup) is lttd in
> standby until someone issues a trace stop, via lttctl or directly from
> some application that detected a trace-worthy event. Then the daemon
> should dump the flight channels and terminate. A script could catch
> this, archive the trace, and even restart the whole thing if desired.
> 
> OK, that may not fit smoothly into the current interface. But what I'm
> currently looking for as a minimum requirement is a way to detect that
> the trace was stopped by someone. Then I could live also with the
> explicit lttd/lttctl invocations. Do I have to extend the netlink
> interface with some wait-on-event service?
> 
> Jan
> 

Sorry for late reply about this one, I was on a small trip in the last
days.

Let's see what we can do here. First, one thing I would like to keep in
mind is that, as a design philosophy, I try to keep the "trace control"
kernel interface (liblttctl, ltt-control.ko, through netlink) as
separate as possible from the "data extraction" kernel interface (lttd,
ltt-relay.ko, through relay). The idea is that we want to allow data
extraction from ltttd, an lttd network daemon, crash dump to work the
the same "trace control" interface, without having to specialize it for
any kind of these data extraction mechanisms.

Another thing we have to keep in mind is that tracing for a set of
buffers can be started/stopped/restarted/stopped... multiple times. If a
lttd daemon is reading the buffers, it will simply read them and wait
for data when they have no more data to provide, and will stay alive
until the channels are destroyed.

That being said, let's try to address your requirements, or first try to
understand them correctly.

What I understand is that you would like to :
do {
1- create trace channels, in hybrid mode
2- hook a lttd on the channels to gather the "normal" channels.
3- start tracing (it will fill the normal channels with state dump
   information)
some event occurs...
4- stop tracing
5- hook a lttd on the channels to gather the "flight recorder" channels.
6- destroy trace channels
} while (continue_tracing);

What I understand is that you would like to have steps 5, 6 executed
automatically after step 4.

I also see that the use case you propose stops the trace from a trigger
within the kernel (compared to an lttctl invocation), am I right ?

So we would have to find a way to execute steps 4, 5, 6 from the kernel.
My initial thought would be to create a "trigger" kernel module : when
the ltt_trigger("trace name") function would be called from within the
kernel, it would set a flag in the trace structure telling a periodical
timer to perform steps 4, 5, 6 (probably from a kernel thread). The
reason for using a flag an not directly creating a kernel thread is that
we would want the trigger to be reentrant from any execution context.
Step 5 would be optional. We would have to record the lttd path
somewhere in the ltt-trigger module. This module would have to be
flexible enough to eventually execute a script, a network lttd or any
other operation upon trigger.

Any thoughts ?

Mathieu

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68