Re: Flight recorder mode

Mathieu Desnoyers wrote:
> * Jan Kiszka ([email protected]) wrote:
>> Mathieu Desnoyers wrote:
>>> Ah, am I dumb... sorry..
>>>
>>> to take a flight recorder trace, making sure everything is read
>>> correctly :
>>>
>>> armall
>>> lttctl -n trace -c -m flight
>>> lttd -n -d -t /tmp/trace -c /sys/kernel/debug/ltt
>>> lttctl -n trace -s
>>> .. do stuff
>>> lttctl -n trace -q
>>> lttd -f -d -t /tmp/trace -c /sys/kernel/debug/ltt
>>> lttctl -m trace -r
>>>
>>> This should work also for hybrid traces if you replace -m flight with -m
>>> hybrid.
>>>
>>> Maybe I should adapt lttctl so we can use it in the same way for hybrid
>>> and flight recorder traces without calling lttd explicitly.
>> What would be even nicer (considering our old, custom setup) is lttd in
>> standby until someone issues a trace stop, via lttctl or directly from
>> some application that detected a trace-worthy event. Then the daemon
>> should dump the flight channels and terminate. A script could catch
>> this, archive the trace, and even restart the whole thing if desired.
>>
>> OK, that may not fit smoothly into the current interface. But what I'm
>> currently looking for as a minimum requirement is a way to detect that
>> the trace was stopped by someone. Then I could live also with the
>> explicit lttd/lttctl invocations. Do I have to extend the netlink
>> interface with some wait-on-event service?
>>
>> Jan
>>
> 
> Sorry for late reply about this one, I was on a small trip in the last
> days.
> 
> Let's see what we can do here. First, one thing I would like to keep in
> mind is that, as a design philosophy, I try to keep the "trace control"
> kernel interface (liblttctl, ltt-control.ko, through netlink) as
> separate as possible from the "data extraction" kernel interface (lttd,
> ltt-relay.ko, through relay). The idea is that we want to allow data
> extraction from ltttd, an lttd network daemon, crash dump to work the
> the same "trace control" interface, without having to specialize it for
> any kind of these data extraction mechanisms.
> 
> Another thing we have to keep in mind is that tracing for a set of
> buffers can be started/stopped/restarted/stopped... multiple times. If a
> lttd daemon is reading the buffers, it will simply read them and wait
> for data when they have no more data to provide, and will stay alive
> until the channels are destroyed.
> 
> That being said, let's try to address your requirements, or first try to
> understand them correctly.
> 
> What I understand is that you would like to :
> do {
> 1- create trace channels, in hybrid mode
> 2- hook a lttd on the channels to gather the "normal" channels.
> 3- start tracing (it will fill the normal channels with state dump
>    information)
> some event occurs...
> 4- stop tracing
> 5- hook a lttd on the channels to gather the "flight recorder" channels.
> 6- destroy trace channels
> } while (continue_tracing);
> 
> What I understand is that you would like to have steps 5, 6 executed
> automatically after step 4.
> 
> I also see that the use case you propose stops the trace from a trigger
> within the kernel (compared to an lttctl invocation), am I right ?

Might become interesting as well one day, but right now it's all about
issuing a stop command (via netlink) from some application.

> 
> So we would have to find a way to execute steps 4, 5, 6 from the kernel.
> My initial thought would be to create a "trigger" kernel module : when
> the ltt_trigger("trace name") function would be called from within the
> kernel, it would set a flag in the trace structure telling a periodical
> timer to perform steps 4, 5, 6 (probably from a kernel thread). The
> reason for using a flag an not directly creating a kernel thread is that
> we would want the trigger to be reentrant from any execution context.
> Step 5 would be optional. We would have to record the lttd path
> somewhere in the ltt-trigger module. This module would have to be
> flexible enough to eventually execute a script, a network lttd or any
> other operation upon trigger.
> 
> Any thoughts ?

Let's focus on the user thingy first: We used to have a script that
looped like that:

while true; do
	start-our-tracer -in-foreground
	tar cjf trace-$date.tar.bz2 trace-dir
done


Now I could imagine something like this:

while true; do
	lttctl -n mytrace -d ...
	lttclt -n mytrace --wait-for-trace-being-stopped
	lttctl -n mytrace -f ...
	tar cjf trace-$date.tar.bz2 trace-dir
done

Means: I would like to block my script until someone else stopped
mytrace, ie. I need some event channel via netlink to track the trace state.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux