Re: Trouble with LTT on ARM

Hi Michael,

* Michael Cashwell ([email protected]) wrote:
> Greetings all, newbie here. This is a long message and I appologize  
> in advance for it, but I need assistance from someone who understands  
> LTT.
> 
> I am making a tool chain for a set of ARM embedded systems and would  
> like to include LTT in what I give the other developers. I'm having  
> some trouble.
> 
> My kernel is a patched version of 2.6.21 (though I think the patches  
> are not related). According to the LTT web site there is a LTT patch  
> set for this kernel and the patches do apply cleanly. However, any  
> make in the kernel (xconfig, uImage, modules, etc.) spits out these  
> warnings:
> 
> ltt/Kconfig:246:warning: 'select' used by config symbol  
> 'LTT_PROBE_LOCKING' refer to undefined symbol 'STACKTRACE_SUPPORT'
> ltt/Kconfig:247:warning: 'select' used by config symbol  
> 'LTT_PROBE_LOCKING' refer to undefined symbol 'LOCKDEP_SUPPORT'
> ltt/Kconfig:271:warning: 'select' used by config symbol  
> 'LTT_PROCESS_STACK' refer to undefined symbol 'UNWIND_INFO'
> ltt/Kconfig:272:warning: 'select' used by config symbol  
> 'LTT_PROCESS_STACK' refer to undefined symbol 'STACK_UNWIND'
> 

These warnings are due to missing if X86_64 (and equivalent if X86) in
ltt/Kconfig. They only apply to these architectures. I tried to play
with it a bit to fix it, but it did not seem possible to use ifs on
selects. Therefore, the right solution might be to create Kconfig.ltt in
x86_64 and x86 subdirs. Anyway, this is mostly a matter of prettyness
and not urgent at all.

>  From what I can find, these stack-trace bits are not in 2.6.21 for  
> ARM. They are in 2.6.22-rc4 but that's not final yet so it seems a  
> little extreme.
> 
> All that said, I have the kernel, modules, control and daemon apps  
> built for the target along with LTTV for the host. When I load the  
> modules I see various complaints about some probes:
> 
> dm145:~ $ find /lib/modules/ -name '*.ko'
> /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-kernel.ko
> /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-fs.ko
> /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-core.ko
> /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe- 
> kernel_arch_arm.ko
> /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-net.ko
> /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-list.ko
> /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-mm.ko
> /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/ltt-control.ko
> /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/ltt-statedump.ko
> dm145:~ $
> dm145:~ $ modprobe ltt-control
> ltt-control init

Load all non probe modules before loading the probes in these LTTng
versions. Only in my lately posted marker infrastructure do I support
arbitrary order in module loading.

Some warnings can be cause by missing markers, others, like the
ltt-probe-list, are due to the fact that ltt-statedump was not loaded.

> dm145:~ $ modprobe ltt-probe-kernel_arch_arm
> LTT : Facility kernel_arch registered with id 86
> dm145:~ $ modprobe ltt-probe-kernel
> LTT : Facility kernel registered with id 211
> LTT unable to register probe kernel_sched_wait_task
> LTT unable to register probe kernel_sched_migrate_task
> dm145:~ $ modprobe ltt-probe-core
> LTT : Facility core registered with id 0
> LTT unable to register probe core_time_heartbeat
> LTT unable to register probe core_time_heartbeat_full
> dm145:~ $ modprobe ltt-probe-mm
> LTT : Facility mm registered with id 59
> LTT unable to register probe mm_swap_out
> dm145:~ $ modprobe ltt-probe-net
> LTT : Facility net registered with id 179
> dm145:~ $ modprobe ltt-probe-list
> LTT : Facility list registered with id 145
> LTT unable to register probe list_file_descriptor
> LTT unable to register probe list_vm_map
> LTT unable to register probe list_interrupt
> LTT unable to register probe list_process_state
> LTT unable to register probe list_network_ip_interface
> LTT unable to register probe list_statedump_end
> dm145:~ $ modprobe ltt-probe-fs
> LTT : Facility fs registered with id 225

The following one should have been loaded before the probes.

> dm145:~ $ modprobe ltt-statedump
> LTT : ltt-facility-statedump init
> 
> The ones listed in the docs that I don't do above (ltt-core, ltt- 
> relay, ltt-tracer) are built into the kernel as the README seems to  
> indicate they should be. The heartbeat timer seems to only apply to  
> MIPS (according to the xconfig help) so I don't understand why that's  
> happening either.
> 

Yeah, ARM is kind of different for timestamping.. the heartbeat timer
does not really apply to it. The correct way to fix this would be to
#ifdef the probe-core registration for core_time_heartbeat* event,
making them conditional on the heartbeat timer config option.

> Despite the missing probe points, I can run a trace:
> 
> dm145:~ $ lttctl -n trace -d -l /mnt/debugfs/ltt -t /var/trace
> Linux Trace Toolkit Trace Controltt-control ltt_control_input
> l 0.40-26042007
> 
> Controlling tltt_control : trace trace
> race : trace
> 
> Creating trace trace
> Linux Trace Toolkit Trace Daemon 0.40-26042007
> 
> Reading from debugfs directory : /mnt/debugfs/ltt/trace
> Writing to trace directory : /var/trace
> 
> Creating supplementary trace files
> Appending facility file kernel_arch_ppc.xml
> Appending facility file kernel_arch_mips.xml
> Appending facility file kernel_arch_x86_64.xml
> Appending facility file locking.xml
> Appending facility file fs.xml
> Appending facility file xen.xml
> Appending facility file compact.xml
> Appending facility file kernel_arch_arm.xml
> Appending facility file kernel.xml
> Appending facility file list.xml
> Appending facility file kernel_arch_c2.xml
> Appending facility file user_generic.xml
> Appending facility file kernel_arch_powerpc.xml
> Appending facility file net.xml
> Appending facility file mm.xml
> Appending facility file core.xml
> Appending facility file stack_arch_x86_64.xml
> Appending facility file stack_arch_i386.xml
> Appending facility file kernel_arch_i386.xml
> ltt-control ltt_control_input
> ltt_control : trace trace
> Start tracing trace
> Dumping facility core
> Dumping facility mm
> Dumping facility kernel_arch
> Dumping facility list
> Dumping facility net
> Dumping facility kernel
> Dumping facility fs
> ltt_statedump_start
> do_ltt_statedump
> do_ltt_statedump end
> 
> And after a while I can stop it:
> 
> dm145:~ $ lttctl -n trace -q
> Linux Trace Toolkit Trace Controltt-control ltt_control_input
> l 0.40-26042007
> 
> Controlling tltt_control : trace trace
> race : trace
> 
> Stop tracing trace
> dm145:~ $
> 

You should also lttctl -n trace -r, to destroy the active trace (meaning
what is taking data from the kernel to disk, not the actual trace
files). If you don't, the last buffers are not flushed to disk and are
therefore missing, what explains that the trace is unreadable.

> But when I transfer that /var/trace to my host and attempt to get  
> anything from it it fails:
> 
> [cashwell@devmec bin]$ ./lttv -m textDump -t ~/netboot/CT/opt/trace
> The input data file /home/cashwell/netboot/CT/opt/trace/cpu_0 does  
> not contain a trace
> The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
> facilities_0 does not contain a trace
> The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
> modules_0 does not contain a trace
> The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
> network_0 does not contain a trace
> The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
> processes_0 does not contain a trace
> The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
> interrupts_0 does not contain a trace
> 
> ** ERROR **: Trace /home/cashwell/netboot/CT/opt/trace has no  
> facility tracefile
> aborting...
> ./lttv: line 15:  3256 Aborted                 $0.real $*
> 
> If I run it again but stop it this time via:
> 
> dm145:~ $ lttctl -n trace -R
> Linux Trace Toolkit Trace Controltt-control ltt_control_input
> l 0.40-26042007
> 

This one should work, since -R is equivalent to -q and -r executed in
sequence.

> Controlling trace : trace
> 
> ltt_control : trace trace
> Stop tracing trace
> ltt-control ltt_control_input
> ltt_control : trace trace
> Destroying trace trace
> dm145:~ $ LTT : cpu : 2749 events lost in cpu channel (cpu 0).
> 
> So that's where I'm confused. If I tell it to dump (discard?) the  
> running trace it tells me some non-zero number have been discarded.  

-R means stop trace, flush buffers, release trace structures in the
kernel.

> But if I end a trace normally it writes no events to the trace file.
> 

no, -r does not flush.


The LTT : cpu : 2749 events lost in cpu channel (cpu 0). means two
things : either your buffers are too small for the amount of events you
want to collect, or you have a problem with your timestamp source, which
causes events to be discarded.

> There are so many moving parts here (especially with the worry that  
> 2.6.21 is not late enough for ARM) that I'm lost and don't know how  
> to track down the problem.
> 
> Can someone see anything or provide or direct me to a how-to for  
> using LTT on an ARM processor?
> 

Can you see the trace taken with -R with LTTV ?

> Thanks, (and sorry again about the length of this post)
> -Mike
> 
> _______________________________________________
> Ltt-dev mailing list
> [email protected]
> http://listserv.shafik.org/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68