Trouble with LTT on ARM

Michael Cashwell <[email protected]>
Newsgroups gmane.linux.kernel.tracing
Message-ID <[email protected]>
Greetings all, newbie here. This is a long message and I appologize  
in advance for it, but I need assistance from someone who understands  
LTT.

I am making a tool chain for a set of ARM embedded systems and would  
like to include LTT in what I give the other developers. I'm having  
some trouble.

My kernel is a patched version of 2.6.21 (though I think the patches  
are not related). According to the LTT web site there is a LTT patch  
set for this kernel and the patches do apply cleanly. However, any  
make in the kernel (xconfig, uImage, modules, etc.) spits out these  
warnings:

ltt/Kconfig:246:warning: 'select' used by config symbol  
'LTT_PROBE_LOCKING' refer to undefined symbol 'STACKTRACE_SUPPORT'
ltt/Kconfig:247:warning: 'select' used by config symbol  
'LTT_PROBE_LOCKING' refer to undefined symbol 'LOCKDEP_SUPPORT'
ltt/Kconfig:271:warning: 'select' used by config symbol  
'LTT_PROCESS_STACK' refer to undefined symbol 'UNWIND_INFO'
ltt/Kconfig:272:warning: 'select' used by config symbol  
'LTT_PROCESS_STACK' refer to undefined symbol 'STACK_UNWIND'

 From what I can find, these stack-trace bits are not in 2.6.21 for  
ARM. They are in 2.6.22-rc4 but that's not final yet so it seems a  
little extreme.

All that said, I have the kernel, modules, control and daemon apps  
built for the target along with LTTV for the host. When I load the  
modules I see various complaints about some probes:

dm145:~ $ find /lib/modules/ -name '*.ko'
/lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-kernel.ko
/lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-fs.ko
/lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-core.ko
/lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe- 
kernel_arch_arm.ko
/lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-net.ko
/lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-list.ko
/lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-mm.ko
/lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/ltt-control.ko
/lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/ltt-statedump.ko
dm145:~ $
dm145:~ $ modprobe ltt-control
ltt-control init
dm145:~ $ modprobe ltt-probe-kernel_arch_arm
LTT : Facility kernel_arch registered with id 86
dm145:~ $ modprobe ltt-probe-kernel
LTT : Facility kernel registered with id 211
LTT unable to register probe kernel_sched_wait_task
LTT unable to register probe kernel_sched_migrate_task
dm145:~ $ modprobe ltt-probe-core
LTT : Facility core registered with id 0
LTT unable to register probe core_time_heartbeat
LTT unable to register probe core_time_heartbeat_full
dm145:~ $ modprobe ltt-probe-mm
LTT : Facility mm registered with id 59
LTT unable to register probe mm_swap_out
dm145:~ $ modprobe ltt-probe-net
LTT : Facility net registered with id 179
dm145:~ $ modprobe ltt-probe-list
LTT : Facility list registered with id 145
LTT unable to register probe list_file_descriptor
LTT unable to register probe list_vm_map
LTT unable to register probe list_interrupt
LTT unable to register probe list_process_state
LTT unable to register probe list_network_ip_interface
LTT unable to register probe list_statedump_end
dm145:~ $ modprobe ltt-probe-fs
LTT : Facility fs registered with id 225
dm145:~ $ modprobe ltt-statedump
LTT : ltt-facility-statedump init

The ones listed in the docs that I don't do above (ltt-core, ltt- 
relay, ltt-tracer) are built into the kernel as the README seems to  
indicate they should be. The heartbeat timer seems to only apply to  
MIPS (according to the xconfig help) so I don't understand why that's  
happening either.

Despite the missing probe points, I can run a trace:

dm145:~ $ lttctl -n trace -d -l /mnt/debugfs/ltt -t /var/trace
Linux Trace Toolkit Trace Controltt-control ltt_control_input
l 0.40-26042007

Controlling tltt_control : trace trace
race : trace

Creating trace trace
Linux Trace Toolkit Trace Daemon 0.40-26042007

Reading from debugfs directory : /mnt/debugfs/ltt/trace
Writing to trace directory : /var/trace

Creating supplementary trace files
Appending facility file kernel_arch_ppc.xml
Appending facility file kernel_arch_mips.xml
Appending facility file kernel_arch_x86_64.xml
Appending facility file locking.xml
Appending facility file fs.xml
Appending facility file xen.xml
Appending facility file compact.xml
Appending facility file kernel_arch_arm.xml
Appending facility file kernel.xml
Appending facility file list.xml
Appending facility file kernel_arch_c2.xml
Appending facility file user_generic.xml
Appending facility file kernel_arch_powerpc.xml
Appending facility file net.xml
Appending facility file mm.xml
Appending facility file core.xml
Appending facility file stack_arch_x86_64.xml
Appending facility file stack_arch_i386.xml
Appending facility file kernel_arch_i386.xml
ltt-control ltt_control_input
ltt_control : trace trace
Start tracing trace
Dumping facility core
Dumping facility mm
Dumping facility kernel_arch
Dumping facility list
Dumping facility net
Dumping facility kernel
Dumping facility fs
ltt_statedump_start
do_ltt_statedump
do_ltt_statedump end

And after a while I can stop it:

dm145:~ $ lttctl -n trace -q
Linux Trace Toolkit Trace Controltt-control ltt_control_input
l 0.40-26042007

Controlling tltt_control : trace trace
race : trace

Stop tracing trace
dm145:~ $

But when I transfer that /var/trace to my host and attempt to get  
anything from it it fails:

[cashwell@devmec bin]$ ./lttv -m textDump -t ~/netboot/CT/opt/trace
The input data file /home/cashwell/netboot/CT/opt/trace/cpu_0 does  
not contain a trace
The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
facilities_0 does not contain a trace
The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
modules_0 does not contain a trace
The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
network_0 does not contain a trace
The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
processes_0 does not contain a trace
The input data file /home/cashwell/netboot/CT/opt/trace/control/ 
interrupts_0 does not contain a trace

** ERROR **: Trace /home/cashwell/netboot/CT/opt/trace has no  
facility tracefile
aborting...
./lttv: line 15:  3256 Aborted                 $0.real $*

If I run it again but stop it this time via:

dm145:~ $ lttctl -n trace -R
Linux Trace Toolkit Trace Controltt-control ltt_control_input
l 0.40-26042007

Controlling trace : trace

ltt_control : trace trace
Stop tracing trace
ltt-control ltt_control_input
ltt_control : trace trace
Destroying trace trace
dm145:~ $ LTT : cpu : 2749 events lost in cpu channel (cpu 0).

So that's where I'm confused. If I tell it to dump (discard?) the  
running trace it tells me some non-zero number have been discarded.  
But if I end a trace normally it writes no events to the trace file.

There are so many moving parts here (especially with the worry that  
2.6.21 is not late enough for ARM) that I'm lost and don't know how  
to track down the problem.

Can someone see anything or provide or direct me to a how-to for  
using LTT on an ARM processor?

Thanks, (and sorry again about the length of this post)
-Mike
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.