Trouble with LTT on ARM
Michael Cashwell <[email protected]>
| Newsgroups | gmane.linux.kernel.tracing |
|---|---|
| Message-ID | <[email protected]> |
Greetings all, newbie here. This is a long message and I appologize in advance for it, but I need assistance from someone who understands LTT. I am making a tool chain for a set of ARM embedded systems and would like to include LTT in what I give the other developers. I'm having some trouble. My kernel is a patched version of 2.6.21 (though I think the patches are not related). According to the LTT web site there is a LTT patch set for this kernel and the patches do apply cleanly. However, any make in the kernel (xconfig, uImage, modules, etc.) spits out these warnings: ltt/Kconfig:246:warning: 'select' used by config symbol 'LTT_PROBE_LOCKING' refer to undefined symbol 'STACKTRACE_SUPPORT' ltt/Kconfig:247:warning: 'select' used by config symbol 'LTT_PROBE_LOCKING' refer to undefined symbol 'LOCKDEP_SUPPORT' ltt/Kconfig:271:warning: 'select' used by config symbol 'LTT_PROCESS_STACK' refer to undefined symbol 'UNWIND_INFO' ltt/Kconfig:272:warning: 'select' used by config symbol 'LTT_PROCESS_STACK' refer to undefined symbol 'STACK_UNWIND' From what I can find, these stack-trace bits are not in 2.6.21 for ARM. They are in 2.6.22-rc4 but that's not final yet so it seems a little extreme. All that said, I have the kernel, modules, control and daemon apps built for the target along with LTTV for the host. When I load the modules I see various complaints about some probes: dm145:~ $ find /lib/modules/ -name '*.ko' /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-kernel.ko /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-fs.ko /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-core.ko /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe- kernel_arch_arm.ko /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-net.ko /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-list.ko /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/probes/ltt-probe-mm.ko /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/ltt-control.ko /lib/modules/2.6.21-clksrc-sdrc-ct/kernel/ltt/ltt-statedump.ko dm145:~ $ dm145:~ $ modprobe ltt-control ltt-control init dm145:~ $ modprobe ltt-probe-kernel_arch_arm LTT : Facility kernel_arch registered with id 86 dm145:~ $ modprobe ltt-probe-kernel LTT : Facility kernel registered with id 211 LTT unable to register probe kernel_sched_wait_task LTT unable to register probe kernel_sched_migrate_task dm145:~ $ modprobe ltt-probe-core LTT : Facility core registered with id 0 LTT unable to register probe core_time_heartbeat LTT unable to register probe core_time_heartbeat_full dm145:~ $ modprobe ltt-probe-mm LTT : Facility mm registered with id 59 LTT unable to register probe mm_swap_out dm145:~ $ modprobe ltt-probe-net LTT : Facility net registered with id 179 dm145:~ $ modprobe ltt-probe-list LTT : Facility list registered with id 145 LTT unable to register probe list_file_descriptor LTT unable to register probe list_vm_map LTT unable to register probe list_interrupt LTT unable to register probe list_process_state LTT unable to register probe list_network_ip_interface LTT unable to register probe list_statedump_end dm145:~ $ modprobe ltt-probe-fs LTT : Facility fs registered with id 225 dm145:~ $ modprobe ltt-statedump LTT : ltt-facility-statedump init The ones listed in the docs that I don't do above (ltt-core, ltt- relay, ltt-tracer) are built into the kernel as the README seems to indicate they should be. The heartbeat timer seems to only apply to MIPS (according to the xconfig help) so I don't understand why that's happening either. Despite the missing probe points, I can run a trace: dm145:~ $ lttctl -n trace -d -l /mnt/debugfs/ltt -t /var/trace Linux Trace Toolkit Trace Controltt-control ltt_control_input l 0.40-26042007 Controlling tltt_control : trace trace race : trace Creating trace trace Linux Trace Toolkit Trace Daemon 0.40-26042007 Reading from debugfs directory : /mnt/debugfs/ltt/trace Writing to trace directory : /var/trace Creating supplementary trace files Appending facility file kernel_arch_ppc.xml Appending facility file kernel_arch_mips.xml Appending facility file kernel_arch_x86_64.xml Appending facility file locking.xml Appending facility file fs.xml Appending facility file xen.xml Appending facility file compact.xml Appending facility file kernel_arch_arm.xml Appending facility file kernel.xml Appending facility file list.xml Appending facility file kernel_arch_c2.xml Appending facility file user_generic.xml Appending facility file kernel_arch_powerpc.xml Appending facility file net.xml Appending facility file mm.xml Appending facility file core.xml Appending facility file stack_arch_x86_64.xml Appending facility file stack_arch_i386.xml Appending facility file kernel_arch_i386.xml ltt-control ltt_control_input ltt_control : trace trace Start tracing trace Dumping facility core Dumping facility mm Dumping facility kernel_arch Dumping facility list Dumping facility net Dumping facility kernel Dumping facility fs ltt_statedump_start do_ltt_statedump do_ltt_statedump end And after a while I can stop it: dm145:~ $ lttctl -n trace -q Linux Trace Toolkit Trace Controltt-control ltt_control_input l 0.40-26042007 Controlling tltt_control : trace trace race : trace Stop tracing trace dm145:~ $ But when I transfer that /var/trace to my host and attempt to get anything from it it fails: [cashwell@devmec bin]$ ./lttv -m textDump -t ~/netboot/CT/opt/trace The input data file /home/cashwell/netboot/CT/opt/trace/cpu_0 does not contain a trace The input data file /home/cashwell/netboot/CT/opt/trace/control/ facilities_0 does not contain a trace The input data file /home/cashwell/netboot/CT/opt/trace/control/ modules_0 does not contain a trace The input data file /home/cashwell/netboot/CT/opt/trace/control/ network_0 does not contain a trace The input data file /home/cashwell/netboot/CT/opt/trace/control/ processes_0 does not contain a trace The input data file /home/cashwell/netboot/CT/opt/trace/control/ interrupts_0 does not contain a trace ** ERROR **: Trace /home/cashwell/netboot/CT/opt/trace has no facility tracefile aborting... ./lttv: line 15: 3256 Aborted $0.real $* If I run it again but stop it this time via: dm145:~ $ lttctl -n trace -R Linux Trace Toolkit Trace Controltt-control ltt_control_input l 0.40-26042007 Controlling trace : trace ltt_control : trace trace Stop tracing trace ltt-control ltt_control_input ltt_control : trace trace Destroying trace trace dm145:~ $ LTT : cpu : 2749 events lost in cpu channel (cpu 0). So that's where I'm confused. If I tell it to dump (discard?) the running trace it tells me some non-zero number have been discarded. But if I end a trace normally it writes no events to the trace file. There are so many moving parts here (especially with the worry that 2.6.21 is not late enough for ARM) that I'm lost and don't know how to track down the problem. Can someone see anything or provide or direct me to a how-to for using LTT on an ARM processor? Thanks, (and sorry again about the length of this post) -Mike