Re: MIPS linux-2.6.12 (highly patched) + LTTng-0.5.69 + ltt-control-0.10-12062006

David Wuertele wrote:
> I am having no success using LTTng to instrument an embedded MIPS system with a
> vendor-supplied linux-2.6.12.  

...
>   Appending facility file kernel_arch_mips.xml
>   Appending facility file kernel_arch_i386.xml
> 
> At this point, the system appears to hang, but if I come back in a half hour, I
> have a prompt again.
>
> Here's what I don't understand:
> 
> 1.  Am I even close to the recommended process?

Sounds like a sane approach to me.  I have done quite a bit of
work with LTT (original) on embedded MIPS on 2.6.11, and with
LTTng on embedded MIPS on 2.6.16.  I'm not sure
what you are interested in, but going back to LTT original
might be another option. Off the cuff, I think that might be
an easier thing to get working.

But then again, since you've already invested in backporting
LTTng, and it looks like you're pretty close, it might be
good to just stick it out.

The interesting question is, did you get the output files
from the trace daemon?

Here's what I see in a short trace I did:
$ ls -lR
.:
total 1036
drwxr-xr-x  2 root root    4096 Jul 26  2007 control/
-rwxr-xr-x  1 root root 1048576 Jul 26  2007 cpu_0*
drwxr-xr-x  2 root root    4096 Jul 26  2007 eventdefs/

./control:
total 5140
-rwxr-xr-x  1 root root 1048576 Jul 26  2007 facilities_0*
-rwxr-xr-x  1 root root 1048576 Jul 26  2007 interrupts_0*
-rwxr-xr-x  1 root root 1048576 Jul 26  2007 modules_0*
-rwxr-xr-x  1 root root 1048576 Jul 26  2007 network_0*
-rwxr-xr-x  1 root root 1048576 Jul 26  2007 processes_0*

./eventdefs:
total 180
-rw-r--r--  1 root root  1588 Jul 26  2007 core.xml
-rw-r--r--  1 root root  1042 Jul 26  2007 fs_data.xml
-rw-r--r--  1 root root  3027 Jul 26  2007 fs.xml
-rw-r--r--  1 root root  1289 Jul 26  2007 ipc.xml
-rw-r--r--  1 root root 12378 Jul 26  2007 kernel_arch_arm.xml
-rw-r--r--  1 root root 24535 Jul 26  2007 kernel_arch_c2.xml
-rw-r--r--  1 root root 12040 Jul 26  2007 kernel_arch_i386.xml
-rw-r--r--  1 root root 35474 Jul 26  2007 kernel_arch_mips.xml
-rw-r--r--  1 root root 11705 Jul 26  2007 kernel_arch_powerpc.xml
-rw-r--r--  1 root root 11697 Jul 26  2007 kernel_arch_ppc.xml
-rw-r--r--  1 root root  2478 Jul 26  2007 kernel.xml
-rw-r--r--  1 root root  1948 Jul 26  2007 locking.xml
-rw-r--r--  1 root root  1537 Jul 26  2007 memory.xml
-rw-r--r--  1 root root   593 Jul 26  2007 network_ip_interface.xml
-rw-r--r--  1 root root   797 Jul 26  2007 network.xml
-rw-r--r--  1 root root  5718 Jul 26  2007 process.xml
-rw-r--r--  1 root root  2132 Jul 26  2007 socket.xml
-rw-r--r--  1 root root   657 Jul 26  2007 stack_arch_i386.xml
-rw-r--r--  1 root root  5267 Jul 26  2007 statedump.xml
-rw-r--r--  1 root root  1663 Jul 26  2007 timer.xml
-rw-r--r--  1 root root  1194 Jul 26  2007 user_generic.xml


> 2.  Is lttctl supposed to hang for a half hour like that?

No. If I were to guess what was happening, I would say it might
be something causing a network delay.  At the point where you
see the long delay, the system is trying to finalize the trace
data to the log, and I think there's a delay specifically
introduced to wait for stuff to flush to the daemon (and fs).

It might be worthwhile to do a tcpdump or wireshark trace
against your target, and see what network traffic is going on.
If you see a steady output of stuff, that points to some network
delay, but otherwise normal LTT operation.  If there's nothing
at all, then you've stopped somewhere else.  In this case,
printk is your friend.  I would add some to the relayfs path
and try to find out if the data transfer is happening as expected,
or if something is interfering.

> 
> 3.  I've seen recommendations to use genevent to generate headers for the
>     kernel ltt.  The instructions for genevent say use the XML in
>     LinuxTraceToolkitViewer/facilities.  But there is no such directory.  There
>     *is* such a directory in ltt-control-0.10-12062006/facilities.  Is that the
>     right one to use?

I believe so.

> 4.  Given that ltt-control-0.10 is supposed to be compatible with LTTng-0.5.69,
>     does that mean I don't need to run genevent?
If you modify the XML, I think you need to run genevent, but I'm not positive.
I never understood genevent very well.

> 5.  Do I need to customize these facility XML files?  Is there any
>     documentation on how to do that?

I recall having to do some minor customizations, but it was not critical.
MIPS has some different signal numbers than other platforms, and I think
that is what I had to change.

> 6.  Can I have LTTng write the trace files to a path on the NFS rootfs, or
>     should I use tmpfs mounts?
They can go over NFS - I did this all the time with LTTng on 2.6.16.
If you want to eliminate the network as a source of your delay, it wouldn't
hurt to try a tmpfs just to see what happens.

Good luck!
 -- Tim



=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================