Re: LTTng usage issue

* Srinivasa Reddy ([email protected]) wrote:
> 
> Hi Mathieu,
> 
> I have patched the lttng-0.6.47-to-0.6.48-for-2.6.19.patch file 
> along with Linux kernel and patch-2.6.19-lttng-0.6.47 and build the
> kernel.
> 
> While building the kernel I got the following warning messages..
> 
> ************************************************************************
> ****
> arch/arm/kernel/traps.c: In function 'notify_die':
> arch/arm/kernel/traps.c:248: warning: format '%d' expects type 'int',
> but argument 2 has type 'long unsigned int'

LTTng instrumentation of this function should take %ld %ld as
parameters. However, the ltt/probes/ltt-probe-kernel_arm.c should be
updated and the XML description in the ltt-control/facilities should be
updated. It has been fixed in newer LTTng versions. Or you could try to
cast the unsigned long to an int. And it should not be the cause of your
problems.

> arch/arm/vfp/vfpmodule.c:298: warning: 'vfp_enable' defined but not used
> kernel/power/pm.c:205: warning: 'pm_register' is deprecated (declared at
> kernel/power/pm.c:64)
> kernel/power/pm.c:205: warning: 'pm_register' is deprecated (declared at
> kernel/power/pm.c:64)
> kernel/power/pm.c:206: warning: 'pm_send_all' is deprecated (declared at
> kernel/power/pm.c:180)
> kernel/power/pm.c:206: warning: 'pm_send_all' is deprecated (declared at
> kernel/power/pm.c:180)
> kernel/module.c: In function 'module_add_driver':
> kernel/module.c:2505: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> kernel/relay.c:487: warning: 'relay_hotcpu_callback' defined but not
> used

I guess in this version it was only used when CONFIG_HOTPLUG is used,
does not matter.

> fs/jffs2/readinode.c: In function 'jffs2_do_read_inode_internal':
> fs/jffs2/readinode.c:667: warning: 'latest_mctime' may be used
> uninitialized in this function
> fs/jffs2/readinode.c:664: warning: 'fd_list' may be used uninitialized
> in this function
> In file included from include/asm/atomic.h:233,
>                  from include/linux/dcache.h:6,
>                  from fs/nfs/namespace.c:10:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'

Depending on your architecture, __xchg might not be declared (see
include/asm-arm/system.h). Since we do not use it in LTTng, it does not
matter.

> fs/partitions/check.c: In function 'add_partition':
> fs/partitions/check.c:351: warning: ignoring return value of
> 'kobject_add', declared with attribute warn_unused_result
> fs/partitions/check.c:354: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> ipc/msg.c: In function 'sys_msgctl':
> ipc/msg.c:392: warning: 'setbuf.qbytes' may be used uninitialized in
> this function
> ipc/msg.c:392: warning: 'setbuf.uid' may be used uninitialized in this
> function
> ipc/msg.c:392: warning: 'setbuf.gid' may be used uninitialized in this
> function
> ipc/msg.c:392: warning: 'setbuf.mode' may be used uninitialized in this
> function
> ipc/sem.c: In function 'sys_semctl':
> ipc/sem.c:863: warning: 'setbuf.uid' may be used uninitialized in this
> function
> ipc/sem.c:863: warning: 'setbuf.gid' may be used uninitialized in this
> function
> ipc/sem.c:863: warning: 'setbuf.mode' may be used uninitialized in this
> function
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from include/ltt/ltt-facility-select-core.h:5,
>                  from ltt/ltt-facilities.c:20:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-core.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-fs.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-fs_data.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-ipc.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from
> ltt/facilities/ltt-facility-loader-kernel_arch_arm.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-kernel.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-memory.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-network.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-process.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-socket.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-timer.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from ltt/facilities/ltt-facility-loader-statedump.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/ltt-facilities.h:21,
>                  from
> ltt/facilities/ltt-facility-loader-network_ip_interface.c:12:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> drivers/amba/bus.c: In function 'amba_device_register':
> drivers/amba/bus.c:234: warning: ignoring return value of
> 'device_create_file', declared with attribute warn_unused_result
> drivers/amba/bus.c:236: warning: ignoring return value of
> 'device_create_file', declared with attribute warn_unused_result
> drivers/amba/bus.c:238: warning: ignoring return value of
> 'device_create_file', declared with attribute warn_unused_result
> drivers/amba/bus.c:239: warning: ignoring return value of
> 'device_create_file', declared with attribute warn_unused_result
> drivers/base/core.c: In function 'device_add':
> drivers/base/core.c:462: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> drivers/base/core.c:464: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> drivers/base/core.c:467: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> drivers/base/core.c:469: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> drivers/base/core.c: In function 'device_rename':
> drivers/base/core.c:831: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> drivers/base/core.c:839: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> drivers/base/class.c: In function 'class_device_rename':
> drivers/base/class.c:818: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> drivers/base/platform.c: In function 'platform_bus_init':
> drivers/base/platform.c:566: warning: ignoring return value of
> 'device_register', declared with attribute warn_unused_result
> drivers/input/mousedev.c: In function 'mousedev_connect':
> drivers/input/mousedev.c:664: warning: ignoring return value of
> 'sysfs_create_link', declared with attribute warn_unused_result
> drivers/input/keyboard/atkbd.c: In function 'atkbd_connect':
> drivers/input/keyboard/atkbd.c:991: warning: ignoring return value of
> 'sysfs_create_group', declared with attribute warn_unused_result
> drivers/input/mouse/psmouse-base.c: In function 'psmouse_connect':
> drivers/input/mouse/psmouse-base.c:1158: warning: ignoring return value
> of 'sysfs_create_group', declared with attribute warn_unused_result
> drivers/input/mouse/logips2pp.c: In function 'ps2pp_init':
> drivers/input/mouse/logips2pp.c:396: warning: ignoring return value of
> 'device_create_file', declared with attribute warn_unused_result
> drivers/input/mouse/trackpoint.c: In function 'trackpoint_detect':
> drivers/input/mouse/trackpoint.c:321: warning: ignoring return value of
> 'sysfs_create_group', declared with attribute warn_unused_result
> In file included from include/asm/atomic.h:233,
>                  from net/ipv4/raw.c:43:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/net/flow.h:11,
>                  from include/net/inet_sock.h:23,
>                  from include/net/icmp.h:23,
>                  from net/ipv4/proc.c:37:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/net/flow.h:11,
>                  from include/net/inet_sock.h:23,
>                  from include/net/ip.h:29,
>                  from net/ipv4/xfrm4_state.c:10:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> In file included from include/asm/atomic.h:233,
>                  from include/linux/kref.h:21,
>                  from lib/kref.c:14:
> include/asm-generic/atomic.h: In function 'atomic_long_xchg':
> include/asm-generic/atomic.h:281: warning: implicit declaration of
> function 'xchg'
> 
> ************************************************************************
> ****
> 
> 
> So finally the image file is ready and when I run the 
> 
> Lttctl -n trace -d -l /mnt/debugfs/ltt -t /home/sreddy/trace12 
> /home/sreddy # lttctl -n trace -d -l /mnt/debugfs/ltt -t
> /home/sreddy/trace12
> Linux Trace Toolkit Trace Control 0.32-05122006
> 
> Controlling trace : trace
> 
> ltt_control : trace trace
> Linux Trace Toolkit Trace Daemon 0.32-05122006
> 
> Reading from debugfs directory : /mnt/debugfs/ltt/trace
> Writing to trace directory : /home/sreddy/trace12
> 
> Creating supplementary trace files
> Appending facility file kernel_arch_ppc.xml
> Appending facility file kernel_arch_i386.xml
> Appending facility file fs.xml
> Appending facility file locking.xml
> Appending facility file core.xml
> Appending facility file kernel_arch_x86_64.xml
> Appending facility file stack.xml
> Appending facility file kernel_arch_powerpc.xml
> Appending facility file process.xml
> Appending facility file kernel.xml
> Appending facility file kernel_arch_arm.xml
> Appending facility file kernel_arch_c2.xml
> Appending facility file user_generic.xml
> Appending facility file fs_data.xml
> Appending facility file ipc.xml
> Appending facility file statedump.xml
> Appending facility file memory.xml
> Appending facility file timer.xml
> Appending facility file network.xml
> Appending facility file socket.xml
> Appending facility file network_ip_interface.xml
> Appending facility file kernel_arch_mips.xml
> ltt_control : trace trace
> 
> /home/sreddy # ps aux
> USER       PID CPU MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
> root         1  2.6  0.3  1484  500 ?        S    00:00   0:14 init [2]
> 
> root         2  0.0  0.0     0    0 ?        SWN  00:00   0:00
> [ksoftirqd/0]
> root         3  0.3  0.0     0    0 ?        SW<  00:00   0:01
> [events/0]
> root         4  0.2  0.0     0    0 ?        SW<  00:00   0:01 [khelper]
> root         5  0.0  0.0     0    0 ?        SW<  00:00   0:00 [kthread]
> root        45  0.0  0.0     0    0 ?        SW<  00:00   0:00
> [kblockd/0]
> root        46  0.0  0.0     0    0 ?        SW<  00:00   0:00 [kseriod]
> root        58  0.0  0.0     0    0 ?        SW   00:00   0:00 [pdflush]
> root        59  0.0  0.0     0    0 ?        SW   00:00   0:00 [pdflush]
> root        60  0.0  0.0     0    0 ?        SW<  00:00   0:00 [kswapd0]
> root        61  0.0  0.0     0    0 ?        SW<  00:00   0:00 [aio/0]
> root       176  0.0  0.0     0    0 ?        SW   00:00   0:00
> [mtdblockd]
> root       183  0.0  0.0     0    0 ?        SW<  00:00   0:00
> [kpsmoused]
> root       195  0.0  0.0     0    0 ?        SW<  00:01   0:00
> [rpciod/0]
> root       240  0.0  0.1  1504  200 ?        S    00:01   0:00
> /sbin/dhcpcd-bin 
> daemon     245  0.4  0.4  1840  560 ?        S    00:02   0:02
> /sbin/portmap
> root       265  0.6  0.4  1732  592 ?        S    00:02   0:02
> /usr/sbin/inetd
> root       266  2.3  0.5  1960  684 ttyAMA0  S    00:02   0:09 -ash
> root       267  0.1  0.3  1488  476 ttyAMA1  S    00:02   0:00
> /sbin/getty -L tt
> root       268  0.1  0.3  1488  476 tty2     S    00:02   0:00
> /sbin/getty -L tt
> root       269  0.1  0.3  1488  476 tty3     S    00:02   0:00
> /sbin/getty -L tt
> root       270  0.1  0.3  1552  456 ?        S    00:02   0:00 /bin/sh
> /etc/init
> root       275  0.1  0.3  1488  476 tty1     S    00:02   0:00
> /sbin/getty -L tt
> root       293  1.7  0.3  9776  424 ?        S    00:09   0:00
> /scratch/sreddy/L
> root       295 99.9  0.6  2688  884 ttyAMA0  R    00:09   0:05 ps aux
> /home/sreddy #
> 
> /home/sreddy # lttctl -n trace -R
> Linux Trace Toolkit Trace Control 0.32-05122006
> 
> Controlling trace : trace
> 
> ltt_control : trace trace
> ltt_control : trace trace
> LTT : unread channel facilities offset is 65536 and cons_off : 0 (cpu 0)
> LTT : facilities : commit count : 65536, subbuf size 65536
> LTT : unread channel interrupts offset is 65536 and cons_off : 0 (cpu 0)
> LTT : interrupts : commit count : 65536, subbuf size 65536
> LTT : unread channel processes offset is 262144 and cons_off : 0 (cpu 0)
> LTT : processes : commit count : 262144, subbuf size 262144
> LTT : unread channel modules offset is 65536 and cons_off : 0 (cpu 0)
> LTT : modules : commit count : 65536, subbuf size 65536
> LTT : cpu : 4633 events lost in cpu channel (cpu 0).
> LTT : unread channel cpu offset is 1048576 and cons_off : 0 (cpu 0)
> LTT : cpu : commit count : 1048576, subbuf size 1048576
> LTT : unread channel network offset is 65536 and cons_off : 0 (cpu 0)
> LTT : network : commit count : 65536, subbuf size 65536
> /home/sreddy #
> 
> /home/sreddy # lttd -c /mnt/debugfs/ltt/trace -t /home/sreddy/traceout1
> &

Starting lttd after issuing lttctl -n trace -R does not make sense,
since the tracing have been stopped and the channels have been removed
from debugfs.

> /home/sreddy # Linux Trace Toolkit Trace Daemon 0.32-05122006
> 
> Reading from debugfs directory : /mnt/debugfs/ltt/trace
> Writing to trace directory : /home/sreddy/traceout1
> 
> Creating trace subdirectory /home/sreddy/traceout1
> Adding inotify for channel /mnt/debugfs/ltt/trace/
> Added inotify for channel /mnt/debugfs/ltt/trace/, wd 4294967295
> No channel to read
> 
> /home/sreddy #
> 
> So is there any problem while building the kernel image?

It looks fine. Did you have any rejects when applying lttng patches on
you 2.6.18 kernel ? Is it a kernel.org vanilla kernel ?

> Who will be creating the channel files in /mnt/debugfs/ltt/trace, which
> are readable by lttd. Lttng or lttctl?
> 

lttng asks relay to create them. lttng is controlled by lttctl.

> And why it's not happening in my case?
> 

Good question. It seems like relay refuses to create the channels.

Could you recompile ltt-control with:

make -DDEBUG and give me the output when running it?

And also I think trying to use a different kernel version could help
finding the problem more quickly, since it seems to be related to
kernel/relay.c, and its version depends on the kernel you are compiling.
Trying 2.6.20 kernels with LTTng 0.9.0 would tell us much more. And then
you could even try a backport from the 0.9.0 patches to you 2.6.19
kernel. You would have to update the userspace tools too.

Mathieu

> Could you please have a look at the warning messages and let me know
> where is the problem?
> Regards,
> -Srinivas.
> 
> -----Original Message-----
> From: Mathieu Desnoyers [mailto:[email protected]] 
> Sent: Monday, July 23, 2007 7:23 PM
> To: Srinivasa Reddy
> Cc: [email protected]
> Subject: Re: [Ltt-dev] LTTng usage issue
> 
> * Srinivasa Reddy ([email protected]) wrote:
> > 
> > Very good morning..
> > 
> > Yes, lttctl and lttd are installed in the make installed path only.
> > See the output of each command given below.
> > 
> > *********************************************************
> > home/sreddy # lttctl -n trace -b
> > 
> > Linux Trace Toolkit Trace Control 0.32-05122006
> > 
> > Controlling trace : trace
> > 
> > ltt_control : trace trace
> > ltt_control : trace trace
> > /home/sreddy #
> > 
> > ***********************************************************
> > /home/sreddy # ls -alR /mnt/debugfs/ltt
> > /mnt/debugfs/ltt:
> > total 12
> > drwxr-xr-x  3 root root 4096 Jul 23  2007 .
> > drwxr-xr-x  3 root root 4096 Jul 19  2007 ..
> > drwxr-xr-x  2 root root 4096 Jul 23  2007 trace
> > 
> > /mnt/debugfs/ltt/trace:
> > total 8
> > drwxr-xr-x  2 root root 4096 Jul 23  2007 .
> > drwxr-xr-x  3 root root 4096 Jul 23  2007 ..
> > /home/sreddy #
> > 
> 
> Here is where the problem shows up:
> 
> LTTng should show files (trace channels) in debugfs. Please try to get
> the LTTng 0.6.48 bits as a first step. I just prepared a patch for the
> upgrade:
> 
> http://ltt.polymtl.ca/lttng/lttng-0.6.47-to-0.6.48-for-2.6.19.patch
> 
> If this doesn't work, I guess you'll have to try a backport of the
> 2.6.20 LTTng to your 2.6.19 kernel.
> 
> Mathieu
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68