Re: [lttng-dev] Trying to understand use of lttng enable-event --kernel --userspace-probe=

On Thu, May 18, 2023 at 9:17 AM Brian Hutchinson <[email protected]> wrote:
>
> On Wed, May 17, 2023 at 12:08 PM Mathieu Desnoyers
> <[email protected]> wrote:
> >
> > On 2023-05-16 22:11, Brian Hutchinson via lttng-dev wrote:
> > > Hi,
> > >
> > > I'm trying to figure out how to use uprobes with lttng.
> > >
> > > I can't use a normal uprobe for a line number just using the address I
> > > want to probe obtained from objdump?  As in:
> > >
> > > echo 'p /usr/local/bin/my_app:0x2c3a8' >>
> > > /sys/kernel/debug/tracing/uprobe_events
> > >
> > > ... which isn't a function entry, it's just a line of code I want to probe on.
> > >
> > > This link says it has to be elf or sdt:
> > > https://lttng.org/man/1/lttng-enable-event/v2.11/#doc-opt--userspace-probe
> > >
> > > So can I not probe on just a line of code by specifying an address???
> > >
> > > It doesn't look like these methods above will do what I'm wanting to
> > > do.  I've tried to find examples of using enable-event --kernel
> > > --userspace-probe= but there doesn't appear to be many.
> > >
> >
> > There are examples here:
> >
> > https://lttng.org/docs/v2.13/#doc-enabling-disabling-events
>
> I added a function that's called when the event I want to probe occurs.
>
> I tried to enable the event with:
> root@localhost:/usr/local/bin# lttng enable-event --kernel
> --userspace-probe=/usr/local/bin/my_app:my_probe_event
> Error: Missing event name(s).
>
> ... and:
>
> root@localhost:/usr/local/bin# lttng enable-event --kernel
> --userspace-probe=elf:/usr/local/bin/my_app:my_probe_event
> Error: Missing event name(s).
>
> I can do ${CROSS_COMPILE}objdump -d and verify that the symbol above
> for the function I added is indeed there.
>
> What am I doing wrong?
>

I updated my hello world to have a function I'd like to use the
--userspace-probe method on with the very original name of
'probe_function':

#include <stdio.h>
#include <lttng/tracef.h>

void probe_function(int i);

int main(int argc, char *argv[])
{
   unsigned int i;
   puts("Hello, World!\nPress Enter to continue...");
   /*
    * The following getchar() call only exists for the purpose of this
    * demonstration, to pause the application in order for you to have
    * time to list its tracepoints. You don't need it otherwise.
    */
   getchar();

   lttng_ust_tracef("Number %d, string %s", 23, "hi there!");
   printf("Number %d, string %s", 23, "hi there!");

   for (i = 0; i < argc; i++) {
       lttng_ust_tracef("Number %d, argv %s", i, argv[i]);
       printf("Number %d, argv %s", i, argv[i]);
   }

   puts("Quitting now!");

   probe_function(i);

   return 0;
}

void probe_function(int i) {

   lttng_ust_tracef("Number %d, string %s", i * i, "i^2");
   printf("Number %d, string %s", i * i, "i^2");

}

... and I get the same error as before when I try to enable the probe:
# lttng enable-event --kernel
--userspace-probe=/usr/local/bin/hello:probe_function
Error: Missing event name(s).

More system details below:


$ cat hello_objdump.txt

hello:     file format elf64-littleaarch64

SYMBOL TABLE:
0000000000000238 l    d  .interp        0000000000000000              .interp
0000000000000254 l    d  .note.gnu.build-id     0000000000000000
       .note.gnu.build-id
0000000000000278 l    d  .note.ABI-tag  0000000000000000
.note.ABI-tag
0000000000000298 l    d  .gnu.hash      0000000000000000              .gnu.hash
00000000000002b8 l    d  .dynsym        0000000000000000              .dynsym
00000000000004c8 l    d  .dynstr        0000000000000000              .dynstr
00000000000005ee l    d  .gnu.version   0000000000000000
.gnu.version
0000000000000620 l    d  .gnu.version_r 0000000000000000
.gnu.version_r
0000000000000660 l    d  .rela.dyn      0000000000000000              .rela.dyn
00000000000007b0 l    d  .rela.plt      0000000000000000              .rela.plt
0000000000000918 l    d  .init  0000000000000000              .init
0000000000000930 l    d  .plt   0000000000000000              .plt
0000000000000a40 l    d  .text  0000000000000000              .text
0000000000001194 l    d  .fini  0000000000000000              .fini
00000000000011a8 l    d  .rodata        0000000000000000              .rodata
00000000000013f0 l    d  .eh_frame_hdr  0000000000000000
.eh_frame_hdr
0000000000001468 l    d  .eh_frame      0000000000000000              .eh_frame
0000000000011d78 l    d  .init_array    0000000000000000
.init_array
0000000000011d88 l    d  .fini_array    0000000000000000
.fini_array
0000000000011d98 l    d  .dynamic       0000000000000000              .dynamic
0000000000011f98 l    d  .got   0000000000000000              .got
0000000000011fe8 l    d  .got.plt       0000000000000000              .got.plt
0000000000012078 l    d  .data  0000000000000000              .data
00000000000120d8 l    d  .bss   0000000000000000              .bss
0000000000000000 l    d  .comment       0000000000000000              .comment
0000000000000000 l    d  .debug_aranges 0000000000000000
.debug_aranges
0000000000000000 l    d  .debug_info    0000000000000000
.debug_info
0000000000000000 l    d  .debug_abbrev  0000000000000000
.debug_abbrev
0000000000000000 l    d  .debug_line    0000000000000000
.debug_line
0000000000000000 l    d  .debug_str     0000000000000000              .debug_str
0000000000000000 l    d  .debug_loc     0000000000000000              .debug_loc
0000000000000000 l    d  .debug_ranges  0000000000000000
.debug_ranges
0000000000000000 l    df *ABS*  0000000000000000              abi-note.S
0000000000000000 l    df *ABS*  0000000000000000              start.S
0000000000000000 l    df *ABS*  0000000000000000              init.c
0000000000000000 l    df *ABS*  0000000000000000              crti.S
0000000000000a78 l     F .text  0000000000000014              call_weak_fn
0000000000000000 l    df *ABS*  0000000000000000              crtn.S
0000000000000000 l    df *ABS*  0000000000000000              crtstuff.c
0000000000000a90 l     F .text  0000000000000000
deregister_tm_clones
0000000000000ac0 l     F .text  0000000000000000              register_tm_clones
0000000000000b00 l     F .text  0000000000000000
__do_global_dtors_aux
00000000000120d8 l     O .bss   0000000000000001              completed.9125
0000000000011d88 l     O .fini_array    0000000000000000
__do_global_dtors_aux_fini_array_entry
0000000000000b50 l     F .text  0000000000000000              frame_dummy
0000000000011d78 l     O .init_array    0000000000000000
__frame_dummy_init_array_entry
0000000000000000 l    df *ABS*  0000000000000000              hello.c
0000000000000b54 l     F .text  0000000000000120
lttng_ust_tracepoint__init_urcu_sym
0000000000000c74 l     F .text  0000000000000028
lttng_ust_tracepoint_logging_debug_enabled
0000000000000c9c l     F .text  000000000000007c
lttng_ust_tracepoints_print_disabled_message
00000000000013c0 l     O .rodata        000000000000002d
__func__.5512
0000000000000d18 l     F .text  00000000000000e8
lttng_ust__tracepoints__init
0000000000000e00 l     F .text  0000000000000170
lttng_ust__tracepoints__destroy
0000000000000000 l    df *ABS*  0000000000000000              elf-init.c
0000000000000000 l    df *ABS*  0000000000000000              crtstuff.c
0000000000001610 l     O .eh_frame      0000000000000000
__FRAME_END__
0000000000000000 l    df *ABS*  0000000000000000
0000000000011d88 l       .init_array    0000000000000000
__init_array_end
0000000000011d98 l     O *ABS*  0000000000000000              _DYNAMIC
0000000000011d78 l       .init_array    0000000000000000
__init_array_start
00000000000013f0 l       .eh_frame_hdr  0000000000000000
__GNU_EH_FRAME_HDR
0000000000011f98 l     O *ABS*  0000000000000000
_GLOBAL_OFFSET_TABLE_
0000000000000918 l     F .init  0000000000000000              _init
0000000000001190 g     F .text  0000000000000004              __libc_csu_fini
0000000000000000  w      *UND*  0000000000000000
_ITM_deregisterTMCloneTable
0000000000012078  w      .data  0000000000000000              data_start
00000000000120f0  w    O .bss   0000000000000008              .hidden
lttng_ust_tracepoint_destructors_syms_ptr
00000000000120d8 g       .bss   0000000000000000              __bss_start__
0000000000000000  w    F *UND*  0000000000000000
__cxa_finalize@@GLIBC_2.17
0000000000000000       O *UND*  0000000000000000              stderr@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000
dlclose@@GLIBC_2.17
00000000000120f8 g       .bss   0000000000000000              _bss_end__
00000000000120d8 g       .data  0000000000000000              _edata
0000000000000000       F *UND*  0000000000000000              dlopen@@GLIBC_2.17
0000000000001194 g     F .fini  0000000000000000              .hidden _fini
00000000000120f8 g       .bss   0000000000000000              __bss_end__
0000000000000000       F *UND*  0000000000000000              getpid@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000              dlsym@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000
__libc_start_main@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000              memset@@GLIBC_2.17
00000000000120e8  w    O .bss   0000000000000008              .hidden
lttng_ust_tracepoint_dlopen_ptr
0000000000012078 g       .data  0000000000000000              __data_start
0000000000000000  w      *UND*  0000000000000000              __gmon_start__
0000000000012080 g     O .data  0000000000000000              .hidden
__dso_handle
00000000000120c0  w    O .data  0000000000000018              .hidden
lttng_ust_tracepoint_destructors_syms
0000000000000000       F *UND*  0000000000000000              abort@@GLIBC_2.17
00000000000120e4  w    O .bss   0000000000000004              .hidden
lttng_ust_tracepoint_ptrs_registered
00000000000011a8 g     O .rodata        0000000000000004
_IO_stdin_used
0000000000012088  w    O .data  0000000000000038              .hidden
lttng_ust_tracepoint_dlopen
0000000000000000       F *UND*  0000000000000000              puts@@GLIBC_2.17
00000000000120e0  w    O .bss   0000000000000004              .hidden
lttng_ust_tracepoint_registered
0000000000001110 g     F .text  000000000000007c              __libc_csu_init
00000000000120f8 g       .bss   0000000000000000              _end
0000000000000a40 g     F .text  0000000000000000              _start
0000000000000000       F *UND*  0000000000000000
getchar@@GLIBC_2.17
00000000000120f8 g       .bss   0000000000000000              __end__
00000000000120d8 g       .bss   0000000000000000              __bss_start
0000000000000f70 g     F .text  000000000000012c              main
000000000000109c g     F .text  0000000000000074              probe_function
0000000000000000       O *UND*  0000000000000000
lttng_ust_tracepoint_lttng_ust_tracef___event
00000000000120d8 g     O .data  0000000000000000              .hidden
__TMC_END__
0000000000000000  w      *UND*  0000000000000000
_ITM_registerTMCloneTable
0000000000000000       F *UND*  0000000000000000              printf@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000              getenv@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000
fprintf@@GLIBC_2.17
0000000000000000       F *UND*  0000000000000000              lttng_ust__tracef

# zcat /proc/config.gz | grep -i probe
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_KPROBES=y
CONFIG_UPROBES=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not set
CONFIG_GENERIC_CPU_AUTOPROBE=y
# CONFIG_MTD_JEDECPROBE is not set
CONFIG_TIMER_PROBE=y
CONFIG_KPROBE_EVENTS=y
# CONFIG_KPROBE_EVENTS_ON_NOTRACE is not set
CONFIG_UPROBE_EVENTS=y
CONFIG_PROBE_EVENTS=y
# CONFIG_KPROBE_EVENT_GEN_TEST is not set
# CONFIG_KPROBES_SANITY_TEST is not set


> >
> > Indeed inserting a lttng-modules uprobe within functions is not
> > supported at the moment, mainly because we prefer to err towards safety
> > and don't have the validation in place to prevent corrupting the
> > program's instructions if an end user would try to insert a uprobe at an
> > address which is not an instruction boundary.
> >
> > So we only support inserting uprobe on functions and SDT probes at
> > the moment.
> >
> > Thanks,
> >
> > Mathieu
> >
> >
> > > Thanks,
> > >
> > > Brian
> > > _______________________________________________
> > > lttng-dev mailing list
> > > [email protected]
> > > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
> > --
> > Mathieu Desnoyers
> > EfficiOS Inc.
> > https://www.efficios.com
> >
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev