Re: instrumentation, was Re: core dump analysis

Hi Michael,

On 11.4.2023 11.24, Michael Schmitz wrote:
> Am 11.04.2023 um 19:19 schrieb Geert Uytterhoeven:
>> On Tue, Apr 11, 2023 at 6:56 AM Michael Schmitz <[email protected]> 
>>> Am 11.04.2023 um 12:20 schrieb Finn Thain:
>>>> A better solution might be be to port the existing instrumentation like
>>>> ftrace, kprobes, uprobes etc. Might be a lot of work though. I wonder
>>>> how portable that stuff is.

Another possibility is just using manually added kernel tracepoints:
https://www.kernel.org/doc/html/latest/trace/tracepoints.html

"git grep TRACE_EVENT" did not given any hits for arch/m68k/ though.

(I don't think those need any particular architecture support. One just 
adds them to locations where it makes sense to trace activity that is 
important, but which does not happen too often for the very small 
overhead from disabled static trace point to be significant.)


> Looking at a few arch implementations, I'm utterly confused. Wouldn't 
> know where to start.

Good article: https://lwn.net/Articles/132196/

(Ignore jprobes stuff as that's deprecated.)


Because probe replaces instruction at the probe point with a breakpoint:
https://www.kernel.org/doc/html/latest/trace/kprobes.html#how-does-a-kprobe-work

It needs architecture specific code to decode and simulate the replaced 
instruction (its side-effects) when it is later executed "out-of-line".

This also causes architecture specific limitations on where the probe 
can be set.  Architecture may not support simulating all possible 
instructions, just the most common ones (e.g. ones typically used on 
function entry and exit points).

Besides that, AFAIK plain kprobes support should not have much 
architecture specific stuff.


(I have no experience with kernel code, I just remember some LWN 
articles about kprobes during past 2 decades, and I was involved with a 
tool that did something similar in user-space, before Linux got uprobes 
support.)



As to what architecture to use as an example, I think it would be best 
to ask tracing maintainers for advice, as there seems to be a lot of 
code which has been copied from each other at different points in time, 
that might be nowadays better as shared rather than under each arch...

These architectures support kprobes:
------------------------------------
$ grep " ok " Documentation/features/debug/kprobes/arch-support.txt
     |         arc: |  ok  |
     |         arm: |  ok  |
     |       arm64: |  ok  |
     |        csky: |  ok  |
     |        ia64: |  ok  |
     |        mips: |  ok  |
     |      parisc: |  ok  |
     |     powerpc: |  ok  |
     |       riscv: |  ok  |
     |        s390: |  ok  |
     |          sh: |  ok  |
     |       sparc: |  ok  |
     |         x86: |  ok  |
------------------------------------

But only these seem to have probing functionality isolated nicely to its 
own directory:
------------------------------------
$ ls arch/*/kernel/probes/

arch/arm64/kernel/probes/:
decode-insn.c  decode-insn.h  kprobes.c  kprobes_trampoline.S  Makefile 
simulate-insn.c  simulate-insn.h  uprobes.c

arch/csky/kernel/probes/:
decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  kprobes_trampoline.S 
Makefile  simulate-insn.c  simulate-insn.h  uprobes.c

arch/riscv/kernel/probes/:
decode-insn.c  decode-insn.h  ftrace.c  kprobes.c  Makefile  rethook.c 
rethook.h  rethook_trampoline.S  simulate-insn.c  simulate-insn.h  uprobes.c
------------------------------------


As to testing whether kprobes work, that could be easier with ftrace 
support also present:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/ftrace/README


	- Eero

PS. Looking at the kernel features list, m68k arch seems to be missing a 
lot of other features too:
https://www.kernel.org/doc/html/latest/m68k/features.html