Re: [lttng-dev] loading modules with modprobe
Kienan Stewart via lttng-dev <[email protected]> Thu, 18 Jul 2024 10:58:20 -0400
| Newsgroups | org.lttng.lists.lttng-dev |
|---|---|
| Message-ID | <[email protected]> |
A couple of addendums inline below On 7/18/24 10:47 AM, Kienan Stewart via lttng-dev wrote: > Hi François, > > The dmesg excerpts are spaced by a number of minutes each, are they from > different attempts to build and load the modules? > > On 7/17/24 8:44 PM, François Belias wrote: >> Hello Kienan, >> >> Thanks to your advice, I managed to load some modules. However, I >> encountered errors like these when running `dmesg | grep lttng`: >> >> ``` >> [ 218.043672] module lttng_kretprobes: .gnu.linkonce.this_module >> section size must match the kernel's built struct module size at run time > > This sounds like the version of the module is built against the wrong > kernel headers or source. Also, are you using lttng-modules master or did you swap back to stable-2.13? If you're still on master and have the most recent changes lttng_kretprobes isn't required anymore. You will also need lttng-tools from master with the corresponding changes so that it doesn't try to load lttng_kretprobes. > >> [ 650.560883] lttng_kretprobes: module verification failed: signature >> and/or required key missing - tainting kernel > > Is your system running with secure boot enabled? If so, you either need > to disable secure boot or sign the built modules with a MOK key that is > correctly enrolled. > > See the following documentation: > > - https://lttng.org/docs/v2.13/#doc-linux-kernel-sig > - https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html > - https://github.com/dell/dkms > - > https://gitlab.com/linux-kernel/stable/-/blob/master/kernel/module/main.c#L2064 > > Whatever you did for this test is pretty closed to working. The previous > errors with the section size check happen earlier in the module load > process. This check seems to be one of the last, c.f. > https://elixir.bootlin.com/linux/v6.10/source/kernel/module/main.c#L2908 > >> [ 1468.329988] module lttng_ring_buffer_client_discard: >> .gnu.linkonce.this_module section size must match the kernel's built >> struct module size at run time >> ``` >> >> I'm unsure of their meaning. After loading my module, how can I be >> certain that my tracepoints are active? I read online that people use >> `sudo lttng list -k` for this purpose, but when I tried it, I got the >> error: "Error: Unable to list kernel events: Kernel tracer not >> available". Is there another way? > > If you're still getting "kernel tracer not available" then either > required kernel modules haven't been loaded or you started lttng-session > with `--no-kernel` option. `lttng-sessiond -vvv`, dmesg, and/or > journalctl should have information on which modules failed to load. > I know you're doing `sudo lttng list -k`, but the basic checks still apply: - is the lttng-sessiond running as the root user - (when running without sudo) does the tracing group exist, and is your user a member of that group > kienan > >> >> Thank you for your assistance. >> >> Le mer. 17 juill. 2024, à 12 h 08, Kienan Stewart >> <[email protected] <mailto:[email protected]>> a écrit : >> >> Hi François, >> >> On 7/17/24 11:47 AM, François Belias wrote: >> > Hello, Kienan >> > >> > >> > *Right. Which kernel are you actually running at this time? You >> can check >> > using: `uname -a`* >> > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP >> PREEMPT_DYNAMIC >> > Mon Jun 3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux >> > >> > *Since you've got a few different images and headers installed, >> it is >> > possible to check that same info the linux-source package: >> `dpkg -l | >> > grep linux-source`.* >> > ii linux-source 5.15.0.116.116 >> > all Linux kernel source >> with >> > Ubuntu patches >> > ii linux-source-5.15.0 5.15.0-116.126 >> > all Linux kernel source for >> version >> > 5.15.0 with Ubuntu patches >> > ii linux-source-6.5.0 6.5.0-44.44~22.04.1 >> > all Linux kernel source for >> version >> >> Your currently running `6.5.0-41-generic #41~22.04.2-Ubuntu` >> >> You have source installed for `6.5.0-44.44~22.04.1`. From your >> previous >> e-mail, there is also a linux image installed: >> >> ``` >> linux-image-6.5.0-44-generic 6.5.0-44.44~22.04.1 >> ``` >> >> Assuming you are using grub, could you verify that one of the options >> for booting in `/boot/grub/grub.cfg` is linux 6.5.0-44.44. Look >> for the >> lines similar to the following inside a menu entry block: >> >> (Note my example is 5.15.0-116, but you will be lookng for >> 6.5.0-44 or >> 6.5.0-44.44) >> >> ``` >> linux /boot/vmlinuz-5.15.0-116-generic root=.... >> initrd /boot/initrd.img-5.15.0-116-generic >> ``` >> >> If you have a grub entry for linux 6.5.0-44.44, could you reboot >> to run >> on the linux 6.5.0-44.44. >> >> After rebooting verify with `uname -a` that your kernel version is >> 6.5.0-44.44. >> >> Make sure you have unpacked the linux sources: >> >> ``` >> # Approximating the version numbers, since I don't have 6.5.0 >> installed >> cd /usr/src/linux-source-6.5.0 >> bunzip2 linux-source-6.5.0.tar.bz2 >> tar -xf linux-source-6.5.0.tar >> ``` >> >> Then retry the build with >> `KERNELDIR=/usr/src/linux-source-6.5.0/linux-source-6.5.0/` >> >> I haven't personally tried the full steps on an Ubuntu >> installation so >> you may still need to tweak a couple of things, but I think this >> should >> get you on the right path. >> >> Really the important thing is to make sure that the kernel source >> code >> matches the kernel you are running. >> >> > 6.5.0 with Ubuntu patches >> > *Could you run either `apt-cache policy` or `apt policy` and >> `apt-mark >> > showhold` in order to know which sources you have configured.* >> > For this part, /*apt-mark showhold*/ gave me nothing I don't know >> what >> > is means. >> >> `apt-mark showhold` will show any packages that are on hold - it is a >> way of marking packages that shouldn't be updated. >> >> As you have the linux-source for 6.5.0-44.44~22.04.1, I don't >> think it >> will be necessary to have the output of `apt-cache policy`. >> >> thanks, >> kienan >> >> > >> > I am looking at the links, but nothing seems to match my kernel >> version. >> > >> > Thanks for your assistance. >> > >> > Le mer. 17 juill. 2024, à 11 h 46, François Belias >> > <[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> a >> écrit : >> > >> > Hello, Kienan >> > >> > >> > *Right. Which kernel are you actually running at this time? >> You can >> > check >> > using: `uname -a`* >> > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP >> > PREEMPT_DYNAMIC Mon Jun 3 11:32:55 UTC 2 x86_64 x86_64 >> x86_64 GNU/Linux >> > >> > *Since you've got a few different images and headers >> installed, it is >> > possible to check that same info the linux-source package: >> `dpkg -l | >> > grep linux-source`.* >> > ii linux-source 5.15.0.116.116 >> > all Linux kernel >> source >> > with Ubuntu patches >> > ii linux-source-5.15.0 5.15.0-116.126 >> > all Linux kernel >> source for >> > version 5.15.0 with Ubuntu patches >> > ii linux-source-6.5.0 >> 6.5.0-44.44~22.04.1 >> > all Linux kernel source >> for >> > version 6.5.0 with Ubuntu patches >> > *Could you run either `apt-cache policy` or `apt policy` and >> `apt-mark >> > showhold` in order to know which sources you have configured.* >> > For this part, /*apt-mark showhold*/ gave me nothing I >> don't know >> > what is means. >> > >> > I am looking at the links, but nothing seems to match my >> kernel >> > version. >> > >> > Thanks for your assistance. >> > >> > Le mer. 17 juill. 2024, à 10 h 55, Kienan Stewart >> > <[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> a >> écrit : >> > >> > Hi François, >> > >> > On 7/16/24 4:34 PM, François Belias wrote: >> > > Hello Kienan, >> > > >> > > When I proceed with the following steps: >> > > >> > > ``` >> > > # Warning: this removes all untracked content from the >> > lttng-modules >> > > directory >> > > cd /path/to/lttng-modules-source >> > > git clean -dxf >> > > >> > > make >> > > sudo make modules_install >> > > sudo depmod -a >> > > ``` >> > > >> > > I encounter a warning in >> > > `/home/philippe/lttng-modules/src/probes/Kbuild:79` >> stating >> > "File >> > > ./arch/x86/kvm/lapic.h not found. Probe 'kvm' >> x86-specific is >> > disabled. >> > > Use full kernel source tree to enable it." Although >> > everything compiles >> > > fine and I'm able to load modules, when I execute >> `sudo lttng >> > list -k`, >> > > I don't see the tracepoints `kvm_entry` and >> `kvm_exit`, which >> > I believe >> > > should be available. Could this be due to not using my >> full >> > kernel tree >> > > during compilation? >> > >> > >> > Exactly. Those probes depend on having the full kernel >> source as >> > they >> > require private headers which aren't shipped as part of >> the >> > `linux-headers` package. >> > >> > > >> > > Here are the answers to your questions: >> > > >> > > - *Where did `/usr/src/linux-source-6.5.0` come from?* >> > > I initially attempted `sudo apt-get update && sudo >> apt-get >> > install >> > > linux-source`, but it gave me this answer >> */linux-source is >> > already the >> > > newest version (5.15.0.116.116)/*`, which doesn't >> match my >> > current >> > > kernel version. So, I downloaded it directly from >> kernel.org <http://kernel.org> >> > <http://kernel.org <http://kernel.org>> >> > > <http://kernel.org <http://kernel.org> >> <http://kernel.org <http://kernel.org>>> using this command: >> > `wget >> > > >> > https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz` >> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz> >> > >> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz >> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>> >> > > >> > >> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz` >> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz> >> > >> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz >> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>>. >> > > >> > >> > Right. Which kernel are you actually running at this >> time? You >> > can check >> > using: `uname -a` >> > >> > In any case, Linux upstream 6.5 is not the same as >> Ubuntus's 6.5. >> > >> > > - *Which Linux distribution and version are you using?* >> > > Running `lsb_release -a` gives me: >> > > ``` >> > > No LSB modules are available. >> > > Distributor ID: Ubuntu >> > > Description: Ubuntu 22.04.4 LTS >> > > Release: 22.04 >> > > Codename: jammy >> > > ``` >> > > >> > > - *Assuming your distribution has kernel packages, >> which do >> > you have >> > > installed, and do you have the corresponding kernel >> headers >> > package >> > > installed?* >> > > >> > > Running `dpkg -l | grep linux-image` shows these >> installed >> > packages: >> > > ``` >> > > ii linux-image-5.15.0-116-generic >> 5.15.0-116.126 >> > > amd64 Signed kernel image generic >> > > ii linux-image-6.5.0-28-generic >> > 6.5.0-28.29~22.04.1 >> > > amd64 Signed kernel image generic >> > > ii linux-image-6.5.0-41-generic >> > 6.5.0-41.41~22.04.2 >> > > amd64 Signed kernel image generic >> > > ii linux-image-6.5.0-44-generic >> > 6.5.0-44.44~22.04.1 >> > > amd64 Signed kernel image generic >> > > ii linux-image-generic 5.15.0.116.116 >> > > amd64 Generic Linux kernel image >> > > ii linux-image-generic-hwe-22.04 >> > 6.5.0.44.44~22.04.1 >> > > amd64 Generic Linux kernel image >> > > ``` >> > > >> > > And `dpkg -l | grep linux-headers` shows these >> installed >> > header packages: >> > > ``` >> > > ii linux-headers-5.15.0-116 >> > 5.15.0-116.126 all >> > > Header files related to Linux kernel version >> 5.15.0 >> > > ii linux-headers-5.15.0-116-generic >> 5.15.0-116.126 >> > > amd64 Linux kernel headers for version 5.15.0 on >> 64 bit >> > x86 SMP >> > > ii linux-headers-6.5.0-28-generic >> > 6.5.0-28.29~22.04.1 >> > > amd64 Linux kernel headers for version 6.5.0 on >> 64 bit >> > x86 SMP >> > > ii linux-headers-6.5.0-41-generic >> > 6.5.0-41.41~22.04.2 >> > > amd64 Linux kernel headers for version 6.5.0 on >> 64 bit >> > x86 SMP >> > > ii linux-headers-6.5.0-44-generic >> > 6.5.0-44.44~22.04.1 >> > > amd64 Linux kernel headers for version 6.5.0 on >> 64 bit >> > x86 SMP >> > > ii linux-headers-generic 5.15.0.116.116 >> > > amd64 Generic Linux kernel headers >> > > ii linux-headers-generic-hwe-22.04 >> > 6.5.0.44.44~22.04.1 >> > > amd64 Generic Linux kernel headers >> > > ``` >> > > >> > >> > Since you've got a few different images and headers >> installed, >> > it is >> > possible to check that same info the linux-source >> package: `dpkg >> > -l | >> > grep linux-source`. >> > >> > Maybe one thing tripping this process up is that you >> don't have >> > an apt >> > source that provides linux-source that matches the >> kernel you >> > are using. >> > Could you run either `apt-cache policy` or `apt policy` >> and >> > `apt-mark >> > showhold` in order to know which sources you have >> configured. >> > >> > There are also alternative sources for the full source >> tree: >> > >> > - You could download the specific package from >> > packages.ubuntu.com <http://packages.ubuntu.com> >> <http://packages.ubuntu.com <http://packages.ubuntu.com>>. See >> > https://packages.ubuntu.com <https://packages.ubuntu.com> >> <https://packages.ubuntu.com <https://packages.ubuntu.com>> and >> > use `apt install ./downloaded_package.deb` >> > >> > - You could clone the kernel source for Ubuntu and >> checkout the >> > appropriate tag. See >> > >> >> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags> <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags>> >> > >> > thanks, >> > kienan >> > >> > _______________________________________________ > lttng-dev mailing list > [email protected] > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev