[PATCH DISCUSSION 0/8] Reducing probe deferral - for profit and fun

Hello,

this patchset is meant as a continuation of the discussion done during
the last Boot Time SIG meeting [1] and not as something that can be
applied as-is. The maintainers for most (if not all) the involved
subsystems are not Cc'd for this reason - if anything useful will come
out of the discussion, a proper submission process will be followed.

I Cc'd the Beagleplay (well, K3 platform) maintainers because they might
be interested, but hey: feel free to ignore the thread.

== TL;DR ==

A probe deferral optimization was attemped on the Beagleplay, leading to
a reduction of 240ms of pre-init boot time and a set of techniques that
can be re-used on other platforms. Patches included.

== Introduction ==

The questions this work arose from were related to the probe deferral
process that happens during boot, and in particular:

  - how much time is "lost" doing probes that don't complete and are
    deferred?
  - is the current infrastructure good at deferring probes in an
    efficient way?
  - what can be done to enhance the current infrastructure?

They pretty much all generated doing boot time characterization on a
Beagleplay, where the probe deferral mechanism is triggered many times
during boot - half of the pre-init time of a vanilla system is spent
inside the deferred_probe_initcall() function, which is the point in
which probes that were deferred is performed.

== Status quo ==

On a Beagleplay booting the mainline kernel (v6.18-rc7) with the default
configuration (arm64 defconfig), the pre-init time is around 1080ms;
during this time span, 89 probe deferrals are triggered. This can be
easily verified using the initcall_debug command line option and
counting the occurrences of a driver returning 517 (or -517, this uneven
behavior bothers me, but I'll keep it for another time). Some example
follows:

[    0.060228] probe of 4201000.gpio returned -517 after 0 usecs
[    0.192159] probe of 2b300050.target-module returned -517 after 12 usecs
[    0.269648] probe of 2850000.serial returned -517 after 18 usecs
[    0.688228] probe of 8000f00.mdio returned 517 after 36137 usecs
[    0.959488] probe of fa20000.mmc returned 517 after 17380 usecs

It shall be noted that most of the deferral events lasts less than 10
microseconds; this is due to the excellent work the fw_devlink
infrastructure is doing under the hood to prevent the probes to even
happen if all the identified suppliers are not already present (that is,
probed) themselves. However, some other are non-negligible at all, like
the mdio friend above, which takes a whopping 36-milliseconds chunk of
time to decide that it's not the time for them to probe.

On this vanilla system, deferred_probe_initcall() takes 504765us to
complete - thats is roughly half of the entire boot process. This is not
unexpected, and not necessarily the sign that something is not working -
but let's investigate a bit.

== The investigation method ==

To investigate the reasons behind the probe deferral, two main tools
can used:

  - the initcall_debug command line option, that without any
    modification to the code offers a nice detailed view of the boot
    process (with initcall levels too, thanks to yours truly [2]);
  - the extensive debug prints that the core driver infrastructure is
    providing; this has to be enabled manually, either with
    CONFIG_DEBUG_DRIVER=y or with the plain old '#define DEBUG' inside
    drivers/base/core.c.

Combining the two, it can be seen what are the consumers and suppliers
for each peripheral and how they interact during boot; for example:

[    0.080959] platform fa00000.mmc: Linked as a consumer to 601000.gpio
[    0.089809] platform fa00000.mmc: Linked as a consumer to regulator-4
[    0.090502] platform fa00000.mmc: Linked as a consumer to regulator-5
[    0.194678] platform fa00000.mmc: Linked as a consumer to 44043000.system-controller:power-controller
[    0.198904] platform fa00000.mmc: Linked as a consumer to 44043000.system-controller:clock-controller
[    0.324348] platform fa00000.mmc: Linked as a consumer to f4000.pinctrl
[    0.720843] platform fa00000.mmc: error -EPROBE_DEFER: supplier regulator-5 not ready

This is a long - and maybe tedious, for someone - work, but in my
experience it helps to truly know a platform; if working more as an
integrator than a developer, this can be a game changer. At least, it
was for me - YMMV.

== Some possible ways out ==

Several methodologies can be exploited to "solve" the probe deferral
problem - assuming it is a problem. The ones that have been used to
obtain the results presented later - and thus covered by the patchset -
are here presented:

1) Ensure that the fw_devlink infrastructure is working correctly
   (spoiler: it is, most of the time) and close the gaps, if any; one
   of such gaps is covered in patch 1 (that was also already submitted
   [3] and will probably see a resend).

2) Help the aforementioned fw_devlink to break the cycles, because this
   might not be straightforward for its logic; this is what is done in
   patch 2 with the introduction of post-init-providers properties.
   It shall be noted that such property is not at the moment working
   with the DTB checker, because it is not considered in any schema.

3) Remove unnecessary dependencies between devices, like it is done in
   patch 3; this can happen in particular with regulators, which are
   often inserted to properly model the hardware but might not really
   be required.

4) Tune the kernel configuration - this can be done extensively with
   excellent results, but something might not be so obvious - see patch
   4 for a particularly sneaky condition that was encountered.

(Beware, from now on be dragons - you have been warned)

5) Change initcall levels. Old drivers for proven IPs - like I2C or SPI
   or GPIO controllers - might have been configured for a specific
   platform to initialize at a odd initcall level and then blindly
   re-used for new platforms. This is not always working well with
   modern systems with a System Management processor - like the
   Beagleplay - where more or less any device depends on a firmware
   interface like SCMI to be present. Some examples can be found in
   patches 5, 6 and 7.

6) Move things around inside the drivers Makefile. This is probably one
   of the most effective AND controversial thing to do, especially if
   coupled with 5), as it changes the init ordering of the various
   subsystems. Patch 8 shows what has been done for the Beagleplay, to
   ensure the SCMI firmware interface and all the services it provide
   (clocks, power domains and so on) are initialized before the drivers.
   Of course a single configuration that can be suitable for all
   platforms cannot be found, but maybe this aspect can be tackled from
   another direction, (mis-)using the LTO support that is already there.
   More work is required on this point, of course. 

== Results ==

With all the patches applied, I was able to reduce the overall pre-init
time to around 840ms, ~240ms less than the original system.

The duration of deferred_probe_initcall() lowered to ~18ms and the
number of pre-init probe deferrals to only 4 - these are not really
avoidable, as there are fixed regulators that depend on the PMIC.

Considering that no driver behavior has been modified... not bad, at
least IMO (but I'm the author and thus don't count).

== Bonus track: deferred probes of modules ==

With the discussion above focused on the pre-init time, nothing has been
said about what happens where the userspace of choice loads modules. The
answer is: a lot of deferrals can take place also there, but the
solution might however be simpler; in fact, the modprobe infrastructure
already supports soft dependencies between modules.

For the Beagleplay, probe deferrals hit the display (tidss) and sound
areas; this is the content I used inside /etc/modprobe.d/beagle.conf: 

softdep ite_it66121 pre: display_connector
softdep tidss pre: ite_it66121
softdep snd_soc_simple_card pre: ite_it66121 snd_soc_davinci_mcasp snd_soc_hdmi_codec

This solved all of the issues and ensured a smooth module experience.

== References == 

[1] https://lore.kernel.org/linux-embedded/MW5PR13MB5632E0103356DB6C0B330FCCFDD5A@MW5PR13MB5632.namprd13.prod.outlook.com/
[2] https://lore.kernel.org/linux-embedded/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

Cheers,
Francesco

Signed-off-by: Francesco Valla <[email protected]>
---
Francesco Valla (8):
      of: property: fw_devlink: Add support for "mmc-pwrseq"
      arm64: dts: ti: k3-am625-beagleplay: set post-init-providers
      arm64: dts: ti: k3-am625-beagleplay: remove inter-regulator dependency
      arm64: defconfig: compile reset-gpio as built-in
      gpio: davinci: reset initcall level
      i2c: omap: reset initcall level
      regulator: fixed: reset initcall level
      drivers: Makefile: reorder driver initialization

 arch/arm64/boot/dts/ti/k3-am625-beagleplay.dts | 12 ++++++++++-
 arch/arm64/configs/defconfig                   |  2 +-
 drivers/Makefile                               | 30 ++++++++++++++------------
 drivers/gpio/gpio-davinci.c                    | 17 +--------------
 drivers/i2c/busses/i2c-omap.c                  | 15 +------------
 drivers/of/property.c                          |  2 ++
 drivers/regulator/fixed.c                      | 13 +----------
 7 files changed, 33 insertions(+), 58 deletions(-)
---
base-commit: 30f09200cc4aefbd8385b01e41bde2e4565a6f0e
change-id: 20251126-beagleplay-probes-f14e5941de10

Best regards,
-- 
Francesco Valla <[email protected]>