Re: [RFC PATCH 00/14] driver core: defer per-VF sysfs creation for fast SR-IOV bring-up

On Fri, Jul 10, 2026 at 7:16 AM Greg Kroah-Hartman
<[email protected]> wrote:
>
> On Thu, Jul 02, 2026 at 07:40:19PM +0200, Pavol Sakac wrote:
> > Virtualization hardware keeps increasing in CPU-core and VF density.
> > Kernel is trending towards preserving VFs using LUO across a kexec
> > which will put SR-IOV in live-update hotpath. VFs must be re-added to
> > the sysfs tree, along with its supporting devices (e.g. vfio,
> > iommu_groups). On production hardware a single VF can consume ~80 kernfs
> > nodes. With thousands of VFs on modern hardware, hundreds of thousands
> > of kernfs nodes need to be created, which can add 100ms+ to the boot
> > time and to guest downtime.
>
> Hundreds of thousands?  You have a hundred sysfs files per device?
>
> > Proposed here is a PoC of deferred materialization of sysfs files until
> > first access to avoid cost of kernfs node creation in hot path. In the
> > reproducer below using a synthetic VF to isolate the sysfs overhead,
> > the per-device sysfs-creation time is reduced by ~74%. It exploits the
> > expected access pattern of hypervisors, where only a subset of device
> > files need to be accessed to attach a VF to a VM on the hot path.
>
> Ick, that access pattern could change with other users, this feels very
> fragile.
>
> > To allow lazy init scheme, a minimal set of files and directories is
> > built per device: a device directory resolves directly to the device's
> > kobject and is used to materialize attributes at access time. The
> > access is trapped in kernfs at the dir-read or path-walk stage and
> > depending on the trigger, either the full directory or a single file is
> > materialized.
> >
> > The changes are two logically distinct parts:
> >
> >   1. Refactor of hardcoded device_add / iommu_group sysfs attribute
> >      creation to a table-driven form to allow walkability of all
> >      attributes and single point of definition for attributes
> >      regardless of lazy opt-in to avoid duplicate definitions and
> >      associated maintainability issues with it.
>
> This is probably a good idea anyway.  Want to just send this as a series
> to start with, so we can take that?
>
> >   2. Lazy-init infra to front-run access to device files at lookup/
> >      readdir time to trigger materialization. Opt-in per device: PCI VFs
> >      and VFIO devices opt in via device_set_sysfs_lazy(); iommu_group
> >      opts in at the kobject level.
>
> This is going to get complex.  Where exactly is the bottleneck?  I'm
> guessing that systems with that many devices also have many hundreds of
> CPUs, right?  Are we hitting a single lock somewhere?  There have been
> changes in kernfs in the past to split up the locks to be more
> fine-grained, perhaps that needs to continue to solve this?

+1 I also am wondering if the time can be eliminated entirely instead
of deferred.

It would be nice to have a section of the cover letter that explains
where the time is being spent and why lazy initialization was chosen
over alternative approaches.

>
> > As a PoC, the code does not yet follow this logical structure in the
> > commit sequence and likely does not yet cover all corner cases.
> >
> > Synthetic reproducer measurements
> > ---------------------------------
> >   devices   per-dev time   total time       kernfs nodes (base -> lazy)
> >   -------   ------------   --------------   ----------------------------
> >      500    31.8->8.3 us    15.9-> 4.1 ms     36,417 ->  3,922  (72.8->7.8/dev)
> >     1000    33.2->8.7 us    33.2-> 8.7 ms     73,427 ->  8,432  (73.4->8.4/dev)
> >     2500    32.8->9.1 us    82.0->22.7 ms    184,421 -> 21,925  (73.8->8.8/dev)
> >     5000    33.0->8.7 us   165.1->43.5 ms    369,401 -> 44,406  (73.9->8.9/dev)
> >
> > The reproducer is synthetic: an in-kernel module registers N unbound
> > platform devices (60 attributes / 4 groups, ~74 kernfs nodes each; no
> > config space, BARs, or probe) in one timed burst that models SR-IOV
> > enablement. It runs as a two-kernel A/B from one vanilla 7.1-rc2 tree
> > (eager baseline vs patched, plus a patched/opt-in-OFF arm), 20 runs per
> > point. Measures only sysfs-creation slice.
>
> I sure hope sr-iov devices are NOT platform devices.  If so, please go
> fix that up first :)
>
> > Deferral removes ~74% of the sysfs-creation time and ~88% of the kernfs
> > nodes. At 5000 devices that is 121.6 ms removed (165.1 -> 43.5 ms) and
> > ~325,000 fewer kernfs nodes (369,401 -> 44,406). The table-driven rework
> > adds a +7..+11% eager-path time overhead to non-opted devices.
>
> You're just deferring this work to happen later, so when is that
> "later"?  And what about multi-threaded probing of the bus, doesn't that
> speed stuff up too?
>
> > Available as a docker image that orchestrates builds in qemu guest and
> > prints results as a table (x86_64 Linux host with /dev/kvm assumed):
> >
> >    docker run --device /dev/kvm ghcr.io/pavsa/linux-lazy-sysfs-vf-bench:7.1-rc2
> >
> > Assumptions / Trade-offs
> > ------------------------
> > The saving depends on the access pattern of hypervisors, and on no
> > component walking the whole device tree on the hot path to avoid full
> > materialization. Both are expected to hold on optimized hypervisors.
>
> Hah, as if.  Many userspace tools, once a device is notified, walk the
> whole tree of it to read the attributes.  You are going to be fighting
> that problem constantly...

This series could still benefit Live Update scenarios where the
initial userspace stack is small and tightly controlled to resume VMs
as quickly as possible after kexec-ing into the new kernel. It enables
userspace to decide when to incur the cost of creating these files (or
never create them).

But I agree that eliminating the time entirely would be better.

>
> > An actual production hypervisor was observed in local testing to
> > materialize an additional 5-10% of total nodes on average on the hot
> > path.
> >
> > Overhead: +7..+11% eager-path time overhead applies to every non-opted
> > device going through device_add() as all struct devices traverse the
> > walker and can't be gated under a config option.
> > The lazy opt-in infra can be gated under config option to have zero
> > impact if disabled.
>
> Again, where exactly is the time being spent?  What lock is "hot"?
>
> > Final words
> > -----------
> > This is an RFC to first get feedback on the decisions taken at driver
> > core + kernfs level before engaging maintainers of other subsystems
> > if we want to pursue this further or pivot to something else.
> >
> > The code is at a proof-of-concept quality written with AI assistance
> > with some known limitations with main target to gather potential
> > savings and initiate conversation.
> >
> > Feedback appreciated for:
> > - table-driven attribute refactor targeting common device add path /
> >   suggestions for more efficient/less invasive approach
>
> What's wrong with the normal way attributes are defined and allocated?
> They should all be tables today, and the driver core creating them when
> needed, no individual driver should ever be doing that on its own.
>
> thanks,
>
> greg k-h