RE: How to correctly reserve prefetchable bridge windows for large, resizable BARs behind PCIe switches (8x GPU, PEX890xx) - seeking guidance on upstreamable approach.

Public

> -----Original Message-----
> From: Geramy Loveless <[email protected]>
> Sent: Wednesday, August 26, 2026 4:22 PM
> To: [email protected]
> Cc: [email protected]; Deucher, Alexander
> <[email protected]>; [email protected]; Koenig,
> Christian <[email protected]>; [email protected];
> Limonciello, Mario <[email protected]>; [email protected];
> [email protected]
> Subject: How to correctly reserve prefetchable bridge windows for large,
> resizable BARs behind PCIe switches (8x GPU, PEX890xx) - seeking guidance on
> upstreamable approach.
>
> Hi all,
>
> We have an 8-GPU host where each GPU wants a 32 GiB resizable BAR, but
> the GPUs sit behind a multi-level Broadcom PEX890xx PCIe switch fabric
> whose prefetchable bridge windows are sized by firmware for the default
> 256 MiB BAR. The kernel/driver never grows those windows, so every BAR
> stays a 256 MiB and the cards run in small-BAR mode.
>
> We have an out-of-tree patch that makes it work (full patch inline below as
> PATCH 1), but we're not confident it's the right shape for upstream, and we've
> hit a nasty interaction with the GPU firmware (PSP) that suggests we're doing
> it at the wrong layer. We'd really appreciate guidance on the correct approach
> before we try to submit anything.
>
> Background / history
> --------------------
> We first developed this reservation logic for Thunderbolt 5 / USB4-attached
> GPUs, where the tunneled PCIe hierarchy has the identical
> problem: firmware sizes the bridge windows for the boot-time BAR, and
> there's no room for the driver to grow a resizable BAR afterward. We found
> the exact same problem applies to GPUs behind on-board PCIe switches, so
> we adapted the same idea here.
>
> Software versions (both inline patches are "git apply --check" clean on these
> exact bases)
> ----------------------------------------------------------------------
> - Kernel base:   torvalds 45c13f3f9 (Makefile VERSION 7.2.0;
>                   "Merge tag 'hwlock-v7.3' ...")
> - amdgpu:        amd-staging-drm-next @ 75a5e1b6b
>                   ("drm/amdkfd: guard against NULL restore_mqd in CRIU queue
>                    restore")
> - PATCH 1 (inline below): our out-of-tree PCI window reservation.
> - PATCH 2 (inline below): Mario Limonciello, commit 63b2896378d,
>                   "drm/amdgpu: restrict BAR0 fallback read to SR-IOV VFs only",
>                   Fixes: ea8ac194077d.
>
> Note on the two amdgpu issues our PCI patch exposes: pre-sizing the fb BAR
> resource causes the *hardware* BAR to be large by the time amdgpu binds,
> which triggers two separate problems on the R9700 - (1) the early BAR0-
> aperture read added in ea8ac194077d returns ~0 for the IP-discovery table
> (PATCH 2 restricts that read to SR-IOV VFs and is what gets us pas discovery
> today), and (2) the PSP teardown described below. Both disappear if the child
> BAR is left at its boot size and the driver performs the resize - which is the core
> of our question.
>
> Hardware
> --------
> - Chassis/board:  ASUSTeK K14PG-D24 Series, BIOS 2001 (2024-08-02)
> - CPU:            2x AMD EPYC 9354 (Genoa, 32C/socket, 2 NUMA nodes)
> - Root complex:   AMD Genoa/Bergamo Root Complex [1022:14a4]
> - PCIe switches:  Broadcom/LSI PEX890xx PCIe Gen5 Switch [1000:c030] (rev
> b0),
>                    cascaded (upstream port -> multiple downstream ports ->
>                    further PEX890xx stages -> one GPU per leaf)
> - GPUs:           8x AMD Navi 48 [Radeon AI PRO R9700] [1002:7551] rev c0,
>                    subsystem ASRock [.. :5413], 32 GiB VRAM each
> (gfx1201/RDNA4)
> - GPU BDFs:       07:00.0 0a:00.0 68:00.0 6d:00.0 87:00.0 8a:00.0
> e8:00.0 ed:00.0
> - Boot cmdline:   iommu.passthrough=0 pci=realloc (+ disable_acs_redir
> on the
>                    eight switch downstream ports)
>
> Topology (abridged; one leg shown, all eight are symmetric):
>
>    [EPYC RC] -> PEX890xx up -> PEX890xx dn(62:00.0) -> PEX890xx(63:00.0)
> -> GPU(68:00.0)
>
> Every prefetchable window in that chain is firmware-sized at 258 MiB
> (256 MiB fb BAR + 2 MiB doorbell BAR), e.g.:
>
>    62:00.0  Prefetchable memory behind bridge: ...  [size=258M]
>    63:00.0  Prefetchable memory behind bridge: ...  [size=258M]
>
> To resize a single GPU BAR to 32 GiB, every bridge window from the leaf up to
> the root must be enlarged, and windows feeding two GPUs need >= 64 GiB in
> my specific use case but this needs to be generally acceptable for all use cases
> of course.
>
> What we observe without any patch
> ---------------------------------
> amdgpu_device_resize_fb_bar() runs and calls pci_resize_resource(pdev, 0,
> <32G>), which returns -ENOSPC ("Not enough PCI address space for a large
> BAR") because the parent switch windows have no room, and nothing grows
> them. All eight BARs stay at 256 MiB.
>
> (This is with current mainline/amd-staging; the recent removal of the driver-
> side re-assignment in db92e3fef53e2 does not change this outcome on our
> topology - we tested a revert and still got 256 MiB, because re-assigning
> unassigned resources does not grow already-assigned switch
> windows.)
>
> Our current out-of-tree patch (PATCH 1, inline below)
> -----------------------------------------------------
> At pci_assign_unassigned_root_bus_resources() time we (a) bump each
> display device's fb BAR resource to its max ReBAR size, (b) release the
> prefetchable BARs/windows so the assignment pass re-sizes the whole
> prefetchable hierarchy large enough to hold the enlarged BARs, plus a small
> setup-res.c tweak so multiple max-size BARs can share one window with
> correct alignment. With this, all eight windows get sized for 32 GiB and the
> resize succeeds.
>
> The problem with our approach
> -------------------------------------------------------
> Bumping the fb BAR *resource* to max causes the assignment pass to
> program the hardware ReBAR to 32 GiB at enumeration time (before amdgpu
> binds). On these R9700s that early hardware BAR resize tears down the
> firmware-loaded PSP "sign of life" (sOS), and amdgpu's later PSP bring-up
> then fails to reload it:
>
>      amdgpu ...: PSP load kdb failed!
>      amdgpu ...: psp reg (0x16080) wait timed out ... read: 30000 exp:
> 80000000
>      amdgpu ...: hw_init of IP block <psp> failed -22
>
> If instead the hardware BAR is left at its BIOS size and amdgpu performs the
> resize itself (which is what happens when firmware pre-enabled a large BAR),
> the PSP stays alive and the GPU comes up fine. So what we actually want is to
> reserve/size the prefetchable *windows* for 32 GiB while leaving the child BAR
> at its boot size, and let the driver do the hardware resize into the pre-sized
> window. Our current patch conflates the two.
>
> Questions
> ---------
> 1. Is there an existing/preferred mechanism for reserving prefetchable bridge-
> window space for large resizable BARs across a switch hierarchy that we
> should be using (something analogous to the hotplug hpmemprefsize
> reservation, but for fixed switch fabrics)? We'd rather use it than carry this.
>
> 2. If a new mechanism is needed, where should it live? Our instinct is that the
> window sizing wants to happen in pci_bus_size_bridges()/the realloc
> (add_size) path so the child BAR resource is never grown - i.e.
> reserve the window, not the BAR. Is that the right direction, and is there a
> sanctioned way to request "N bytes of extra prefetchable window on this
> bridge" during the sizing pass?
>
> 3. Given the PSP interaction, is the expectation that the *driver* always owns
> the hardware BAR resize (so core should only ever arrange windows, never
> program the device ReBAR)? If so, is the resource-size bump we do simply the
> wrong tool?

I'm not sure why this would affect the PSP bootloader.  It seems like having the bios resize the BAR would have the same effect (e.g., systems with ReBAR enabled in the bios ), but that works.  For background, the PSP has firmware on the board that runs as soon as the device powers up to initialize the card for use by software.  Sound like maybe whatever you are doing is putting the card into reset or some other bad state.  If the board is reset, driver will need to wait for the PSP bootloader to complete before the GPU can be initialized.  The PSP bootloader also populates the IP discovery table.  Without that, the driver can't initialize the device.  Additionally, the driver resizes the BAR really early in its init sequence, before it's even touched the PSP so I don't see why it would matter if the driver did the resize or not.

Alex


>
> 4. We're happy to write this properly and carry the testing - we have the 8x
> R9700 / PEX890xx box and can iterate quickly. We can also share the full lspci -
> tvvv, dmesg, and the Thunderbolt/USB4 variant of the patch if useful.
>
> Thanks a lot for any pointers,
> Geramy Loveless
>
> The two patches follow inline below.
>
> ===================================================================
> ===========
> PATCH 1/2 - PCI: reserve prefetchable bridge windows for resizable BARs
>              (our out-of-tree patch; git apply --check clean on 45c13f3f9)
> ===================================================================
> ===========
> From: Geramy Loveless <[email protected]>
> Date: Wed, 26 Aug 2026 00:00:00 +0000
> Subject: [PATCH] PCI: reserve prefetchable bridge windows for multi-child
> resizable BARs
>
> Out-of-tree patch we carry to make 8x resizable-BAR GPUs behind a cascaded
> PCIe switch fabric usable. Firmware sizes every prefetchable bridge window for
> the boot-time (256 MiB) BAR, so a driver's later resize to 32 GiB fails with -
> ENOSPC because no window in the chain has room.
>
> Before sizing the bridge windows, bump each display device's fb BAR resource
> to its max ReBAR size and release the prefetchable BARs/windows so the
> assignment pass re-sizes the prefetchable hierarchy large enough.
> A small pci_align_resource() tweak lets several max-size BARs share one
> window.
>
> NOTE (seeking review): bumping the BAR *resource* also programs the
> hardware ReBAR at enumeration time, which on AMD R9700 tears down
> firmware PSP state and breaks GPU init. The correct shape is likely to size the
> *window* only and leave the child BAR at boot size for the driver to resize.
>
> Signed-off-by: Geramy Loveless <[email protected]>
> ---
>   drivers/pci/rebar.c     |  1 +
>   drivers/pci/setup-bus.c | 62
> +++++++++++++++++++++++++++++++++++++++++++++++++
>   drivers/pci/setup-res.c |  7 ++++++
>   include/linux/pci.h     |  1 +
>   4 files changed, 71 insertions(+)
>
> diff --git a/drivers/pci/rebar.c b/drivers/pci/rebar.c index
> 5bbdc9470..3b621fa3c 100644
> --- a/drivers/pci/rebar.c
> +++ b/drivers/pci/rebar.c
> @@ -190,6 +190,7 @@ int pci_rebar_get_current_size(struct pci_dev *pdev,
> int bar)
>       pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
>       return FIELD_GET(PCI_REBAR_CTRL_BAR_SIZE, ctrl);
>   }
> +EXPORT_SYMBOL_GPL(pci_rebar_get_current_size);
>
>   /**
>    * pci_rebar_set_size - set a new size for a Resizable BAR
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index e8c94aa1d..1bb018e3a 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -2179,6 +2179,59 @@ static void pci_prepare_next_assign_round(struct
> list_head *fail_head,
>    * Second and later try will clear small leaf bridge res.
>    * Will stop till to the max depth if can not find good one.
>    */
> +static int pci_reserve_rebar_cb(struct pci_dev *dev, void *data)
> +{
> +     struct resource *r;
> +     unsigned int i;
> +     int max, cur;
> +
> +     if ((dev->class >> 16) != PCI_BASE_CLASS_DISPLAY)
> +             return 0;
> +     max = pci_rebar_get_max_size(dev, 0);
> +     if (max < 0)
> +             return 0;
> +     cur = pci_rebar_get_current_size(dev, 0);
> +     if (cur < 0 || cur >= max)
> +             return 0;
> +     pci_resize_resource_set_size(dev, 0, max);
> +
> +     /*
> +      * Release every prefetchable BAR so the 64-bit prefetchable window
> +      * enclosing them empties and can be re-sized for the enlarged BAR0.
> +      * The hardware BAR is left untouched for the driver to resize.
> +      */
> +     pci_dev_for_each_resource(dev, r, i) {
> +             if (i >= PCI_BRIDGE_RESOURCES)
> +                     break;
> +             if (r->parent && (r->flags & IORESOURCE_MEM_64) &&
> +                 (r->flags & IORESOURCE_PREFETCH))
> +                     pci_release_resource(dev, i);
> +     }
> +     return 0;
> +}
> +
> +/*
> + * Release the prefetchable bridge windows emptied by
> pci_reserve_rebar_cb() so
> + * the assignment pass re-sizes them to fit the reserved BARs. Recurse
> first so
> + * windows are released bottom-up; the !child test skips windows that
> still hold
> + * other devices' resources.
> + */
> +static void pci_release_rebar_windows(struct pci_bus *bus)
> +{
> +     struct pci_dev *dev;
> +     struct resource *w;
> +
> +     list_for_each_entry(dev, &bus->devices, bus_list)
> +             if (dev->subordinate)
> +                     pci_release_rebar_windows(dev->subordinate);
> +
> +     if (!bus->self)
> +             return;
> +     w = &bus->self->resource[PCI_BRIDGE_PREF_MEM_WINDOW];
> +     if (w->parent && (w->flags & IORESOURCE_MEM_64) && !w->child)
> +             pci_release_resource(bus->self,
> PCI_BRIDGE_PREF_MEM_WINDOW);
> +}
> +
>   void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
>   {
>       LIST_HEAD(realloc_head);
> @@ -2190,6 +2243,15 @@ void
> pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
>       int pci_try_num = 1;
>       enum enable_type enable_local;
>
> +     /*
> +      * Reserve window space for resizable device BARs at their maximum
> +      * size before sizing the bridge windows, so the windows fit the BARs
> +      * a driver will later grow. Only the resource size is set here; the
> +      * hardware BAR is left untouched for the driver to resize.
> +      */
> +     pci_walk_bus(bus, pci_reserve_rebar_cb, NULL);
> +     pci_release_rebar_windows(bus);
> +
>       /* Don't realloc if asked to do so */
>       enable_local = pci_realloc_detect(bus, pci_realloc_enable);
>       if (pci_realloc_enabled(enable_local)) {
> diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
> index 376f09630..db12092e1 100644
> --- a/drivers/pci/setup-res.c
> +++ b/drivers/pci/setup-res.c
> @@ -280,6 +280,13 @@ resource_size_t pci_align_resource(struct pci_dev
> *dev,
>               return res->start;
>
>       remainder = size - ALIGN_DOWN(size, align);
> +     /*
> +      * A window holding several align-sized resources has one tail per
> +      * resource, but only the lowest can tuck below the first aligned
> +      * boundary; the rest sit above it. Relocate a single tail's worth.
> +      */
> +     if (ALIGN_DOWN(size, align) > align)
> +             remainder /= ALIGN_DOWN(size, align) / align;
>       /* Don't mess with size that doesn't align with window size
> granularity */
>       if (!IS_ALIGNED(remainder, pci_min_window_alignment(dev->bus,
> res->flags)))
>               return res->start;
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index d31a8d107..e15d78a31 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1500,6 +1500,7 @@ resource_size_t pci_rebar_size_to_bytes(int size);
>   u64 pci_rebar_get_possible_sizes(struct pci_dev *pdev, int bar);
>   bool pci_rebar_size_supported(struct pci_dev *pdev, int bar, int size);
>   int pci_rebar_get_max_size(struct pci_dev *pdev, int bar);
> +int pci_rebar_get_current_size(struct pci_dev *pdev, int bar);
>   int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size,
>                                    int exclude_bars);
>
>
> ===================================================================
> ===========
> PATCH 2/2 - drm/amdgpu: restrict BAR0 fallback read to SR-IOV VFs only
>              (Mario Limonciello, commit 63b2896378d; verified applies to
> 75a5e1b6b)
> ===================================================================
> ===========
>
>  From 63b2896378d284e2043b44b81a0c044aef0e95fe Mon Sep 17
> 00:00:00 2001
> From: Mario Limonciello <[email protected]>
> Date: Wed, 26 Aug 2026 12:02:53 -0500
> Subject: [PATCH] drm/amdgpu: restrict BAR0 fallback read to SR-IOV VFs only
>
> The BAR0 fallback read path in amdgpu_device_read_fb_via_bar0() was
> introduced as a workaround for SR-IOV virtual functions where the VRAM
> aperture (adev->mman.aper_base_kaddr) is not available during early init.
>
> However, the function currently allows any device (VF, PF, or bare metal)
> to use this fallback path, which is unnecessary overhead for non-VF
> configurations.
>
> Since amdgpu_virt_init() runs during early initialization and sets
> adev->virt.caps before any runtime framebuffer access occurs, we can
> safely check amdgpu_sriov_vf() to restrict this workaround to only
> SR-IOV VFs where it's actually needed.
>
> Fixes: ea8ac194077d ("drm/amdgpu: reduce early full GPU access during
> SR-IOV init")
> Signed-off-by: Mario Limonciello <[email protected]>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++++
>   1 file changed, 4 insertions(+)
>
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -774,6 +774,10 @@
>       u64 end;
>
>       if (!buf || !size)
> +             return -EINVAL;
> +
> +     /* BAR0 workaround only needed for SR-IOV VFs */
> +     if (!amdgpu_sriov_vf(adev))
>               return -EINVAL;
>
>       flags = pci_resource_flags(adev->pdev, 0);
> --
> 2.43.0