RPi4B boot -v "pci0: failed to allocate bus number" and "device_attach: pci0 attach returned 6"; more

For reference, the example context used is
(long line output split for readability):

# uname -apKU
FreeBSD generic 13.1-STABLE FreeBSD 13.1-STABLE #0
stable/13-n253133-b51ee7ac252c: Wed Nov 23 03:36:16 UTC 2022
[email protected]:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC
arm64 aarch64 1301509 1301509

(So: not my build.)

From the boot -v capture:

pcib0: <BCM2838-compatible PCI-express controller> mem 0x7d500000-0x7d50930f irq 80,81 on simplebus2
pcib0: parsing FDT for ECAM0:
pcib0:  PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0:  PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: Bus is not cache-coherent
pcib0: hardware identifies as revision 0x304.
pcib0: note: reported link speed is 5.0 GT/s.
pci1: <PCI bus> on pcib0
pci1: domain=0, physical bus=0
found-> vendor=0x14e4, dev=0x2711, revid=0x00
        domain=0, bus=0, slot=0, func=0
        class=06-04-00, hdrtype=0x01, mfdev=0
        cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
        lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
        intpin=a, irq=0
        powerspec 3  supports D0 D3  current D0
        secbus=1, subbus=1
pcib1: <PCI-PCI bridge> irq 91 at device 0.0 on pci1
pcib0: rman_reserve_resource: start=0xc0000000, end=0xc00fffff, count=0x100000
pcib1:   domain            0
pcib1:   secondary bus     1
pcib1:   subordinate bus   1
pcib1:   memory decode     0xc0000000-0xc00fffff
pci2: <PCI bus> on pcib1
pcib1: allocated bus range (1-1) for rid 0 of pci2
pci2: domain=0, physical bus=1
found-> vendor=0x1106, dev=0x3483, revid=0x01
        domain=0, bus=1, slot=0, func=0
        class=0c-03-30, hdrtype=0x00, mfdev=0
        cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
        lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
        intpin=a, irq=0
        powerspec 3  supports D0 D3  current D0
        MSI supports 4 messages, 64 bit
        map[10]: type Memory, range 64, base 0, size 12, memory disabled
pcib1: slot 0 INTA is routed to irq 92
bcm_xhci0: <VL805 USB 3.0 controller (on the Raspberry Pi 4b)> irq 92 at device 0.0 on pci2
bcm_xhci0: note: xhci firmware not found.
bcm_xhci0: note: installing xhci firmware.
bcm_xhci0: note: xhci firmware detected; firmware is revision 138a1.
pcib1: allocated memory range (0xc0000000-0xc0000fff) for rid 10 of bcm_xhci0
bcm_xhci0: Lazy allocation of 0x1000 bytes rid 0x10 type 3 at 0xc0000000
bcm_xhci0: 32 bytes context size, 64-bit DMA
bcm_xhci0: attempting to allocate 1 MSI vectors (4 supported)
bcm_xhci0: using IRQ 93 for MSI
bcm_xhci0: MSI enabled
bcm_xhci0: (New XHCI DeviceId=0x34831106)
usbus0 on bcm_xhci0
bcm_xhci0: usbpf: Attached
bcm_xhci0: note: switched to 32-bit DMA.
pci0: <PCI bus> on pcib0
pci0: failed to allocate bus number
device_attach: pci0 attach returned 6

In order to get more directly comparable output, I tried
using lspci port and use of lspci on raspiOS64 (my
abbreviation) and fedora 37.

raspiOS64:

# lspci -v
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 10) (prog-if 00 [Normal decode])
        Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff [size=4K]
        Memory behind bridge: c0000000-c00fffff [size=1M]
        Prefetchable memory behind bridge: [disabled]
        Capabilities: [48] Power Management version 3
        Capabilities: [ac] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
        Capabilities: [240] L1 PM Substates

01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01) (prog-if 30 [XHCI])
        Subsystem: VIA Technologies, Inc. VL805 USB 3.0 Host Controller
        Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0
        Flags: bus master, fast devsel, latency 0, IRQ 63
        Memory at 600000000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Capabilities: [c4] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: xhci_hcd

fedora:

# lspci -v
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (rev 20) (prog-if 00 [Normal decode])
        Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0
        Flags: bus master, fast devsel, latency 0, IRQ 36
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        Memory behind bridge: 00000000-000fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Capabilities: [48] Power Management version 3
        Capabilities: [ac] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
        Capabilities: [240] L1 PM Substates
        Kernel driver in use: pcieport

01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller (rev 01) (prog-if 30 [XHCI])
        Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller
        Device tree node: /sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0
        Flags: bus master, fast devsel, latency 0, IRQ 38
        Memory at 600000000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Capabilities: [c4] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: xhci_hcd

(00:00.0's "Memory behind bridge" displayed based on
different spaces, unfortunately. Not as uniform as I
was intending. Still. . .)

FreeBSD:

# lspci -v
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 91
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000-0fff [size=4K] [16-bit]
        Memory behind bridge: c0000000-c00fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Capabilities: [48] Power Management version 3
        Capabilities: [ac] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
        Capabilities: [240] L1 PM Substates

01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller (rev 01) (prog-if 30 [XHCI])
        Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller
        Flags: bus master, fast devsel, latency 0, IRQ 92
        Memory at c0000000 (64-bit, non-prefetchable)
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Capabilities: [c4] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting

Note the 01:00.0's "Memory at": c0000000 vs. 600000000 for
the two linux exmaples. (Also: No size shown for FreeBSD.)

According to "PhilE Raspberry Pi Engineer & Forum Moderator" at:

https://forums.raspberrypi.com/viewtopic.php?t=288902&sid=792675eb2c6513e936b1cd9fe385a038&start=75

QUOTE
It's worth stating a few facts:
(1) The PCIe controller in BCM2711 has a single outbound window. This means that all addresses for PCIe devices have to be mapped in a single block, such that the spacing in PCIe space matches the spacing in host physical address space.
(2) The size of the outbound window must be a power of two, and the base address of the outbound window must be aligned to that size.
END QUOTE

According to bcm2711-peripherals.pdf figure 1 the standard
placement of PCIe space is 0x600000000..0x7ffffffff (avoiding
overlaps with anything else in any space).

For reference, the live Device Tree for the FreeBSD context
has for pcie@7d500000 :

			ranges = <0x02000000 0x00000000 0xc0000000 0x00000006 0x00000000 0x00000000 0x40000000>;

The "0x00000006 0x00000000" part agrees with that "figure 1".
(More recent vintages agree for that part as well.)

I'm far from expert for such things but the 01:00.0 having
"Memory at c0000000 (64-bit, non-prefetchable)" looks
risky for PCI'e outbound memory mapping due to overlaps
having potential consequences.


An additional note of something that might be involved:

The RPi* firmware version that FreeBSD uses still has:

		pcie@7d500000 {
. . .
			pci@1,0 {
. . .
				usb@1,0 {
. . .

but modern firmware (such as in the raspiOS64 and
fedora I used) has:

                pcie@7d500000 {
. . .
                        pci@0,0 {
. . .
                                usb@0,0 {
. . .

I do not know if that is somehow related to
what I'm reporting or not.

===
Mark Millard
marklmi at yahoo.com