[Bug 297064] em(4) 82541PI silent TX/RX wedge

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297064

            Bug ID: 297064
           Summary: em(4) 82541PI silent TX/RX wedge
           Product: Base System
           Version: Unspecified
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: [email protected]
          Reporter: [email protected]

This is an issue since 2017, see bug #220997.

Diagnosis behind the attached patch.

**Hardware.** em(4) 82541PI (a 32-bit PCI adapter) as the WAN NIC on an AMD
RS780 northbridge + SB700 southbridge board with 8 GB RAM. The adapter sits on
the legacy 33 MHz PCI bus behind the SB700 PCI-to-PCI bridge.

**Symptom.** After some uptime the interface silently stops passing traffic:
the transmit and receive engines freeze together (TDH/RDH stop advancing) with
no kernel message, and only a reset/reboot recovers it. Instrumentation showed
the wedge coincides exactly with a PCI **Received Master Abort** latching in
the
adapter's PCI status word, and it occurs only for DMA to buffers above the 4 GB
line. Constraining the adapter's DMA to below 4 GB stops the aborts completely.

In other words the adapter master-aborts its own bus-master DMA whenever a
descriptor or packet buffer is placed above 4 GB — the >4 GB dual-address-cycle
(DAC) path from this adapter to host memory does not work on this chipset.

**What was checked and ruled out as the cause:**

- **The adapter / 82541 errata.** The 82541PI Specification Update lists no
  >4 GB / DAC / DMA-master-abort erratum, and the controller is specified as
  64-bit-DMA (DAC) capable. Not the NIC.
- **Northbridge (RS780) `TOP_OF_DRAM2`.** Read live: enabled and sized to the
  true top of RAM — the northbridge's >4 GB DMA window is correctly configured
  (bit-for-bit what the Linux RS690 64-bit-DMA quirk would program). Not the
  cause.
- **Southbridge (SB700) PCI-bridge "Upstream Dual Address Window"** (PCIB
config
  `0x50` bit 0, per the AMD SB7xx Register Programming Requirements). Read
live:
  enabled. Not the cause.
- **Remaining candidates.** With both documented >4 GB-DMA enables correctly
  set, the only possibilities left are the SB700 A-Link 64-bit forwarding
  settings (silicon-revision-specific, reachable only via indirect I/O) or an
  undocumented chipset erratum — neither safely determinable on a production
  machine, and both chip-specific.

**Conclusion.** On this RS780+SB700 board the >4 GB DMA path for a 32-bit PCI
bus-master does not work despite the documented enables being set. Rather than
chase an undocumented chipset defect, the robust and low-cost fix is to keep
the
adapter's DMA below 4 GB.

**Fix (attached patch).** Adds a read-only tunable `hw.em.dma_width` that caps
the DMA addressing width; set it to 32 and busdma bounces any buffer that would
land above the limit, so the adapter only ever sees addresses it can reach. The
default of 0 keeps current behaviour and imposes no limit, so the change is
inert unless enabled per machine. With `hw.em.dma_width=32` set, the master
aborts stop (busdma shows the expected bounce activity) and the interface stays
up.

-- 
You are receiving this mail because:
You are the assignee for the bug.