[Bug 296989] ice: failed Tx scheduler t opology config leaves global config lock held ; AdminQ dead, Safe Mode, unusable E810 (NVM ≥4.60)

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296989

            Bug ID: 296989
           Summary: ice: failed Tx scheduler topology config leaves global
                    config lock held; AdminQ dead, Safe Mode, unusable
                    E810 (NVM ≥4.60)
           Product: Base System
           Version: 15.1-STABLE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Many People
          Priority: ---
         Component: kern
          Assignee: [email protected]
          Reporter: [email protected]

On Intel E810 (E810-XXV-2) with NVM ≥4.60, FreeBSD 15.1 ice (1.43.3-k) can fail
during transmit-balancing / 5-layer Tx topology setup. After the failure the
interface is effectively dead: DDP load reports an unknown error, the driver
enters Safe Mode, and subsequent AdminQ commands time out.

The same card and NVM work under Linux ice (Proxmox/kernel 7.x) on the same
host.

Hardware / firmware

• NIC: Intel Ethernet Network Adapter E810-XXV-2 (8086:159b)
• NVM 4.50: path largely avoided; interface usable (separate FEC issue not the
subject of this PR)
• NVM ≥4.60 / 5.01: tx_sched_topo_comp_mode_en advertised; Tx topology path
runs
• Under Linux after boot: tx_scheduling_layers permanent value 5; DDP 1.3.43.0
loads; link 25G RS-FEC OK

FreeBSD / OPNsense symptoms (dmesg)

ice0: Error configuring transmit balancing: ICE_ERR_AQ_ERROR
ice0: An unknown error occurred when loading the DDP package.  Entering Safe
Mode.
ice0: fw ... nvm 4.60 ...
ice0: Using 1 Tx and Rx queues
...
ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT
aq_err OK
ice0: ice_intersect_phy_types_and_speeds: ice_aq_get_phy_caps (ACTIVE) failed;
status ICE_ERR_AQ_TIMEOUT
ice0: Failed to set LAN Tx queue 0 (TC 0, handle 0) context, err
ICE_ERR_AQ_TIMEOUT
ice0: Unable to configure the main VSI for Tx: ENODEV
ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT

Root cause (FreeBSD 15.1 sources)

In sys/dev/ice/ice_ddp_common.c, ice_cfg_tx_topo():

1. Acquires ICE_GLOBAL_CFG_LOCK_RES_ID (firmware global configuration lock used
for DDP / topology programming).
2. Calls ice_get_set_tx_topo(..., set=true) (AdminQ opcode 0x0417).
3. On failure, returns immediately without ice_reset(..., ICE_RESET_CORER).
4. CORER is only issued on the success path; CORER is what clears the global
configuration lock.

While that lock is held, firmware rejects most AdminQ commands. The caller then
tries ice_copy_and_init_pkg(), fails to complete global config, reports an
unknown DDP error, enters Safe Mode, and every later AQ op times out — matching
the dmesg cascade above.

This is not primarily “missing ice_ddp module.” Even with ice_ddp_load="YES", a
failed topology SET that leaks the lock makes DDP download and normal attach
fail.

The path is gated on FW capability tx_sched_topo_comp_mode_en (set on NVM
≥4.60), which is why NVM 4.50 does not hit this brick path the same way.

Upstream Linux: known and fixed

Linux fixed the identical failure mode:

• Patch: “ice: don't leave device non-functional if Tx scheduler config fails”
  Jacob Keller (Intel), 17 July 2025
  https://www.mail-archive.com/[email protected]/msg12135.html
• Fixes: 91427e6d9030 (“ice: Support 5 layer topology”)

That commit message describes the same lock/CORER/DDP/Safe Mode chain. The fix:

• Always trigger CORER after the global lock has been acquired for topology
programming, whether SET succeeds or fails
• Re-init HW after CORER inside ice_cfg_tx_topo()
• Treat topology apply failure as non-fatal so the device still loads (default
topology); only post-CORER re-init failure aborts probe

Important note from that patch (applies to FreeBSD port as well): you must not
“release” the global lock with a normal unlock on SET failure — that signals FW
that global configuration completed and breaks subsequent DDP loads. CORER is
the required recovery.

FreeBSD releng/15.1 / current main sys/dev/ice still has the early return
status after failed set_tx_topo and does not always CORER after taking the
lock.

Suggested FreeBSD fix

Port the Linux change to:

• sys/dev/ice/ice_ddp_common.c — ice_cfg_tx_topo()
• sys/dev/ice/ice_lib.c — ice_load_pkg_file() (caller: do not treat topology AQ
failure as fatal; continue DDP after successful re-init)

Optional hardening (seen when NVM already selected 5-layer tree): treat
num_tx_sched_layers == 5 as already applied and skip reprogramming; match Linux
by copying the 5-layer section into a zeroed ICE_AQ_MAX_BUF_LEN buffer before
SET.

Workarounds (for users)

1. Stay on NVM 4.50 (avoids capability path; may need FEC sysctls for 25G).
2. Loader tunable: hw.ice.debug.tx_balance_en=0 (skips topology step; does not
fix the lock leak if something else hits the path).
3. Host Linux PF + SR-IOV, guest uses iavf (avoids FreeBSD ice topology path
entirely).

How to reproduce

1. E810-XXV-2 with NVM ≥4.60, FreeBSD 15.1 / OPNsense 26.7, PF passed through
(or bare metal).
2. Boot with if_ice + ice_ddp loaded.
3. Observe transmit-balancing AQ error, Safe Mode, then AdminQ timeouts.
4. Same card under Linux: DDP loads, link up, no Safe Mode.

Related (not the same)

• FreeBSD Bug 274440 — different historical “Failed to set LAN Tx queue”
reports, not this topology lock-leak.

-- 
You are receiving this mail because:
You are the assignee for the bug.