AW: [BUG] ice: E810-CAM1 custom board - topology/netlist resolution fails (EINVAL on all AQ topology commands), no I2C activity on GPIO expander bus

Hi all,

Follow-up to my earlier report (no replies yet, but we made concrete
progress on our end that narrows this down significantly).

We built the ice driver from a 6.14-based kernel (our original test
system was on 6.12, which lacks the devlink health reporters) and
tested the same board on different host hardware. The original EINVAL/
EIO behavior on every topology-dependent AQ command reproduces
identically on this new setup - so that part is confirmed independent
of host platform, CPU, and kernel version.

With devlink health now available, we finally got a direct,
plain-text firmware diagnosis instead of raw AQ error codes:

  $ devlink health diagnose pci/0000:05:00.0 reporter fw
  Syndrome: 274  Description: LOM topology netlist is corrupted.
  Possible Solution: Update to the latest NVM image.

  $ devlink health diagnose pci/0000:05:00.0 reporter port
  Syndrome: 275  Description: Unrecoverable netlist error.
  Possible Solution: Update to the latest NVM image.
  Port Number: 0

  $ devlink health diagnose pci/0000:05:00.1 reporter port
  Syndrome: 0
  (ports .4 and .5 likewise report Syndrome: 0 on the port reporter)

So: the "fw" reporter (global) reports netlist corruption on all 4
PFs identically, but the "port" reporter only flags an actual
unrecoverable error on port 0 - ports 1-3 come back clean. Combined
with the dmesg output being identical on all 4 ports ("Get PHY
capabilities failed status = -5" / "ice_init_nvm_phy_type failed: -5"),
our working theory is that netlist/topology processing walks the
node table in order, hits an unrecoverable condition at the first
cage node (port 0), and aborts the entire walk - which would explain
why ports 1-3 are simply never reached/checked (hence "healthy" by
omission) while the overall topology init still fails for all 4 PFs.

We're fairly confident this isn't a NVM data-integrity problem in the
usual sense:
- The board is already on the latest available NVM/FW (5.01,
  EETRACK 80021FBF; nvmupdate reports "Update Not Available")
- We reverse-engineered the Link Topology Netlist (TLV 0x11B) from
  our image and confirmed it's byte-for-byte identical to Intel's own
  E810_XXVDA4 reference image at the node/IO table level (PCA9575 I2C
  GPIO expander correctly present at Bus 4, RESET_N/INT_N on native
  SDP5/SDP6, matching the datasheet's "4x SFP Native" reference
  schematic, Section 17.2.2.1)
- Re-verified this against the actual flashed chip content (ethtool -e
  readback) after a fresh NVM update - identical result
- NVM Update Checksum (AQ 0x0706) reports valid
- GL_MNG_FWSM and GL_MNG_SHA_EXTEND_STATUS (read via UEFI Shell/BAR0,
  pre-OS) show a completely clean boot: main FW authenticated (STAGE=
  100b), no POR_AL_FAILURE/PCIR_AL_FAILURE, no EXT_ERR_IND, FW_MODES
  bit1 (Recovery Mode) = 0

Given all of that checks out, "corrupted" seems to mean something
more like a structural/semantic validation failure specific to the
port-0 cage node (or something processed immediately before it),
rather than actual bit-level data corruption - but we don't have
visibility into what that validation actually checks.

Questions, now with concrete syndrome data attached:

- What does the firmware's netlist walk actually validate at/around
  the first cage node that could produce "Unrecoverable netlist error"
  (syndrome 275) specifically for port 0, while leaving ports 1-3
  unflagged?
- Is there a way to get more granular logging out of this
  specifically (we tried fwlog modules i2c/link_topo/health/adminq via
  debugfs, and can correlate some opaque message IDs with specific
  AQ calls, but can't decode their actual content without an internal
  format-string table)?
- The suggested remediation ("update to latest NVM image") doesn't
  apply since we're already on the latest - is there a separate
  netlist/board-config bundle that could be out of sync independent
  of the FW/NVM version number?

Happy to share the full netlist dump, oscilloscope captures, and raw
AQ traces if that's useful to anyone looking into this.

Thanks,
Fabrice


________________________________
Von: Intel-wired-lan <[email protected]> im Auftrag von Hoffmann, Fabrice <[email protected]>
Gesendet: Montag, 17. August 2026 18:25
An: [email protected] <[email protected]>
Betreff: [Intel-wired-lan] [BUG] ice: E810-CAM1 custom board - topology/netlist resolution fails (EINVAL on all AQ topology commands), no I2C activity on GPIO expander bus

Hi all,

We're bringing up a custom 4x SFP+ 10G NIC (2x Intel E810-CAM1, following
Intel's documented "4x SFP Native" reference topology from the E810
datasheet, Section 17.2.2.1, including the PCA9575 I2C GPIO expander on
Bus 4 / MDC4_SCL4 / MDIO4_SDA4). We've hit a reproducible issue where
topology-dependent Admin Queue commands consistently fail with EINVAL,
and no I2C activity is ever observed on Bus 4, even though the NVM
netlist content has been verified byte-for-byte identical to Intel's
own reference image. We'd appreciate any input, since we've run out of
public documentation to check against.

System: kernel 6.12.42, ice driver, 4 PFs enumerate correctly
(8086:1593, functions .0/.1/.4/.5). NVM: E810_SFP_10G_NRB_SEC_FW,
currently 1p8p0p2 / NVM 5p01 (EETRACK 80021FBF), successfully flashed
and verified. An earlier, unrelated boot issue (NVM/QSPI access
occasionally skipped at POR, thermal sensitivity) has already been
root-caused and resolved separately.

WHAT WE'VE VERIFIED:

1. Netlist content is correct
   We reverse-engineered the Link Topology Netlist (TLV TypeID 0x11B)
   from our NVM image and confirmed the PCA9575 (I2C addr 0x20, Handle
   29) is present with RESET_N/INT_N on native GPIO (SDP5/SDP6) and I2C
   on Bus 4 (Driving IO#4 of the SoC I/O Widget node, Handle 0) - byte
   for byte identical to Intel's E810_XXVDA4 reference NVM image. We
   re-verified this against the actual flashed chip content (via
   `ethtool -e`) after a fresh NVM update, with identical results.

2. Admin Queue commands consistently fail with EINVAL, not EBUSY
   `ethtool -m <iface>` (Read/Write SFF EEPROM, opcode 0x06EE) returns
   "Input/output error" every time. Raw AQ trace via dynamic debug shows
   the command is correctly formed (logical port 0, I2C addr 0x50,
   offset 0, read op), but firmware responds with retval 0x000E
   (EINVAL) - not 0x0C (EBUSY), which is the documented code for a real
   I2C bus timeout/busy condition. Response time across 50 consecutive
   calls: 40-68us, essentially constant with no jitter - too fast and
   too regular for an actual I2C bus attempt (100kHz would need >90us
   just for address+ACK).

   Same failure class shows up independently via `ethtool -d`: the
   "Extended Regs" (SerDes equalization) section, which
   ice_get_extended_regs() only populates after a successful
   ice_get_port_topology() -> "Get Netlist Node" AQ call (Cage context),
   is entirely zero-filled.

   At boot, dmesg shows the same thing on all 4 ports identically:
     ice 0000:01:00.0: Get PHY capabilities failed status = -5, continuing anyway
     ice 0000:01:00.0: ice_init_nvm_phy_type failed: -5
   (via ICE_AQC_REPORT_TOPO_CAP)

3. No I2C activity observed, ever
   Oscilloscope on Bus 4 (I2C_CLK/I2C_DAT) shows the line pulled high at
   power-up (pull-up confirmed functional), and going low briefly very
   early in boot - but never toggling again, confirmed over a 10s
   capture window. Same result on Bus 0 (native SFP0 module bus) when
   triggered explicitly at runtime via ethtool -m. A known-good Intel
   reference card (E810-XXVDA4, tested standalone on a motherboard with
   no CPU installed at all) reliably shows a clean, well-formed 100kHz
   I2C burst on Bus 4 around 860-920ms post power-on. Our board shows
   nothing there.

4. QSPI timing shows a reproducible 1.7s stall + oversized retry
   Comparing raw QSPI_CLK captures (CSV, edge timestamps) between our
   board and the reference: both match closely up through ~860ms
   (including an ~334ms linear read phase we believe is FW SHA
   authentication). At that point the reference proceeds into small,
   regular ~90-100ms housekeeping bursts continuing indefinitely. Our
   board instead goes completely silent for ~1.72s, then produces a
   single burst ~3x larger (by edge count) than the reference's
   equivalent config-read burst, then goes silent again. This
   stall-then-oversized-retry pattern repeats up to 3 times within a
   10s window, never resolving into the reference's steady-state
   pattern, and never followed by any I2C activity.

5. Chip-level status registers show a clean boot, no recovery mode
   Read directly via UEFI Shell (BAR0 memory-mapped, before any OS/
   driver loads):
     GL_MNG_SHA_EXTEND_STATUS (0x00083148): STAGE=100b, FW_HALTED=0,
       DONE=1 -> main FW authenticated and running cleanly
     GL_MNG_FWSM (0x000B6134): 0x00000000 -> no POR_AL_FAILURE, no
       PCIR_AL_FAILURE, no EXT_ERR_IND. Per datasheet Section 15.4.3,
       FW_MODES bit 1 (Recovery Mode indication) = 0, so Recovery Mode
       is explicitly ruled out.
     GL_FWSTS (0x00083048): FWROWD (bit 8, Firmware Reset on Watchdog) =
       0 -> no watchdog-triggered reset flagged.

QUESTIONS:

- What internal criteria does firmware use to decide a netlist
  configuration is invalid/incompatible for a given port (this exact
  phrase - "Incompatible Netlist configuration" - appears in datasheet
  Section 15.4.1.1 as a named recovery-triggering failure scenario, but
  the detection logic itself isn't documented anywhere we can find)?

- Is there a way to enable FW logging (i2c / link_topology modules)
  early enough to catch whatever happens in the ~860ms-2.6s window,
  given the host driver isn't loaded yet at that point on a normal
  boot?

- Given verified-identical netlist content, clean POR/auth status, and
  no recovery mode - but consistent EINVAL on every topology-dependent
  AQ command on all 4 ports - what would cause exactly this
  combination?

Happy to share the full netlist dump, raw AQ traces, and oscilloscope
CSVs if useful. Thanks for reading this far.

Best regards,

Fabrice Hoffmann

Development



IPETRONIK GmbH & Co.KG

Im Rollfeld 28

D-76532 Baden-Baden



Phone: +49 7221 9922-486

Fax: +49 7221 9922-59486

mailto: <mailto:%0dmailto:%[email protected]> [email protected]<mailto:%0dmailto:%[email protected]>

web: http://www.ipetronik.com




Impressum/Imprint: https://www.ipetronik.com/impressum

Impressum/Imprint: https://www.ipetronik.com/impressum