[Intel-wired-lan] [BUG] ice: E810-CAM1 custom board - topology/netlist resolution fails (EINVAL on all AQ topology commands), no I2C activity on GPIO expander bus

Hi all,

We're bringing up a custom 4x SFP+ 10G NIC (2x Intel E810-CAM1, following
Intel's documented "4x SFP Native" reference topology from the E810
datasheet, Section 17.2.2.1, including the PCA9575 I2C GPIO expander on
Bus 4 / MDC4_SCL4 / MDIO4_SDA4). We've hit a reproducible issue where
topology-dependent Admin Queue commands consistently fail with EINVAL,
and no I2C activity is ever observed on Bus 4, even though the NVM
netlist content has been verified byte-for-byte identical to Intel's
own reference image. We'd appreciate any input, since we've run out of
public documentation to check against.

System: kernel 6.12.42, ice driver, 4 PFs enumerate correctly
(8086:1593, functions .0/.1/.4/.5). NVM: E810_SFP_10G_NRB_SEC_FW,
currently 1p8p0p2 / NVM 5p01 (EETRACK 80021FBF), successfully flashed
and verified. An earlier, unrelated boot issue (NVM/QSPI access
occasionally skipped at POR, thermal sensitivity) has already been
root-caused and resolved separately.

WHAT WE'VE VERIFIED:

1. Netlist content is correct
   We reverse-engineered the Link Topology Netlist (TLV TypeID 0x11B)
   from our NVM image and confirmed the PCA9575 (I2C addr 0x20, Handle
   29) is present with RESET_N/INT_N on native GPIO (SDP5/SDP6) and I2C
   on Bus 4 (Driving IO#4 of the SoC I/O Widget node, Handle 0) - byte
   for byte identical to Intel's E810_XXVDA4 reference NVM image. We
   re-verified this against the actual flashed chip content (via
   `ethtool -e`) after a fresh NVM update, with identical results.

2. Admin Queue commands consistently fail with EINVAL, not EBUSY
   `ethtool -m <iface>` (Read/Write SFF EEPROM, opcode 0x06EE) returns
   "Input/output error" every time. Raw AQ trace via dynamic debug shows
   the command is correctly formed (logical port 0, I2C addr 0x50,
   offset 0, read op), but firmware responds with retval 0x000E
   (EINVAL) - not 0x0C (EBUSY), which is the documented code for a real
   I2C bus timeout/busy condition. Response time across 50 consecutive
   calls: 40-68us, essentially constant with no jitter - too fast and
   too regular for an actual I2C bus attempt (100kHz would need >90us
   just for address+ACK).

   Same failure class shows up independently via `ethtool -d`: the
   "Extended Regs" (SerDes equalization) section, which
   ice_get_extended_regs() only populates after a successful
   ice_get_port_topology() -> "Get Netlist Node" AQ call (Cage context),
   is entirely zero-filled.

   At boot, dmesg shows the same thing on all 4 ports identically:
     ice 0000:01:00.0: Get PHY capabilities failed status = -5, continuing anyway
     ice 0000:01:00.0: ice_init_nvm_phy_type failed: -5
   (via ICE_AQC_REPORT_TOPO_CAP)

3. No I2C activity observed, ever
   Oscilloscope on Bus 4 (I2C_CLK/I2C_DAT) shows the line pulled high at
   power-up (pull-up confirmed functional), and going low briefly very
   early in boot - but never toggling again, confirmed over a 10s
   capture window. Same result on Bus 0 (native SFP0 module bus) when
   triggered explicitly at runtime via ethtool -m. A known-good Intel
   reference card (E810-XXVDA4, tested standalone on a motherboard with
   no CPU installed at all) reliably shows a clean, well-formed 100kHz
   I2C burst on Bus 4 around 860-920ms post power-on. Our board shows
   nothing there.

4. QSPI timing shows a reproducible 1.7s stall + oversized retry
   Comparing raw QSPI_CLK captures (CSV, edge timestamps) between our
   board and the reference: both match closely up through ~860ms
   (including an ~334ms linear read phase we believe is FW SHA
   authentication). At that point the reference proceeds into small,
   regular ~90-100ms housekeeping bursts continuing indefinitely. Our
   board instead goes completely silent for ~1.72s, then produces a
   single burst ~3x larger (by edge count) than the reference's
   equivalent config-read burst, then goes silent again. This
   stall-then-oversized-retry pattern repeats up to 3 times within a
   10s window, never resolving into the reference's steady-state
   pattern, and never followed by any I2C activity.

5. Chip-level status registers show a clean boot, no recovery mode
   Read directly via UEFI Shell (BAR0 memory-mapped, before any OS/
   driver loads):
     GL_MNG_SHA_EXTEND_STATUS (0x00083148): STAGE=100b, FW_HALTED=0,
       DONE=1 -> main FW authenticated and running cleanly
     GL_MNG_FWSM (0x000B6134): 0x00000000 -> no POR_AL_FAILURE, no
       PCIR_AL_FAILURE, no EXT_ERR_IND. Per datasheet Section 15.4.3,
       FW_MODES bit 1 (Recovery Mode indication) = 0, so Recovery Mode
       is explicitly ruled out.
     GL_FWSTS (0x00083048): FWROWD (bit 8, Firmware Reset on Watchdog) =
       0 -> no watchdog-triggered reset flagged.

QUESTIONS:

- What internal criteria does firmware use to decide a netlist
  configuration is invalid/incompatible for a given port (this exact
  phrase - "Incompatible Netlist configuration" - appears in datasheet
  Section 15.4.1.1 as a named recovery-triggering failure scenario, but
  the detection logic itself isn't documented anywhere we can find)?

- Is there a way to enable FW logging (i2c / link_topology modules)
  early enough to catch whatever happens in the ~860ms-2.6s window,
  given the host driver isn't loaded yet at that point on a normal
  boot?

- Given verified-identical netlist content, clean POR/auth status, and
  no recovery mode - but consistent EINVAL on every topology-dependent
  AQ command on all 4 ports - what would cause exactly this
  combination?

Happy to share the full netlist dump, raw AQ traces, and oscilloscope
CSVs if useful. Thanks for reading this far.

Best regards,

Fabrice Hoffmann

Development



IPETRONIK GmbH & Co.KG

Im Rollfeld 28

D-76532 Baden-Baden



Phone: +49 7221 9922-486

Fax: +49 7221 9922-59486

mailto: <mailto:%0dmailto:%[email protected]> [email protected]<mailto:%0dmailto:%[email protected]>

web: http://www.ipetronik.com



Impressum/Imprint: https://www.ipetronik.com/impressum