Re: Oracle S7-2 and ConnextX4-LX, ConnectX3 (VPI), and i40e

On 9/9/25 23:41, Tony Rodriguez wrote:
> For those with an Oracle S7-2, but this may be valid for other Sparc 
> systems as well: (just a heads-up)
> 
> 1)  The "ConnectX-4 lx" NIC appears to work with Solaris 11.4 and Debian 
> 12.  However, on Debian 12, the "ConnectX-4 lx" will generate many 
> tainted kernel messages (related to mlx5 module).  Tried latest 
> firmwares for this card, (from Dell and Nvidia), all have the same 
> issues.  In my case, the tainted messages appear when configured to use 
> systemd-networkd and during system boot/reboot/poweroff.
> 
> 2) As such, decided to downgrade and use "ConnectX-3 CX354A-FCBT 
> (VPI)".  This particular card works with Solaris 11.4 and Debian 12.  It 
> also works in Ethernet and Infiniband modes.  Both Solaris 11.4 and 
> Debian 12 see this nic in either mode.  It uses the mlx4 module and 
> appears to avoid generating tainted kernel messages.
> 
> 
> Note(S):
> 
> 2A) For infiniband verbs, the (rdma-core package) a.k.a. (infiniband- 
> diags package replacement) will need to add a specific build option to 
> create the ibverbs executables.   I did so by building from source. 
> Might be helpful to modify the rdma-core build so it will automatically 
> generate all ibverbs such as ibstat, ibstatus, etc.
> 
> 2B) opensm package has a dependency with infiniband-diags package. 
> However, the infiniband-diags package will not install (possibly because 
> of rdma-core).  Guessing opensm package will require an update to use 
> rdma-core and possibly dropping infiniband-diags.  Didn't attempt to 
> build opensm from scratch, so I can't say for sure yet. Unsure if 
> infiniband-diags is required by other packages at this point.
> 
> 2C) On another note, the "ConnectX-3 CX314A-BCCT" will work with Debian 
> 12 in Ethernet only mode, but it is not detected/recognized by Solaris 
> 11.4.  If Solaris 11.4, Debian 12, Ethernet, and Infiniband 
> compatibility are required, make sure to use the "ConnectX-3 CX354A-FCBT 
> (VPI)" NIC.
> 
> 
> 3) The built-in i40e 10GBE ports on the S7-2 generate the following 
> messages within Debian 12. It works and doesn't appear to be a major 
> issue. Just unsure if this unaligned access may impact performance on 
> some level?  iperf3 didn't show any noticeable performance drop, but it 
> may be something to keep an eye on.
> 
> For Example:
> i40e 0000:03:00.1: DCB init failed -16, disabled
> Kernel unaligned access at TPC[101a7f28] i40e_config_netdev+0x1e8/0x2e0 
> [i40e]
> 


Thank you for this very valuable information. I have an S7-2 being
shipped to me and I expect it before end of month. I look forward
to working on this and coordinating with you.