https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297383
--- Comment #2 from tunnelweb <[email protected]> ---
With the four throughput fixes from earlier in this bug in place the adapter
does hold 979 Mbps, and then the link starts flapping. Hundreds of "ue0: link
state changed" a minute, the WAN lease goes with it, and the house has no
internet until it settles. This is not fix 2 coming back: the uninitialised
stack reads in axge_read_cmd_1() and axge_read_cmd_2() are fixed, and the USB
error counters sit at zero the whole time, so the register reads are succeeding
and returning correct values. Something is mangling the result after the read.
Every down event I logged carried mii_media_active == 0x20. That is a bare
IFM_ETHER with subtype 0. ukphy_status() opens by setting mii_media_status =
IFM_AVALID and mii_media_active = IFM_ETHER and only fills the subtype in
later, and every exit path out of it sets either a real subtype or IFM_NONE. So
subtype 0 is a value no finished pass can leave behind. Seeing it arrive at
miibus_statchg means the reader picked up the media words while another pass
was halfway through building them.
The timing says the same thing. Every DOWN was followed by an UP microseconds
later, and an autoneg cycle takes on the order of a second, so nothing was
negotiated in the gap. Every one of those UPs reported 0x100030. Across
hundreds of events it never once came back at 100baseTX or half duplex.
The MII layer assumes a pass over mii_media_status and mii_media_active is
atomic under the driver lock. Over USB it isn't. Every PHY register access is a
control transfer, and usbd_do_request_flags(udev, mtx, ...) drops the mutex it
is handed while it waits, so the driver lock goes away and comes back on every
register read. mii_tick() and mii_pollstat() interleave freely.
What fooled me for a long time is that both callers look correct.
axge_ifmedia_sts() does hold AXGE_LOCK across mii_pollstat(), and axge_tick()
runs with it held. Both look correctly serialised. They are not, because the
lock does not survive the transfer.
Then it feeds itself. mii_linkchg() sees IFM_ACTIVE clear and calls
if_link_state_change(). Whatever reacts to that goes off and queries interface
media, and that query is another concurrent pass, so it tears too. I have not
pinned down exactly which consumer is doing the querying on my box, but the PHY
read rate drops by an order of magnitude once the storm stops, so something was
clearly reacting to it. It once ran 2h18m completely clean at 979 Mbps, then
after an unrelated power cycle, same kernel, it went straight back to flapping
constantly, and that cost me a lot of time before I stopped trusting a clean
run as evidence of anything.
For a while I was sure something was clearing BMSR_LINK, so I built a kernel
that lied and always reported BMSR_LINK | BMSR_ACOMP. Made no difference at
all. That is what pushed me towards the clobber happening between the reads
rather than in the bit.
What else I ruled out, and how. The n_ figures are counters I added to the
driver.
- MII restarting autoneg on its own. I instrumented every BMCR write:
n_bmcr_reset = 2 and n_bmcr_aneg = 2 for a whole run.
- USB transfer errors: n_tx_err = 0, n_rx_err = 0, last_tx_error = none, the
USB layer is healthy the whole way through.
- The driver resetting the PHY itself, n_init = 1, n_mediaupd = 1, n_reset = 1
on the run where n_statchg reached 659.
- It isn't load or contention with the bulk path either. It flaps at idle, 426
link events in 2.5 minutes at roughly 13 pkt/s.
USB link speed (SUPER every time), CPU (97% idle at 571 Mbps), checksum offload
and USB power were all ruled out earlier in the bug.
So I serialised whole passes. axge_mii_enter() and axge_mii_exit() take and
release a busy flag in the softc, msleep()ing on it with sc_mtx, and
mii_tick(), mii_pollstat() and mii_mediachg() now run between them. A second
mutex can't do this job, because it would have to be held across a sleeping
control transfer. The flag works because sc_mtx is the lock the USB stack drops
and picks back up for you, so the flag is the thing that survives the drop, and
a second pass finds it set and sleeps. The ioctl path goes through the same
enter/exit. Detach is the part I am least sure of and worth a look in review.
Per 40 second window, before and after. The first row is three consecutive
windows, the rest are single windows off the same two runs.
link state changes 186 / 313 / 139 -> 0 / 0 / 0
miibus_statchg calls 659 -> 4
PHY register reads 57010 -> 4536
torn media words 170 -> 0
Torn media words counts arrivals at statchg with subtype 0. The four statchg
calls left are the startup transitions, and it stays at four. Register reads
fall that far because the statchg storm stops and the media queries it provoked
go with it. An instrumented build counted 254 passes that had to wait on
another to finish, and 538 on a later run, so passes are colliding constantly.
Interface counters after the fix: Ierrs 0, Idrop 0, Oerrs 0 across 1.89 GB in
and 1.05 GB out.
Throughput lands at 979 down, best single measurement 983, on a 930/500 plan
and against a 57-73 Mbps stock baseline. Upload is 465, but that needs a TX
queue depth change (1 to 4) I have locally and haven't submitted; it was 250
before that.
I don't think any of this is specific to axge(4). if_axe, if_muge, if_smsc and
if_ure all call mii_attach(), mii_tick() and mii_pollstat() the same way axge
does, over the same mutex-dropping control transfers, and I suspect it is part
of why USB ethernet has the reputation it does on FreeBSD. Someone who knows
the MII layer better than me should say whether the proper fix belongs in
mii(4), or somewhere generic in usb/net, rather than being open-coded per
driver.
Patch attached as series/axge-mii-race.patch, 100 lines, touching
sys/dev/usb/net/if_axge.c and sys/dev/usb/net/if_axgereg.h. It applies on its
own, no dependency on the four earlier fixes. Built and running on OPNsense's
26.7 src tree, which is FreeBSD 15.1-RELEASE-p1. I have not tried it against
stable/15 or main.
--
You are receiving this mail because:
You are the assignee for the bug.
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.