nvme-pci: ADATA LEGEND 860 [1cc1:5772] recurring controller hang, abort ineffective, reset required; one reset failure with CSTS=0x1; 12 months of data, no host mitigation effective

Subject: nvme-pci: ADATA LEGEND 860 [1cc1:5772] recurring controller hang=
, abort ineffective, reset required; one reset failure with CSTS=3D0x1; 1=
2 months of data, no host mitigation effective

Hi,

Reporting a recurring controller hang on ADATA LEGEND 860 NVMe SSDs
(DRAM-less/HMB), with 12 months of event data. No host-side mitigation
tested had any effect on the event rate. Reporting for the archive and
in case a quirk entry or further debugging is useful.

Hardware/software:
- 2x ADATA LEGEND 860 2TB, firmware VF115C6C (both drives identical)
- PCI ID 1cc1:5772 -- note the PCI ID string reads "LEGEND 850 LITE
  NVMe SSD (DRAM-less)", so this platform appears shared across
  retail SKUs
- frmw 0x2 (single firmware slot); vendor offers no firmware update
- Kernel 7.0.12+deb13-amd64 (Debian trixie); same behavior on earlier
  kernels over the past year
- The two drives are whole-disk members of a single md raid1 (md127,
  ext4, hosting KVM guest images); the host OS runs from a separate
  SATA md mirror, so each hang stalls guest storage but not the host
  root. Light, bursty I/O workload.

Symptoms:
Roughly 6 times/month across the pair (similar rate on each drive), an
I/O command times out and only a controller reset recovers:

  nvme nvme1: I/O tag 485 (51e5) opcode 0x0 (I/O Cmd) QID 10 timeout, abo=
rting req_op:FLUSH(2) size:0
  nvme nvme1: Abort status: 0x0
  nvme nvme1: I/O tag 485 (51e5) opcode 0x0 (I/O Cmd) QID 10 timeout, res=
et controller
  nvme nvme1: 12/0/4 default/read/poll queues

~70 events over 12 months. Victim opcode is usually FLUSH; READ and
WRITE also occur. Key observations:

- The stuck command never completes regardless of wait time. io_timeout
  values of 30s, 90s, 120s and ~240s were used over the year; no stuck
  command ever completed, and reset was always required. Abort always
  completes with status 0x0 but never rescues the command.
- Events occur only during low-I/O periods. Multiple full md resyncs
  and sustained sequential transfers completed over the year with zero
  events during them.
- The drives log nothing: after ~70 controller resets, both report
  "Error Information Log Entries: 0", Critical Warning 0x00, clean
  SMART, 0% used.
- No host-side change over 12 months affected the event rate:
  * nvme_core.default_ps_max_latency_us=3D0 (APST disabled; verified via
    get-feature: APSTE=3D0, all table entries zero), pcie_aspm=3Doff,
    pcie_port_pm=3Doff
  * nr_requests 256 -> 8 -> 16
  * ext4 commit interval 5s -> 180s
  * io_timeout in the 30s - 240s range
  * a 5s direct-read keepalive to both controllers (one event occurred
    on day 3 with it running)
  * a 60s unconditional "nvme flush" keepalive, tested for a full
    month: event rate unchanged (~6/month-pair, matching the 11-month
    baseline). One victim was the keepalive flush itself, submitted
    <60s after the previous successful flush.
  The only visible change: after APST/ASPM were disabled, victims
  became almost exclusively FLUSH, where earlier events included more
  READ/WRITE victims.

Worst-case event (1 of ~70): reset failure and device removal.
On one occasion the controller failed the reset handshake entirely:

  nvme nvme0: I/O tag 961 (53c1) opcode 0x0 (I/O Cmd) QID 13 timeout, abo=
rting req_op:FLUSH(2) size:0
  nvme nvme0: I/O tag 961 (53c1) opcode 0x0 (I/O Cmd) QID 13 timeout, res=
et controller
  nvme nvme0: Device not ready; aborting reset, CSTS=3D0x1
  nvme0n1: I/O Cmd(0x2) @ LBA 97441816, 48 blocks, I/O Error (sct 0x3 / s=
c 0x71)
  nvme nvme0: Abort status: 0x371
  nvme nvme0: Device not ready; aborting reset, CSTS=3D0x1
  nvme nvme0: Device not ready; aborting reset, CSTS=3D0x1
  nvme nvme0: Disabling device after reset failure: -19
  md/raid1:md127: Disk failure on nvme0n1, disabling device.

CSTS stayed 0x1 through three reset attempts; the device was removed
(-19) and required a host power cycle to recover, after which the md
member re-added cleanly via the write-intent bitmap. Note the abort
returned 0x371 in this event versus 0x0 in all recoverable events.

Interpretation (host-side view only):
Consistent with an internal controller state entered on the firmware's
own schedule during low activity -- below the NVMe power-state layer,
since APST is disabled and the host never enables non-operational
states -- with a wake/recovery path that hangs. I obviously can't
confirm internals from the host.

Happy to provide full logs, id-ctrl output, or to test patches.

Thanks,
Chris Quin