[Bug 297337] nvme: timeouts and resets with Intel DC P3700

[email protected]
Newsgroups gmane.os.freebsd.bugs
Message-ID <[email protected]/bugzilla/> 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297337

            Bug ID: 297337
           Summary: nvme: timeouts and resets with Intel DC P3700
           Product: Base System
           Version: 16.0-CURRENT
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: [email protected]
          Reporter: [email protected]

I've put two slightly outdated Intel DC P3700 NVMe cards into my FreeBSD test
machine the other day, created a mirrored zpool on them, put a source tree
there and tried to do a buildworld. After a few seconds, I saw a gazillion of
these messages:

nvme0: Resetting controller due to a timeout and fatal error status.
nvme0: event="start"
nvme0: Waiting for reset to complete
nvme0: Waiting for reset to complete
...

nvme0: controller ready did not become 0 within 20500 ms
nvme0: event="timed_out"
nvme0: failing queued i/o
nvme0: Failed controller, stopping watchdog timeout.
nvme0: Failed controller, stopping watchdog timeout.
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:112486 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:112500 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:112501 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
(nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=1b766
0 0 0 0 0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:112502 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: Failed controller, stopping watchdog timeout.
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:112503 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:112504 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115392 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115393 len:2
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115395 len:2
(nda0:nvme0:0:0:1): CAM status: NVME Status Error
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
(nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
(nda0:nvme0:0:0:1): Error 5, Retries exhausted
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115397 len:2
(nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=1b774
0 0 0 0 0
(nda0:nvme0:0:0:1): CAM status: NVME Status Error
(nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
(nda0:nvme0:0:0:1): Error 5, Retries exhausted
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115399 len:2
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115401 len:2
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
(nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=1b775
0 0 0 0 0
(nda0:nvme0:0:0:1): CAM status: NVME Status Error
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115403 len:2
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115412 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115413 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115414 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115426 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115427 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
(nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
(nda0:nvme0:0:0:1): Error 5, Retries exhausted
(nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=1b776
0 0 0 0 0
(nda0:nvme0:0:0:1): CAM status: NVME Status Error
(nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
(nda0:nvme0:0:0:1): Error 5, Retries exhausted
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115429 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115430 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115431 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115432 len:1
(nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=1b777
0 0 0 0 0
(nda0:nvme0:0:0:1): CAM status: NVME Status Error
(nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
(nda0:nvme0:0:0:1): Error 5, Retries exhausted
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115433 len:1
(nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0 cdw=1b778
0 0 0 0 0
(nda0:nvme0:0:0:1): CAM status: NVME Status Error
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115434 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: Failed controller, stopping watchdog timeout.
nvme0: Failed controller, stopping watchdog timeout.
(nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
(nda0:nvme0:0:0:1): Error 5, Retries exhausted
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115435 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115436 len:1
(nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0
cdw=110500 0 0 0 0 0
(nda0:nvme0:0:0:1): CAM status: NVME Status Error
(nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
(nda0:nvme0:0:0:1): Error 5, Retries exhausted
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115437 len:1
(nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0
cdw=110501 0 1 0 0 0
(nda0:nvme0:0:0:1): CAM status: NVME Status Error
(nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
(nda0:nvme0:0:0:1): Error 5, Retries exhausted
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115438 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
(nda0:nvme0:0:0:1): READ (02). NCB: opc=2 fuse=0 nsid=1 prp1=0 prp2=0
cdw=110503 0 1 0 0 0
(nda0:nvme0:0:0:1): CAM status: NVME Status Error
(nda0:nvme0:0:0:1): NVMe status: ABORTED_BY_REQUEST (00/07) DNR
(nda0:nvme0:0:0:1): Error 5, Retries exhausted
nvme0: READ (02) sqid:4 cid:0 nsid:1 lba:1115439 len:1
nvme0: ABORTED_BY_REQUEST (00/07) crd:0 m:0 dnr:1 p:0 sqid:4 cid:0 cdw0:0
nda0 at nvme0 bus 0 scbus5 target 0 lun 1
nda0: <INTEL SSDPEDMD800G4 8DV10049 CVFT41740038800CGN> s/n CVFT41740038800CGN
detached
nvme0: READ (02) sqid:4 cid:0 nsid:1

Then the system panicked (see bug #297336).

While the P3700 drives are a bit old (NVMe 1.0), they are still working fine
and have 97% health remaining. They work flawlessly on illumos.

-- 
You are receiving this mail because:
You are the assignee for the bug.
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.