Re: [PATCH 7/6 RFC] nvme: test per-command retry delay

On Aug 23, 2026 / 11:49, Sagi Grimberg wrote:
> Add tests to exercise host command retry delays handling.
> 
> 070: check that basic command RETRY disposition works and respect ctrl
>      crd
> 071: check that basic command FAILOVER disposition works and respects
>      ctrl crd
> 072: check that different commands completed with different crd levels
>      are retried independently, each respecting its paired completion
>      crd level
> 073: check that different commands completed with different crd levels
>      are failed-over independently, each respecting its paired completion
>      crd level
> 
> These tests rely on nvmet support for subsystem crdt attributes
> (_require_nvmet_crdt) and nvme host crd error injection support.
> 
> In addition we add some common nvme helpers to set nvmet attributes,
> inject errors, and leverage nvme diags to count retries/failovers.
> 
> Signed-off-by: Sagi Grimberg <[email protected]>

Thank you for the patch. I ran the added four test cases using the kernel with
the kernel patches, and observed the all four test cases passed. Good.

I walked through the new test cases. Overall, they look good. One point to
improve is the global variable used to return a value. I will comment it in-
line.

I found the new test cases measure some numbers like retry count, failover
count, or elapsed times. Those numbers are used as pass/fail criteria. The
numbers are logged in the FULL file, but it might be useful to print the
numbers in the test run console like this:

nvme/070 (tr=loop) (test NVMe CRD per-request retry under fio) [passed]
    retries after   41       ...  42
    retries before  0        ...  0
    runtime         24.808s  ...  24.844s
nvme/071 (tr=loop) (test NVMe CRD multipath failover under fio) [passed]
    failovers after   100      ...  100
    failovers before  0        ...  0
    runtime           13.764s  ...  13.789s
nvme/072 (tr=loop) (test NVMe CRD per-request retry timer independence) [passed]
    CRD1 write  1090ms   ...  1079ms
    CRD2 write  10563ms  ...  10416ms
    runtime     12.363s  ...  12.166s
nvme/073 (tr=loop) (test NVMe CRD multipath failover timer independence) [passed]
    CRD1 failover  1068ms   ...  1065ms
    CRD2 failover  10091ms  ...  10349ms
    runtime        13.153s  ...  13.432s

FYI, I attached the script changes to print the numbers [*].It uses TEST_RUN[*]
feature of blktests.

Also, please find my in-line comments below:

> diff --git a/common/nvme b/common/nvme
> index f3999378db2d..a224dca61840 100644
> --- a/common/nvme
> +++ b/common/nvme

[...]

> +_nvme_now_ms() {
> +	echo $(($(date +%s%N) / 1000000))
> +}

Just comment: this funcion might worth moving to common/rc: tests/thtrol/*
scripts do almost same thing.

[...]

> +# Background timed direct write. Sets nvme_bg_timed_pid (do NOT call from $()).
> +# Writes elapsed ms to <result_file>, or FAIL on I/O error.
> +_nvme_bg_timed_direct_write() {
> +	local dev=$1
> +	local result=$2
> +
> +	rm -f "${result}"
> +	(
> +		local start end
> +		start="$(_nvme_now_ms)"
> +		if dd if=/dev/zero of="${dev}" bs=4k count=1 oflag=direct \
> +			status=none conv=notrunc 2>>"$FULL"; then
> +			end="$(_nvme_now_ms)"
> +			echo $((end - start)) >"${result}"
> +		else
> +			echo FAIL >"${result}"
> +		fi
> +	) &
> +	nvme_bg_timed_pid=$!
> +}

_nvme_bg_timed_direct_write() uses the global variable nvme_bg_timed_pid to
return the pid to the caller. It is not the best to use global variables for
that purpose. Also, shellcheck warns this:

   common/nvme:1773:2: warning: nvme_bg_timed_pid appears unused. Verify use (or export if used externally). [SC2034]

In general, bash functions return value with the "echo back" method. But I
understand this method won't work here, since sub-shell $() is required to pass
the echoed value to the caller. With this, the background task is no longer a
child of the caller, then the wait command for the received pid fails with the
error "wait: pid x is not a child of this shell".

As the solution for such scenarios, bash provides "nameref" feature (local -n),
which is like the pointer of the C language. The hunk below will add the third
argument to return the pid to the caller. I will comment how the caller sides
will change later.

diff --git a/common/nvme b/common/nvme
index f323c6c..273fa66 100644
--- a/common/nvme
+++ b/common/nvme
@@ -1752,14 +1752,15 @@ _nvme_timed_direct_write() {
 	echo $((end - start))
 }
 
-# Background timed direct write. Sets nvme_bg_timed_pid (do NOT call from $()).
-# Writes elapsed ms to <result_file>, or FAIL on I/O error.
+# Background timed direct write. Writes elapsed ms to <result_file>, or FAIL on
+# I/O error. Return the pid of the background process with bash nameref feature.
 _nvme_bg_timed_direct_write() {
 	local dev=$1
 	local result=$2
+	local -n pid=$3
 
 	rm -f "${result}"
-	(
+	{
 		local start end
 		start="$(_nvme_now_ms)"
 		if dd if=/dev/zero of="${dev}" bs=4k count=1 oflag=direct \
@@ -1769,8 +1770,8 @@ _nvme_bg_timed_direct_write() {
 		else
 			echo FAIL >"${result}"
 		fi
-	) &
-	nvme_bg_timed_pid=$!
+	} &
+	pid=$!
 }
 

> diff --git a/tests/nvme/070 b/tests/nvme/070
> new file mode 100755
> index 000000000000..14d7160664a0
> --- /dev/null
> +++ b/tests/nvme/070
> @@ -0,0 +1,95 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-3.0+
> +# Copyright (C) 2026 Sagi Grimberg <[email protected]>
> +#
> +# Test NVMe command retry delay (CRD) with the per-request retry timer.
> +# Requires nvmet attr_crdt* and host fault_inject/crd.
> +
> +. tests/nvme/rc
> +
> +DESCRIPTION="test NVMe CRD per-request retry under fio"
> +QUICK=1
> +
> +# NVME_SC_INTERNAL

Nit: the line above does not look meaningful when I see the line below.

> +NVME_SC_INTERNAL=0x6
> +

[...]

> +test() {
> +	local fio_pid
> +	local ns
> +	local retries_before
> +	local retries_after
> +	local inject_dev
> +
> +	echo "Running ${TEST_NAME}"
> +
> +	_setup_nvmet
> +	_nvmet_target_setup
> +	# CRDT1 = 5 * 100ms = 500ms
> +	_nvmet_set_crdt 5 0 0
> +
> +	_nvme_connect_subsys
> +	ns=$(_find_nvme_ns "${def_subsys_uuid}")
> +
> +	inject_dev=$(_nvme_fault_inject_devs "${ns}" | head -1)
> +	_nvme_set_ns_diag "${inject_dev}" command_retries_count 0 || true
> +	retries_before=$(_nvme_get_ns_diag "${inject_dev}" command_retries_count)
> +
> +	_run_fio_verify_io --filename="/dev/${ns}" \
> +			   --group_reporting --ramp_time=2 \
> +			   --time_based --runtime=20 &> "$FULL" &

Nit: It is a bit safer to use "&>>" instaed of "&>" in case prep helper
functions leave logs in the FULL file.

[...]

> diff --git a/tests/nvme/072 b/tests/nvme/072
> new file mode 100755
> index 000000000000..2acda72cdd44
> --- /dev/null
> +++ b/tests/nvme/072

[...]

> +test() {
> +	local ns
> +	local inject_dev
> +	local crd2_pid crd1_pid
> +	local crd2_result crd1_result
> +	local crd2_elapsed crd1_elapsed
> +
> +	echo "Running ${TEST_NAME}"
> +
> +	_setup_nvmet
> +	_nvmet_target_setup
> +	_nvmet_set_crdt "${CRDT1}" "${CRDT2}" 0
> +
> +	_nvme_connect_subsys
> +	ns=$(_find_nvme_ns "${def_subsys_uuid}")
> +	inject_dev=$(_nvme_fault_inject_devs "${ns}" | head -1)
> +
> +	echo "ns=${ns} inject=${inject_dev}" >>"$FULL"
> +	echo "CRDT CRD1=${CRD1_MS}ms CRD2=${CRD2_MS}ms"
> +
> +	if [[ ! -e /sys/kernel/debug/${inject_dev}/fault_inject/crd ]]; then
> +		echo "FAIL: missing fault_inject/crd on ${inject_dev}"
> +		_nvme_disconnect_subsys
> +		_nvmet_target_cleanup
> +		return 1
> +	fi
> +
> +	crd2_result="${TMPDIR}/crd2_elapsed"
> +	crd1_result="${TMPDIR}/crd1_elapsed"
> +
> +	echo "Arming CRD2 and starting first write"
> +	_nvme_arm_crd_inject "${inject_dev}" 0 "${NVME_SC_INTERNAL}" 2 1 || return 1
> +	echo "inject: status=${NVME_SC_INTERNAL} crd=2 times=1" >>"$FULL"
> +	_nvme_bg_timed_direct_write "/dev/${ns}" "${crd2_result}"
> +	crd2_pid=$nvme_bg_timed_pid

With the nameref, the two lines above are to be modified as follows:

	_nvme_bg_timed_direct_write "/dev/${ns}" "${crd2_result}" crd2_pid

> +	echo "CRD2 write pid=${crd2_pid}" >>"$FULL"
> +
> +	if ! _nvme_wait_inject_consumed "${inject_dev}"; then
> +		echo "FAIL: CRD2 inject was not consumed"
> +		_nvme_disarm_crd_inject "${inject_dev}"
> +		kill "${crd2_pid}" 2>/dev/null || true
> +		wait "${crd2_pid}" 2>/dev/null || true
> +		_nvme_disconnect_subsys
> +		_nvmet_target_cleanup
> +		return 1
> +	fi
> +	echo "CRD2 latched; arming CRD1 while CRD2 retry is pending" >>"$FULL"
> +
> +	echo "Arming CRD1 and starting second write (CRD2 still pending)"
> +	_nvme_arm_crd_inject "${inject_dev}" 0 "${NVME_SC_INTERNAL}" 1 1 || return 1
> +	echo "inject: status=${NVME_SC_INTERNAL} crd=1 times=1" >>"$FULL"
> +	_nvme_bg_timed_direct_write "/dev/${ns}" "${crd1_result}"
> +	crd1_pid=$nvme_bg_timed_pid

Same here:

	_nvme_bg_timed_direct_write "/dev/${ns}" "${crd1_result}" crd1_pid

> +	echo "CRD1 write pid=${crd1_pid}" >>"$FULL"
> +
> +	echo "Waiting for both writes (expect CRD1~${CRD1_MS}ms then CRD2~${CRD2_MS}ms)"
> +	wait "${crd1_pid}" || true
> +	wait "${crd2_pid}" || true
> +	_nvme_disarm_crd_inject "${inject_dev}"
> +	udevadm settle >/dev/null 2>&1 || true
> +
> +	crd1_elapsed="$(cat "${crd1_result}" 2>/dev/null || echo FAIL)"
> +	crd2_elapsed="$(cat "${crd2_result}" 2>/dev/null || echo FAIL)"
> +	echo "CRD1 write elapsed ${crd1_elapsed}ms (expected ~${CRD1_MS}ms)" >>"$FULL"
> +	echo "CRD2 write elapsed ${crd2_elapsed}ms (expected ~${CRD2_MS}ms)" >>"$FULL"
> +
> +	if [[ "${crd1_elapsed}" == "FAIL" || -z "${crd1_elapsed}" ]]; then
> +		echo "FAIL: CRD1 write did not complete"
> +	else
> +		echo "CRD1 write elapsed ${crd1_elapsed}ms" | sed -E 's/(elapsed )[0-9]+/\1NUM/'
> +		_nvme_check_crd_elapsed "CRD1 write" "${crd1_elapsed}" "${CRD1_MS}" "$((CRD2_MS / 2))"
> +	fi
> +	if [[ "${crd2_elapsed}" == "FAIL" || -z "${crd2_elapsed}" ]]; then
> +		echo "FAIL: CRD2 write did not complete"
> +	else
> +		echo "CRD2 write elapsed ${crd2_elapsed}ms" | sed -E 's/(elapsed )[0-9]+/\1NUM/'
> +		_nvme_check_crd_elapsed "CRD2 write" "${crd2_elapsed}" "${CRD2_MS}"
> +	fi
> +
> +	_nvme_disconnect_subsys
> +	_nvmet_target_cleanup
> +	udevadm settle >/dev/null 2>&1 || true
> +
> +	echo "Test complete"
> +}
> diff --git a/tests/nvme/072.out b/tests/nvme/072.out
> new file mode 100644
> index 000000000000..d9f4520c8d23
> --- /dev/null
> +++ b/tests/nvme/072.out
> @@ -0,0 +1,8 @@
> +Running nvme/072
> +CRDT CRD1=1000ms CRD2=10000ms
> +Arming CRD2 and starting first write
> +Arming CRD1 and starting second write (CRD2 still pending)
> +Waiting for both writes (expect CRD1~1000ms then CRD2~10000ms)
> +CRD1 write elapsed NUMms
> +CRD2 write elapsed NUMms
> +Test complete
> diff --git a/tests/nvme/073 b/tests/nvme/073
> new file mode 100755
> index 000000000000..34911392437a
> --- /dev/null
> +++ b/tests/nvme/073

[...]

> +test() {
> +	local ns
> +	local port
> +	local -a ports
> +	local -a path_devs
> +	local path0
> +	local path1
> +	local fo_before fo_after
> +	local crd2_pid crd1_pid
> +	local crd2_result crd1_result
> +	local crd2_elapsed crd1_elapsed
> +	local sync_start
> +
> +	echo "Running ${TEST_NAME}"
> +
> +	_setup_nvmet
> +	_nvmet_target_setup --ports 2
> +	_nvmet_set_crdt "${CRDT1}" "${CRDT2}" 0
> +
> +	_get_nvmet_ports "${def_subsysnqn}" ports
> +	for port in "${ports[@]}"; do
> +		_setup_nvmet_port_ana "${port}" 1 "optimized"
> +		_nvme_connect_subsys --port "${port}" --no-wait-ns
> +	done
> +	sleep 1
> +
> +	ns=$(_find_nvme_ns "${def_subsys_uuid}")
> +	mapfile -t path_devs < <(_nvme_path_ns_devs "${ns}")
> +	if ((${#path_devs[@]} < 2)); then
> +		echo "FAIL: need >=2 path namespaces, found ${#path_devs[@]} (${path_devs[*]})"
> +		_nvme_disconnect_subsys
> +		_nvmet_target_cleanup
> +		return 1
> +	fi
> +
> +	path0=${path_devs[0]}
> +	path1=${path_devs[1]}
> +	echo "ns=${ns} path0=${path0} path1=${path1}" >>"$FULL"
> +	echo "CRDT CRD1=${CRD1_MS}ms CRD2=${CRD2_MS}ms"
> +
> +	if [[ ! -e /sys/kernel/debug/${path0}/fault_inject/crd ]]; then
> +		echo "FAIL: missing fault_inject/crd on ${path0}"
> +		_nvme_disconnect_subsys
> +		_nvmet_target_cleanup
> +		return 1
> +	fi
> +
> +	_nvme_set_ns_diag "${path0}" multipath_failover_count 0 || true
> +	_nvme_set_ns_diag "${path1}" multipath_failover_count 0 || true
> +	fo_before=$(_nvme_get_ns_diag "${path0}" multipath_failover_count)
> +
> +	crd2_result="${TMPDIR}/crd2_elapsed"
> +	crd1_result="${TMPDIR}/crd1_elapsed"
> +
> +	echo "Arming CRD2 on path0 and starting first write"
> +	_nvme_arm_crd_inject "${path0}" 0 "${NVME_SC_INTERNAL_PATH_ERROR}" 2 1 || return 1
> +	echo "path0 inject: path_error crd=2 times=1" >>"$FULL"
> +	_nvme_bg_timed_direct_write "/dev/${ns}" "${crd2_result}"
> +	crd2_pid=$nvme_bg_timed_pid

Same here:

	_nvme_bg_timed_direct_write "/dev/${ns}" "${crd2_result}" crd2_pid

> +	echo "CRD2 write pid=${crd2_pid}" >>"$FULL"
> +
> +	sync_start="$(_nvme_now_ms)"
> +	while (( $(_nvme_get_ns_diag "${path0}" multipath_failover_count) <= fo_before )); do
> +		if (( $(_nvme_now_ms) - sync_start > 2000 )); then
> +			echo "FAIL: CRD2 failover was not scheduled on ${path0}"
> +			_nvme_disarm_crd_inject "${path0}"
> +			kill "${crd2_pid}" 2>/dev/null || true
> +			wait "${crd2_pid}" 2>/dev/null || true
> +			_nvme_disconnect_subsys
> +			_nvmet_target_cleanup
> +			return 1
> +		fi
> +		sleep 0.01
> +	done
> +	echo "CRD2 failover pending; path0 still selectable (NUMA/current)" >>"$FULL"
> +
> +	# Same path again: latch CRD1 while the CRD2 fot is still pending.
> +	echo "Arming CRD1 on path0 and starting second write (CRD2 still pending)"
> +	_nvme_arm_crd_inject "${path0}" 0 "${NVME_SC_INTERNAL_PATH_ERROR}" 1 1 || return 1
> +	echo "path0 inject: path_error crd=1 times=1" >>"$FULL"
> +	_nvme_bg_timed_direct_write "/dev/${ns}" "${crd1_result}"
> +	crd1_pid=$nvme_bg_timed_pid

Same here:

	_nvme_bg_timed_direct_write "/dev/${ns}" "${crd1_result}" crd1_pid

> +	echo "CRD1 write pid=${crd1_pid}" >>"$FULL"
> +
> +	echo "Waiting for both writes (expect CRD1~${CRD1_MS}ms then CRD2~${CRD2_MS}ms)"
> +	wait "${crd1_pid}" || true
> +	wait "${crd2_pid}" || true
> +	_nvme_disarm_crd_inject "${path0}"
> +	udevadm settle >/dev/null 2>&1 || true
> +
> +	crd1_elapsed="$(cat "${crd1_result}" 2>/dev/null || echo FAIL)"
> +	crd2_elapsed="$(cat "${crd2_result}" 2>/dev/null || echo FAIL)"
> +	fo_after=$(_nvme_get_ns_diag "${path0}" multipath_failover_count)
> +	echo "CRD1 failover elapsed ${crd1_elapsed}ms (expected ~${CRD1_MS}ms)" >>"$FULL"
> +	echo "CRD2 failover elapsed ${crd2_elapsed}ms (expected ~${CRD2_MS}ms)" >>"$FULL"
> +	echo "path0 failover count ${fo_before} -> ${fo_after}" >>"$FULL"

The lines above causes a shellcheck warn:

  tests/nvme/073:131:2: note: Consider using { cmd1; cmd2; } >> file instead of individual redirects. [SC2129]

I suggest to modify the lines as follows:

	{
		echo "CRD1 failover elapsed ${crd1_elapsed}ms (expected ~${CRD1_MS}ms)"
		echo "CRD2 failover elapsed ${crd2_elapsed}ms (expected ~${CRD2_MS}ms)"
		echo "path0 failover count ${fo_before} -> ${fo_after}"
	} >>"$FULL"

> +
> +	if (( fo_after < fo_before + 2 )); then
> +		echo "FAIL: expected two failovers on path0, got ${fo_before} -> ${fo_after}"
> +	fi
> +
> +	if [[ "${crd1_elapsed}" == "FAIL" || -z "${crd1_elapsed}" ]]; then
> +		echo "FAIL: CRD1 failover write did not complete"
> +	else
> +		echo "CRD1 failover elapsed ${crd1_elapsed}ms" | sed -E 's/(elapsed )[0-9]+/\1NUM/'
> +		_nvme_check_crd_elapsed "CRD1 failover" "${crd1_elapsed}" "${CRD1_MS}" "$((CRD2_MS / 2))"
> +	fi
> +	if [[ "${crd2_elapsed}" == "FAIL" || -z "${crd2_elapsed}" ]]; then
> +		echo "FAIL: CRD2 failover write did not complete"
> +	else
> +		echo "CRD2 failover elapsed ${crd2_elapsed}ms" | sed -E 's/(elapsed )[0-9]+/\1NUM/'
> +		_nvme_check_crd_elapsed "CRD2 failover" "${crd2_elapsed}" "${CRD2_MS}"
> +	fi
> +
> +	_nvme_disconnect_subsys
> +	_nvmet_target_cleanup
> +	udevadm settle >/dev/null 2>&1 || true
> +
> +	echo "Test complete"
> +}

[...]

> diff --git a/tests/nvme/rc b/tests/nvme/rc
> index 31a0fc59ff4b..f286d9a95cce 100644
> --- a/tests/nvme/rc
> +++ b/tests/nvme/rc
> @@ -505,17 +505,39 @@ _nvme_err_inject_cleanup()
>  
>  _nvme_enable_err_inject()
>  {
> +	# Set status/dont_retry[/crd] before arming probability/times so concurrent
> +	# I/O cannot observe the debugfs defaults (INVALID_OPCODE + DNR).
>          _set_attr "$2" /sys/kernel/debug/"$1"/fault_inject/verbose
> -        _set_attr "$3" /sys/kernel/debug/"$1"/fault_inject/probability
>          _set_attr "$4" /sys/kernel/debug/"$1"/fault_inject/dont_retry
>          _set_attr "$5" /sys/kernel/debug/"$1"/fault_inject/status
> +	if [[ -n "${7:-}" && -e /sys/kernel/debug/"$1"/fault_inject/crd ]]; then
> +		_set_attr "$7" /sys/kernel/debug/"$1"/fault_inject/crd
> +	fi
>          _set_attr "$6" /sys/kernel/debug/"$1"/fault_inject/times
> +        _set_attr "$3" /sys/kernel/debug/"$1"/fault_inject/probability
> +}

Just comment: this function uses spaces for indent regardless of the patch. The
hunk above also uses spaces for indent, but I think it's fine to keep the
consistency. It is ideal to replace the spaces for indent in the function later.
I found three other injection related function in tests/nvme/rc uses spaces for
indent.



[*] Changes to report pass/fail criteria numbers in the test run console

diff --git a/tests/nvme/070 b/tests/nvme/070
index 14d7160..8fbbd77 100755
--- a/tests/nvme/070
+++ b/tests/nvme/070
@@ -84,6 +84,8 @@ test() {
 	wait "${fio_pid}" || echo "FAIL: fio exited with errors (see $FULL)"
 
 	retries_after=$(_nvme_get_ns_diag "${inject_dev}" command_retries_count)
+	TEST_RUN["retries before"]=$retries_before
+	TEST_RUN["retries after"]=$retries_after
 	if (( retries_after <= retries_before )); then
 		echo "command_retries_count did not increase (${retries_before} -> ${retries_after})"
 	fi
diff --git a/tests/nvme/071 b/tests/nvme/071
index 897a883..644921a 100755
--- a/tests/nvme/071
+++ b/tests/nvme/071
@@ -173,6 +173,8 @@ test() {
 
 	inject_after=$(_nvme_get_ns_diag "${inject_dev}" multipath_failover_count)
 	echo "failovers ${inject_dev}: ${inject_before} -> ${inject_after}" >> "$FULL"
+	TEST_RUN["failovers before"]=$inject_before
+	TEST_RUN["failovers after"]=$inject_after
 	if (( inject_after <= inject_before )); then
 		echo "FAIL: multipath_failover_count on ${inject_dev} did not increase (${inject_before} -> ${inject_after})"
 		dump_fault_inject "${inject_dev}"
diff --git a/tests/nvme/072 b/tests/nvme/072
index 2acda72..4515285 100755
--- a/tests/nvme/072
+++ b/tests/nvme/072
@@ -101,12 +99,14 @@ test() {
 		echo "FAIL: CRD1 write did not complete"
 	else
 		echo "CRD1 write elapsed ${crd1_elapsed}ms" | sed -E 's/(elapsed )[0-9]+/\1NUM/'
+		TEST_RUN["CRD1 write"]="${crd1_elapsed}"ms
 		_nvme_check_crd_elapsed "CRD1 write" "${crd1_elapsed}" "${CRD1_MS}" "$((CRD2_MS / 2))"
 	fi
 	if [[ "${crd2_elapsed}" == "FAIL" || -z "${crd2_elapsed}" ]]; then
 		echo "FAIL: CRD2 write did not complete"
 	else
 		echo "CRD2 write elapsed ${crd2_elapsed}ms" | sed -E 's/(elapsed )[0-9]+/\1NUM/'
+		TEST_RUN["CRD2 write"]="${crd2_elapsed}"ms
 		_nvme_check_crd_elapsed "CRD2 write" "${crd2_elapsed}" "${CRD2_MS}"
 	fi
 
diff --git a/tests/nvme/073 b/tests/nvme/073
index 3491139..27b682d 100755
--- a/tests/nvme/073
+++ b/tests/nvme/073
@@ -140,12 +140,14 @@ test() {
 		echo "FAIL: CRD1 failover write did not complete"
 	else
 		echo "CRD1 failover elapsed ${crd1_elapsed}ms" | sed -E 's/(elapsed )[0-9]+/\1NUM/'
+		TEST_RUN["CRD1 failover"]="${crd1_elapsed}"ms
 		_nvme_check_crd_elapsed "CRD1 failover" "${crd1_elapsed}" "${CRD1_MS}" "$((CRD2_MS / 2))"
 	fi
 	if [[ "${crd2_elapsed}" == "FAIL" || -z "${crd2_elapsed}" ]]; then
 		echo "FAIL: CRD2 failover write did not complete"
 	else
 		echo "CRD2 failover elapsed ${crd2_elapsed}ms" | sed -E 's/(elapsed )[0-9]+/\1NUM/'
+		TEST_RUN["CRD2 failover"]="${crd2_elapsed}"ms
 		_nvme_check_crd_elapsed "CRD2 failover" "${crd2_elapsed}" "${CRD2_MS}"
 	fi