回复：回复：[RFC PATCH] virtio: add VIRTIO_F_DEV ICE_READY and device_ready_status register
"韩为礼(修北)" <[email protected]>
| Newsgroups | dev.linux.lists.virtio-comment |
|---|---|
| Message-ID | <[email protected]> |
--- On 2026-08-25, Parav Pandit wrote: > > 1. patch should be in plain text format Apologies for the formatting. I'll resend as proper plain text in v2, with the diff inline in the email body. > 2. signaling -> signalling Will fix. I'll follow the spec's British spelling convention consistently. > 3. The timeout mechanism should be exposed without a feature bit > because by the time you get to feature bit negotiation, it is > already too late. Agreed. The revised approach: make device_ready_status a standard field in virtio_pci_common_cfg, always present — no feature bit needed. After writing DRIVER_OK, the driver polls device_ready_status: - Devices that don't need extra init time set READY=1 immediately, so the poll completes in one read with negligible overhead. - Devices that do need time (DPU/SmartNIC) simply set READY later, or set INIT_FAILED on failure. Backward compatibility: legacy drivers that don't know about the field simply don't read it — their behavior is unchanged. The field occupies previously reserved space in common_cfg, so existing device implementations are not affected. One question: should the driver always poll unconditionally, or should there be a lightweight discovery mechanism (e.g., the field reads 0xFF when not implemented)? Unconditional polling adds one register read to every device's init path, which I think is acceptable, but I'd like your preference. > 4. device_ready_status should be sufficient to signal all queues > are ready. Per-queue queue_armed is overly complex. Agreed. I'll remove queue_armed entirely. A single READY bit in device_ready_status covering all queues is much simpler and removes the per-queue register overhead for hardware. > 5. If per-queue bidirectional signaling is truly necessary, use > admin commands for queue lifecycles — eliminates polling and > avoids slow register reads. Agreed. For the primary use case (sensing overall device readiness after DRIVER_OK), device_ready_status is sufficient. I'll note admin-command-based per-queue lifecycle management as a future extension direction, not part of this RFC. > 6. Resolve high-level architecture before normative wording. Agreed. I'll focus on architecture alignment first and defer detailed spec wording until we have consensus on the structure. --- Summary of v2 changes: - Resend in proper plain text format - Fix "signaling" -> "signalling" throughout - Remove VIRTIO_F_DEVICE_READY feature bit; make device_ready_status a standard common_cfg field - Remove per-queue queue_armed register - Note admin-command-based queue lifecycle as future work - Focus on architecture, defer normative wording Does this direction look right? Happy to adjust before sending v2. Thanks for the detailed review. Best regards, Weili Han (修北) ------------------------------------------------------------------ 发件人：Parav Pandit <[email protected]> 发送时间：2026年8月25日(周二) 12:02 收件人："韩为礼(修北)"<[email protected]>; "virtio-comment"<[email protected]> 抄 送：mst<[email protected]> 主 题：RE: 回复：[RFC PATCH] virtio: add VIRTIO_F_DEVICE_READY and device_ready_status register Few small comments. 1. patch should be in plain text format From: 韩为礼(修北) <[email protected]> Sent: 25 August 2026 09:03 AM To: virtio-comment <[email protected]> Cc: mst <[email protected]>; 韩为礼(修北) <[email protected]> Subject: 回复：[RFC PATCH] virtio: add VIRTIO_F_DEVICE_READY and device_ready_status register Hi all, Gentle ping on this RFC — it's been about five weeks since the initial submission and I haven't received any feedback yet. To recap, this patch introduces VIRTIO_F_DEVICE_READY (bit 44) to let devices signal initialization completion after DRIVER_OK, addressing the VQ arming latency problem commonly seen in hardware offload implementations (DPU/SmartNIC). The design follows the NVMe CC.EN / CSTS.RDY handshake pattern and the VIRTIO_F_SUSPEND precedent. Original submission: https://lore.kernel.org/virtio-comment/b76a77c1-a5cb-4411-bb39-4c017319aaf2.xiubei.hwl@ali <https://lore.kernel.org/virtio-comment/b76a77c1-a5cb-4411-bb39-4c017319aaf2.xiubei.hwl@ali > baba-inc.com/ GitHub issue: https://github.com/oasis-tcs/virtio-spec/issues/250 <https://github.com/oasis-tcs/virtio-spec/issues/250 > I'd appreciate any review comments, even if just to flag fundamental concerns about the direction. Happy to rework the design, split the patch. Thanks, Weili Han ------------------------------------------------------------------ 发件人：韩为礼(修北) <mailto:[email protected]> 发送时间：2026年7月21日(周二) 09:35 收件人："virtio-comment"<mailto:[email protected]> 抄 送：mst<mailto:[email protected]>; "韩为礼(修北)"<mailto:[email protected]> 主 题：[RFC PATCH] virtio: add VIRTIO_F_DEVICE_READY and device_ready_status register From fa46387ee54a451eba91d0d52992eac78dfd8b87 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BF=AE=E5=8C=97?= <mailto:[email protected]> Date: Mon, 20 Jul 2026 15:09:55 +0800 Subject: [RFC PATCH] virtio: add VIRTIO_F_DEVICE_READY and device_ready_status register MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Problem: After writing DRIVER_OK to device_status, hardware virtio implementations (e.g. DPU/NIC SmartNIC offload devices) require additional time for internal virtqueue arming and firmware initialization. No protocol mechanism exists for the device to signal readiness to the driver, causing the following failure modes: - Userspace drivers (e.g. DPDK-based) that submit I/O immediately after DRIVER_OK may hit uninitialized virtqueue hardware, resulting in PCIe timeout errors or host hangs. - Drivers must resort to empirical sleep timers scaled to VQ count (e.g. max(500ms, num_vqs * 100ms)), which are both fragile and wasteful. - The existing VIRTIO_F_SUSPEND handshake (bit 43) demonstrates that bidirectional status signaling is already a precedent in the spec, but there is no equivalent for the initial DRIVER_OK -> device-live transition. s/ signaling/signalling at above and mor eplaces below. This commit introduces: 1. VIRTIO_F_DEVICE_READY (feature bit 44): opt-in feature indicating that the device supports explicit ready signaling. 2. A new §2.X "Device Ready Status" section in the core spec defining: - device_ready_status register (device-written, 8-bit): bit 0 = VIRTIO_DEV_RDY_READY (device fully armed, I/O safe) bit 1 = VIRTIO_DEV_RDY_INIT_FAILED (initialization error) - Timeout semantics: driver polls device_ready_status after DRIVER_OK up to device_ready_timeout * 10ms (0 = unspecified, driver uses max(500ms, num_vqs * 100ms) as default). - INIT_FAILED retry: up to 3 attempts with 500ms backoff via full device reset (device_status = 0); INIT_FAILED clears on reset. - I/O gating: driver MUST NOT submit I/O until READY=1; I/O requests received before READY SHOULD be re-queued, not dropped. - Per-queue armed readback via queue_armed field. 3. Step 9 "Wait DEVICE-READY" inserted into the device initialization sequence (§3.1) immediately after the existing step 8 (Set DRIVER_OK). 4. PCI transport extensions (transport-pci.tex): Three new fields appended to virtio_pci_common_cfg (present only if VIRTIO_F_DEVICE_READY is negotiated): u8 device_ready_status /* read-only for driver */ u8 reserved_drs /* reserved, must be zero */ le16 device_ready_timeout /* read-only for driver, units of 10ms */ This timeout is more useful at the device reset time instead of queue configuration time. Need to think of a way to expose this without feature bit, because by the time you get to feature bit negotiation, it is already too late. le16 queue_armed /* read-only for driver */ This fields seems overkill and trying to encode register based signalling at per VQ level is burdensome for hw anyway. When queue reset was introduced, admin command infrastructure was not present. And CVQ and rest was tied up with the DRIVER_OK signalling. So I believe device_ready_status should be sufficient to signal all queues are ready. And if you find the need of complex per VQ bidirectional signalling, it is time to move to VQ create/destroy life cycle using admin commands. All the infrastructure is in place of capabilities, resources and commands, which even eliminates polling by driver (cpu wise and avoids slow register reads). Before diving into the each normative line and spec line below, would like to complete above discussion with you. With corresponding device and driver normative requirements. Design rationale: - Analogous to NVMe CC.EN / CSTS.RDY handshake (NVMe 1.4 §3.5.4). - Backward compatible: devices not supporting the feature simply do not negotiate VIRTIO_F_DEVICE_READY; drivers fall back to timer-based workarounds. - VIRTIO_F_RING_RESET's per-queue reset (queue_reset=1) does NOT clear device_ready_status.READY; only full device reset (device_status=0) does. - MSI-X vector allocation (config_msix_vector, queue_msix_vector) must be completed by the driver before DRIVER_OK so the device can include interrupt routing in its arming sequence. Empirical basis: Observed on production hardware virtio implementations where internal VQ arming latency scales with queue count (approximately 100ms/VQ, 500ms minimum). Timer-based workarounds are in use today by multiple userspace virtio drivers on DPU/SmartNIC platforms. RFC: This patch is intended as a starting point for virtio-comment discussion. Implementation experience data from hardware deployments is available upon request. Fixes: https://github.com/oasis-tcs/virtio-spec/issues/0 <https://github.com/oasis-tcs/virtio-spec/issues/0 > (placeholder) Signed-off-by: 修北 <mailto:[email protected]> --- content.tex | 135 +++++++++++++++++++++++++++++++++++++++++++++- transport-pci.tex | 68 +++++++++++++++++++++-- 2 files changed, 199 insertions(+), 4 deletions(-) diff --git a/content.tex b/content.tex index 243ce2d..19ff9aa 100644 --- a/content.tex +++ b/content.tex @@ -85,6 +85,116 @@ \section{\field{Device Status} Field}\label{sec:Basic Facilities of a Virtio Dev that a reset is needed. If DRIVER_OK is set, after it sets DEVICE_NEEDS_RESET, the device MUST send a device configuration change notification to the driver. +\section{Device Ready Status}\label{sec:Basic Facilities of a Virtio Device / Device Ready Status} + +When VIRTIO_F_DEVICE_READY has been negotiated, the device exposes an +8-bit read-only register \field{device\_ready\_status}. For the PCI +transport this register is located in the common configuration structure +(see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device +Layout / Common configuration structure layout}). + +The following bits are defined: + +\begin{description} +\item[VIRTIO_DEV_RDY_READY (1)] Indicates that the device has completed + its internal initialization subsequent to the driver setting DRIVER\_OK + in the \field{device\_status} field, and is prepared to process + available buffers and produce used buffers on all enabled virtqueues. + This bit is written by the device and read by the driver. + +\item[VIRTIO_DEV_RDY_INIT_FAILED (2)] Indicates that the device has + failed to complete its internal initialization after DRIVER\_OK was + set. The driver SHOULD reset the device before retrying. + This bit is written by the device and read by the driver. +\end{description} + +Bits 2-7 are reserved for future extensions and MUST be presented as 0 +by the device until defined. + +The \field{device\_ready\_status} register is reinitialized to 0 when +the device is reset, i.e., when the driver sets \field{device\_status} +to 0. A per-queue reset (using \field{queue\_reset} under +VIRTIO\_F\_RING\_RESET) does \emph{not} affect +\field{device\_ready\_status}; only \field{queue\_armed} for the +affected queue is cleared. + +\begin{note} +This register is the device-side counterpart to DRIVER\_OK: +DRIVER\_OK signals that the driver has finished its setup; +VIRTIO\_DEV\_RDY\_READY signals that the device has finished \emph{its} +setup. This is analogous to the NVMe \texttt{CC.EN}/\texttt{CSTS.RDY} +handshake. + +The v1.4 SUSPEND mechanism (see +\ref{sec:Basic Facilities of a Virtio Device / Device Status Field}) +is a precedent for this bidirectional model: the driver writes +SUSPEND=1, and the device writes DRIVER\_OK=0 when suspension is +complete. This proposal extends the same pattern to the initialization +direction. +\end{note} + +\drivernormative{\subsection}{Device Ready Status}{Basic Facilities of a Virtio Device / Device Ready Status} + +If VIRTIO_F_DEVICE_READY has been negotiated, after setting DRIVER\_OK +the driver MUST poll the \field{device\_ready\_status} register until +VIRTIO\_DEV\_RDY\_READY or VIRTIO\_DEV\_RDY\_INIT\_FAILED is observed, +or a timeout elapses. + +The driver SHOULD derive the polling timeout as follows: +\begin{itemize} +\item If \field{device\_ready\_timeout} is non-zero, use + \field{device\_ready\_timeout} $\times$ 10\,ms as the timeout. +\item Otherwise, use $\max(500\,\text{ms},\ N_{vq} \times 100\,\text{ms})$, + where $N_{vq}$ is the number of enabled virtqueues, as a + recommended default; a hard upper bound of 30\,s SHOULD be applied. +\end{itemize} + +If VIRTIO\_DEV\_RDY\_INIT\_FAILED is observed, or the timeout elapses, +the driver MUST reset the device before retrying. The driver SHOULD +retry at most 3 times, with a backoff of at least 500\,ms between +retries, before declaring the device permanently unavailable. + +If VIRTIO_F_DEVICE_READY has \emph{not} been negotiated, the driver MAY +proceed immediately after setting DRIVER\_OK, and MAY additionally +perform an implementation-specific readiness probe (e.g.\ issuing a +no-op I/O or control command and awaiting its completion). + +\devicenormative{\subsection}{Device Ready Status}{Basic Facilities of a Virtio Device / Device Ready Status} + +If VIRTIO_F_DEVICE_READY has been negotiated, the device MUST, within a +finite time after the driver sets DRIVER\_OK in \field{device\_status}, +set exactly one of VIRTIO\_DEV\_RDY\_READY or +VIRTIO\_DEV\_RDY\_INIT\_FAILED in \field{device\_ready\_status}. + +Prior to setting VIRTIO\_DEV\_RDY\_READY, the device MAY defer or ignore +notifications but MUST NOT consume available buffers nor produce used +buffers. + +Upon setting VIRTIO\_DEV\_RDY\_READY, the device guarantees that: +\begin{enumerate} +\item Every virtqueue for which \field{queue\_enable} is 1 can process + subsequent notifications and produce used buffers within normal + operation timing. +\item Device-specific configuration fields are stable and reflect the + operational state of the device. +\item If MSI-X has been negotiated, interrupt vectors are routed. + \begin{note} + MSI-X vector allocation is performed by the driver before writing + DRIVER\_OK; the device can therefore guarantee routing at the point + it sets VIRTIO\_DEV\_RDY\_READY. + \end{note} +\end{enumerate} + +The device SHOULD expose a \field{device\_ready\_timeout} value (units +of 10\,ms). If the value is 0, the device does not commit to a bound. + +If the device cannot complete initialization, it MUST set +VIRTIO\_DEV\_RDY\_INIT\_FAILED and SHOULD additionally set +DEVICE\_NEEDS\_RESET in \field{device\_status}. + +VIRTIO\_DEV\_RDY\_READY is monotonic: once set it MUST NOT be cleared +until the device is reset. + \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature Bits} Each virtio device offers all the features it understands. During @@ -105,7 +215,9 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \item[24 to 40, and 43] Feature bits reserved for extensions to the queue and feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits} -\item[44 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[44] Feature bit allocated to VIRTIO_F_DEVICE_READY, see \ref{sec:Reserved Feature Bits}. + +\item[45 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note} @@ -545,6 +657,14 @@ \section{Device Initialization}\label{sec:General Initialization And Device Oper \item\label{itm:General Initialization And Device Operation / Device Initialization / Set DRIVER-OK} Set the DRIVER_OK status bit. At this point the device is ``live''. + +\item\label{itm:General Initialization And Device Operation / Device Initialization / Wait DEVICE-READY} If VIRTIO_F_DEVICE_READY has been negotiated, poll + \field{device\_ready\_status} until VIRTIO\_DEV\_RDY\_READY or + VIRTIO\_DEV\_RDY\_INIT\_FAILED is observed, or until the timeout + indicated by \field{device\_ready\_timeout} elapses (see + \ref{sec:Basic Facilities of a Virtio Device / Device Ready Status}). + If VIRTIO\_DEV\_RDY\_INIT\_FAILED is set or the timeout elapses, the + driver MUST reset the device before retrying. \end{enumerate} If any of these steps go irrecoverably wrong, the driver SHOULD @@ -555,6 +675,10 @@ \section{Device Initialization}\label{sec:General Initialization And Device Oper The driver MUST NOT send any buffer available notifications to the device before setting DRIVER_OK. +If VIRTIO_F_DEVICE_READY has been negotiated, the driver MUST NOT +submit I/O requests to any virtqueue before +\field{device\_ready\_status} VIRTIO\_DEV\_RDY\_READY is observed. + \subsection{Legacy Interface: Device Initialization}\label{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization} Legacy devices did not support the FEATURES_OK status bit, and thus did not have a graceful way for the device to indicate unsupported feature @@ -946,6 +1070,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} suspend the device by set the SUSPEND bit to 1. See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}. + \item[VIRTIO_F_DEVICE_READY(44)] This feature indicates that the device + supports the \field{device\_ready\_status} register and the + \field{device\_ready\_timeout} field in the common configuration structure. + When this feature is negotiated, after the driver sets DRIVER\_OK, the + device will write VIRTIO\_DEV\_RDY\_READY or VIRTIO\_DEV\_RDY\_INIT\_FAILED + to \field{device\_ready\_status} to signal the completion or failure of + its internal initialization. + See \ref{sec:Basic Facilities of a Virtio Device / Device Ready Status}. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} diff --git a/transport-pci.tex b/transport-pci.tex index 95b08b8..2209321 100644 --- a/transport-pci.tex +++ b/transport-pci.tex @@ -324,7 +324,15 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport /* About the administration virtqueue. */ le16 admin_queue_index; /* read-only for driver */ - le16 admin_queue_num; /* read-only for driver */ + le16 admin_queue_num; /* read-only for driver */ + + /* Device readiness (present only if VIRTIO_F_DEVICE_READY negotiated). */ + u8 device_ready_status; /* read-only for driver */ + u8 reserved_drs; /* reserved, must be zero */ + le16 device_ready_timeout; /* read-only for driver */ + + /* Per-queue armed readback (present only if VIRTIO_F_DEVICE_READY negotiated). */ + le16 queue_armed; /* read-only for driver */ }; \end{lstlisting} @@ -428,6 +436,32 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport The value 0 indicates no supported administration virtqueues. This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated. + +\item[\field{device\_ready\_status}] + Device-side readiness register. + This field exists only if VIRTIO\_F\_DEVICE\_READY has been + negotiated. It is read-only for the driver; the device writes it. + See \ref{sec:Basic Facilities of a Virtio Device / Device Ready Status} + for bit definitions (VIRTIO\_DEV\_RDY\_READY and + VIRTIO\_DEV\_RDY\_INIT\_FAILED). + +\item[\field{device\_ready\_timeout}] + Maximum time, in units of 10\,ms, the driver should expect to + wait between DRIVER\_OK being set and + \field{device\_ready\_status} VIRTIO\_DEV\_RDY\_READY or + VIRTIO\_DEV\_RDY\_INIT\_FAILED being set. A value of 0 + indicates that the device does not commit to a bound. + This field exists only if VIRTIO\_F\_DEVICE\_READY has been + negotiated. It is read-only for the driver. + +\item[\field{queue\_armed}] + Per-queue armed readback register, indexed by \field{queue\_select}. + This field exists only if VIRTIO\_F\_DEVICE\_READY has been + negotiated. It is read-only for the driver; the device writes it. + Bit~0: set by the device when the selected virtqueue has been armed + and is ready to process notifications. Bits 1-15 are reserved. + The field is cleared to 0 when the queue is disabled or reset. + \end{description} \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout} @@ -495,11 +529,39 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport to ensure that indices of valid admin queues fit into a 16 bit range beyond all other virtqueues. +If VIRTIO_F_DEVICE_READY has been negotiated, the device MUST present +\field{device\_ready\_status} = 0 on reset and after reset until +internal initialization completes. The device MUST then set exactly +one of VIRTIO\_DEV\_RDY\_READY (bit~0) or VIRTIO\_DEV\_RDY\_INIT\_FAILED +(bit~1) within a finite time after DRIVER\_OK is written. +All other bits of \field{device\_ready\_status} MUST be presented as 0. + +If VIRTIO_F_DEVICE_READY has been negotiated, the device MUST set +bit~0 of \field{queue\_armed} after the selected virtqueue has been +armed subsequent to \field{queue\_enable} being written to~1. +The device MUST present \field{queue\_armed} = 0 on reset, after +\field{queue\_enable} is written to~0, and after a per-queue reset +completes under VIRTIO\_F\_RING\_RESET. + \drivernormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout} The driver MUST NOT write to \field{device_feature}, \field{num_queues}, -\field{config_generation}, \field{queue_notify_off} or -\field{queue_notif_config_data}. +\field{config_generation}, \field{queue_notify_off}, +\field{queue_notif_config_data}, \field{device\_ready\_status}, +\field{device\_ready\_timeout} or \field{queue\_armed}. + +If VIRTIO_F_DEVICE_READY has been negotiated, after writing DRIVER\_OK +the driver MUST poll \field{device\_ready\_status} until +VIRTIO\_DEV\_RDY\_READY or VIRTIO\_DEV\_RDY\_INIT\_FAILED is set, or +until the polling timeout derived from \field{device\_ready\_timeout} +elapses. The driver MUST NOT submit I/O to any virtqueue before +observing VIRTIO\_DEV\_RDY\_READY. + +If VIRTIO_F_DEVICE_READY has been negotiated, after writing +\field{queue\_enable} = 1 the driver SHOULD poll bit~0 of +\field{queue\_armed} until it reads 1 or a per-queue timeout elapses. +The driver MUST NOT rely on \field{queue\_armed} if +VIRTIO\_F\_DEVICE\_READY has not been negotiated. If VIRTIO_F_RING_PACKED has been negotiated, the driver MUST NOT write the value 0 to \field{queue_size}. -- 2.47.0