Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer

Hi Peter,

> On 12 Feb 2026, at 12:16, Peter Hilber <[email protected]> wrote:
>
> On Tue, Feb 10, 2026 at 09:39:09AM +0000, Bertrand Marquis wrote:
>> Hi Peter,
>>
>
> Hi Bertrand,
>
> thank you for the fast response! I added replies where I think they are
> relevant. I also added a few new comments.
>
> Best regards,
>
> Peter
>
>>> On 6 Feb 2026, at 17:28, Peter Hilber <[email protected]> wrote:
>>>
>>> On Mon, Jan 26, 2026 at 11:32:28AM -0500, Bill Mills wrote:
>
> [...]
>
>>>> +\busnormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Bus}
>>>> +\begin{itemize}
>>>> +  \item A bus instance that advertises transport revision $N$ MUST satisfy every
>>>> +        bus requirement defined for revisions $1$ through $N$.
>>>
>>> Are transport feature bits supposed to be negotiated? I think this could
>>> make sense so that behavior can change without affecting
>>> interoperability between implementations of different revisions of the
>>> Virtio standard.
>>
>> Yes transport feature bits are supposed to be negotiated.
>> It is expected that the bus will determine what is the common set
>> of features supported and will only declare those to the transport.
>>
>> A simple bus implementation could just get from the device side the
>> mask of bits supported and mask it with the list of features bits supported
>> on the driver side and give back the common denominator to the transport
>> so that only features supported by both sides are advertised and used.
>>
>> We will make that clearer in v2 if you agree with that principle.
>>
>
> Ack.
>
>>>
>>>> +  \item If a bus instance cannot forward a message because it requires an
>>>> +        unsupported revision or transport feature, it MUST surface a transport
>>>> +        error or drop the message without forwarding it.
>>>> +\end{itemize}
>>>> +
>>>> +\subsubsection{Device Numbers and Enumeration}
>>>> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts /
>>>> +Device Numbers}
>>>> +
>>>> +Each virtio-msg bus instance contains one or more \emph{devices}, identified
>
> According to a statement above, an instance can also have zero devices.

Right I need to fix that to be coherent.
Point of allowing 0 devices is to allow the case where devices are hotplugged.

>
>>>> +by a 16-bit \textbf{device number}. Buses discover these device numbers through
>>>> +mechanisms such as:
>>>> +\begin{itemize}
>>>> +  \item \textbf{Message-Based Enumeration}: Using \busref{GET_DEVICES} to query
>>>> +        which numbers exist (optional).
>>>> +  \item \textbf{Platform Data}: A device tree, ACPI tables, or hypervisor calls
>>>> +        might inform the bus of available device numbers and their properties.
>>>> +\end{itemize}
>>>> +
>>>> +Once a bus confirms that a device number is valid—regardless of the discovery
>>>> +method—it normally issues \msgref{GET_DEVICE_INFO} to retrieve the device and
>>>> +vendor IDs before registering the device with the host OS so the usual Virtio
>>>> +driver binding process can begin.
>>>> +
>>>> +\busnormative{\paragraph}{Device Number Assignment}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Device Numbers / Assignment}
>>>> +\begin{itemize}
>>>> +  \item A bus implementation MUST assign a unique device number to every
>>>> +        device on a given bus instance and MUST NOT forward transport messages
>>>> +        for a device number that has not been validated.
>
> An additional requirement could be added to limit the reuse of a device
> number after the device has disappeared, in order to avoid race
> conditions.

That is an interesting point that would require to increase the number of
device IDs as Demi pointed out.

Point noted and we will investigate (i will answer the increase of device IDs
directly to Demi's mail).

>
>>>> +  \item A bus implementation SHOULD provide the driver with sufficient
>>>> +        information—either via \busref{GET_DEVICES} or equivalent platform
>>>> +        data—to discover each valid device number.
>>>> +\end{itemize}
>>>> +
>
> [...]
>
>>>> +\busnormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Bus}
>>>> +\begin{itemize}
>>>> +  \item A bus implementation MAY report a transport-visible failure (for
>>>> +        example, after exhausting a bounded retry policy) when it cannot deliver
>>>> +        a request or obtain a response.
>>>> +  \item A bus implementation MUST treat malformed headers or unsupported
>>>> +        \field{msg_id} values as invalid, MUST discard them without generating
>>>> +        additional protocol traffic, and MAY log the condition for diagnostics.
>
> The bus does not have to understand transport messages, so does it have
> to recognize all unsupported msg_id values?

Very true, i will need to fix the wording to state invalid bus msg ids, transport should
discard invalid transport message IDs.

>
>>>> +  \item A bus implementation MUST NOT generate error responses to event
>>>> +        (one-way) messages.
>>>> +\end{itemize}
>>>> +
>>>> +\devicenormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Device}
>>>> +\begin{itemize}
>>>> +  \item A device receiving a malformed or unsupported transport message MUST
>>>> +        discard it without producing further protocol traffic.
>>>> +  \item Recovery actions taken in response to an error (such as retries,
>>>> +        selective resets, or device removal) MUST follow the normative reset and
>>>> +        status semantics defined in
>>>> +        \ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation}.
>>>> +\end{itemize}
>>>> +
>>>> +This specification does not define a dedicated error-reporting message; it only
>>>> +permits implementations to surface failures when silent recovery is not
>>>> +feasible.
>>>> +
>
> [...]
>
>>>> +\drivernormative{\paragraph}{Initialization Flow}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview / Driver}
>>>> +\begin{itemize}
>>>> +  \item A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature
>>>> +        negotiation or queue setup.
>>>> +  \item A driver MUST complete feature negotiation via
>>>> +        \msgref{GET_DEVICE_FEATURES}/\msgref{SET_DRIVER_FEATURES}
>>>> +        and confirm the FEATURES\_OK state via \msgref{SET_DEVICE_STATUS} before
>>>> +        enabling virtqueues.
>>>> +  \item A driver MUST configure each virtqueue via \msgref{SET_VQUEUE} and
>>>> +        confirm its parameters (e.g., with \msgref{GET_VQUEUE}) before marking
>>>> +        the queue ready for I/O.
>
> The Linux kernel driver mentioned in the cover letter currently doesn't
> confirm using GET_VQUEUE. But if SET_VQUEUE would return whether the
> queue is enabled, as was discussed below, the driver would only need to
> check whether the queue was successfully enabled.

GET_VQUEUE would still be need to have functionalities equivalent to PCI or MMIO.

But agree that it will not be needed to be done anymore there in the driver if we have
the enabled status in the answer to SET.

>
>>>> +  \item A driver MUST drive the device status transitions using
>>>> +        \msgref{SET_DEVICE_STATUS}, ensuring the device reaches DRIVER\_OK
>>>> +        before issuing normal I/O.
>>>> +\end{itemize}
>>>> +
>
> [...]
>
>>>> +\devicenormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device}
>>>> +\begin{itemize}
>>>> +  \item A device (or device-side bus) MUST send \msgref{EVENT_CONFIG} whenever
>>>> +        it makes a configuration change or status update that becomes visible to
>>>> +        the driver. The message MUST include the new configuration generation
>>>> +        count and MAY include the updated configuration data.
>>>> +  \item If the configuration data is omitted from \msgref{EVENT_CONFIG}, the
>>>> +        device SHOULD include the relevant offsets/lengths so the driver can
>>>> +        re-fetch the data via \msgref{GET_CONFIG}.
>>>> +  \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when
>>>> +        buffers on a virtqueue have been consumed, unless the device relies on
>>>> +        an alternative, agreed-upon completion mechanism.
>>>
>>> Should this not be a MUST requirement? The driver might hang or need to
>>> poll otherwise.
>>
>> The idea here was to let this open so that implementation relying only on poll are
>> still compliant hence the SHOULD.
>>
>> Maybe we can try to rephrase this in a better way.
>
> In my understanding rephrasing as MUST would be OK, since that only
> applies "unless the device relies on an alternative [...]".

Ack

>
>>
>>>
>>>> +\end{itemize}
>>>> +
>
> [...]
>
>>>> +\drivernormative{\paragraph}{Runtime Virtqueue Changes}{Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Driver}
>>>> +\begin{itemize}
>>>> +  \item A driver MAY configure additional virtqueues after initialization using
>>>> +        \msgref{SET_VQUEUE}, provided it follows the same validation steps
>>>> +        (e.g., checking the maximum queue size).
>>>> +  \item If VIRTIO\_F\_RING\_RESET is negotiated, the driver SHOULD use
>>>> +        \msgref{RESET_VQUEUE} before reprogramming a queue to avoid races with
>>>> +        the device.
>>>
>>> I do not understand how this avoids races which could not be prevented
>>> through a single message.
>>
>> The idea here is to say that an active queue should first be stopped/reseted before
>> reconfiguring it so that there could not any ongoing usage of the queue when it being
>> modified.
>>
>> At the end i guess this is not specific to that transport, so we might simply remove this.
>
> I think it should be fine to remove the requirement.

Ack.

>
> [...]
>
>>>> +\msgdef{GET_DEVICE_FEATURES}
>>>> +
>>>> +Drivers retrieve device feature bits in 32-bit blocks via
>>>> +\msgref{GET_DEVICE_FEATURES}; the response echoes the requested block index and
>>>> +returns one or more 32-bit values with the feature bits in that range.
>>>
>>> I do not understand the purpose of echoing request values. Looking at
>>> virtqueue messages, the driver keeps track of its outstanding requests
>>> internally and looks them up upon receiving the response. Doesn't a
>>> virtio-msg transport also keep track internally? What advantage has
>>> echoing?
>>
>> We only echo the index in the response (there is no data in the request).
>> The idea here is to allow asynchronous requests/response and provide
>> an easy way to identify the content of the data in the response.
>
> I would have thought that it is simpler and more robust if
> implementations would refer to the token to identify which request is
> being responded to (instead of possibly taking the response echoes at
> face value). Implementations might even have at most one outstanding
> request at a time.

We want to be future proof so that this is possible in the future.
If tokens are purely handled by the bus, having all information might be
easier for the transport.

>
>>
>>>
>>>> +
>>>> +\begin{lstlisting}
>>>> +struct virtio_msg_get_device_features_req {
>>>> +        le32 block_index; /* starting block (0 == bits 0-31) */
>>>> +        le32 num_blocks;  /* number of 32-bit blocks requested */
>>>> +};
>>>> +
>>>> +struct virtio_msg_get_device_features_resp {
>>>> +        le32 block_index; /* echoed starting block */
>>>> +        le32 num_blocks;  /* echoed number of blocks */
>>>> +        le32 features[];  /* num_blocks entries, zero-padded if needed */
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>
> [...]
>
>>>> +\drivernormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Driver}
>>>> +\begin{itemize}
>>>> +  \item A driver MUST set only the feature bits it intends to enable in the
>>>> +        blocks supplied to \msgref{SET_DRIVER_FEATURES}.
>>>> +\end{itemize}
>>>> +
>>>> +\devicenormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Device}
>>>> +\begin{itemize}
>>>> +  \item After processing \msgref{SET_DRIVER_FEATURES}, a device MUST update its
>>>> +        acknowledged feature set to match the data supplied and MUST report that
>>>> +        set consistently in subsequent \msgref{GET_DEVICE_FEATURES} responses.
>>>> +        If it cannot support the requested set, it SHOULD clear the FEATURES\_OK
>>>> +        bit in the device status.
>>>
>>> W.r.t. device status I think this rewording would be clearer:
>>>
>>> it SHOULD clear -> it MUST NOT set
>>
>> How about: It MUST clear the FEATURES_OK bit in the device status if it cannot support the requested set ?
>>
>> MUST NOT set implies that it was necessarily unset before which be a bit less precise.
>
> My understanding:
>
> "Device Requirements: Feature Bits" mandates that
>
> The device SHOULD accept any valid subset of features the driver
> accepts, otherwise it MUST fail to set the FEATURES_OK device
> status bit when the driver writes it.
>
> According to "Driver Requirements: Device Initialization", the
> FEATURES_OK bit in the device status is reset in step 1. It is only set
> in step 5 if the device accepts the subset accepted by the driver.
>
> Therefore, the FEATURES_OK bit in the device status is not set in the
> scenario outlined in the last sentence. But the last sentence also seems
> unnecessary as a transport requirement, so could just be removed.

Agree

>
>>
>>>
>>>> +\end{itemize}
>>>> +
>
> [...]
>
>>>> +\msgdef{SET_DEVICE_STATUS}
>>>> +
>>>> +\msgref{SET_DEVICE_STATUS} writes a new device status value. Drivers use it to
>>>> +progress through the virtio-defined states or to request a reset by writing 0.
>>>> +The device responds with its resulting status, which may differ (for example,
>>>> +if it refuses FEATURES\_OK or sets DEVICE\_NEEDS\_RESET).
>>>> +
>>>> +\begin{lstlisting}
>>>> +struct virtio_msg_set_device_status_req {
>>>> +        le32 status; /* desired device status value */
>>>> +};
>>>> +
>>>> +struct virtio_msg_set_device_status_resp {
>>>> +        le32 status; /* resulting device status */
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +\drivernormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Driver}
>>>> +\begin{itemize}
>>>> +  \item A driver MUST write 0 via \msgref{SET_DEVICE_STATUS} to request a device
>>>> +        reset and MUST re-read the status (e.g., via \msgref{GET_DEVICE_STATUS})
>>>> +        if it needs to confirm acceptance.
>
> SET_DEVICE_STATUS already responds with the resulting device status.
> Why would re-reading the status be necessary?

We identified some cases in which a reset cannot be handled atomically hence the need to have a way to confirm or wait until reset is finished.

>
>>>> +\end{itemize}
>>>> +
>>>> +\devicenormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Device}
>>>> +\begin{itemize}
>>>> +  \item A device MAY clear FEATURES\_OK or set DEVICE\_NEEDS\_RESET in its
>>>> +        response if it cannot accept the requested status, but it MUST report
>>>> +        the resulting status accurately.
>>>> +\end{itemize}
>>>> +

Cheers
Bertrand

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.