Re: [PATCH v1 0/4] virtio-msg transport layer

Hi Parav,

> On 13 Feb 2026, at 14:52, Parav Pandit <[email protected]> wrote:
>
> Hi Bill,
>
>> From: Bill Mills <[email protected]>
>> Sent: 26 January 2026 10:02 PM
>>
>> This series adds the virtio-msg transport layer.
>>
>> The individuals and organizations involved in this effort have had difficulty in
>> using the existing virtio-transports in various situations and desire to add one
>> more transport that performs its transport layer operations by sending and
>> receiving messages.
>>
>> Implementations of virtio-msg will normally be done in multiple layers:
>> * common / device level
>> * bus level
>>
>> The common / device level defines the messages exchanged between the driver
>> and a device. This common part should lead to a common driver holding most
>> of the virtio specifics and can be shared by all virtio-msg bus implementations.
>> The kernel implementation in [3] shows this separation. As with other transport
>> layers, virtio-msg should not require modifications to existing virtio device
>> implementations (virtio-net, virtio-blk etc). The common / device level is the
>> main focus of this version of the patch series.
>>
>> The virtio-msg bus level implements the normal things a bus defines
>> (enumeration, dma operations, etc) but also implements the message send and
>> receive operations. A number of bus implementations are envisioned,
>> some of which will be reusable and general purpose. Other bus implementations
>> might be unique to a given situation, for example only used by a PCIe card
>> and its driver.
>>
>> The standard bus messages are an effort to avoid different bus implementations
>> doing the same thing in different ways for no good reason. However the
>> different environments will require different things. Instead of trying to
>> anticipate all needs and provide something very abstract, we think
>> implementation specific messages will be needed at the bus level. Over time,
>> if we see similar messages across multiple bus implementations, we will move to
>> standardize a bus level message for that.
>>
>
> I would review more, had first round of sparse review.
> Please find few comments/questions below.
>
> 1. device number should be 32-bit in struct virtio_msg_header.
> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices.
> Also mapping PCI BDF wont be enough in 16-bits considering domain field.

That is a very interesting feedback, we will definitely take this into account.
We will have to decide how big and I will answer that in Demi's mail as there might be some drawbacks with
having very big sizes for the device ID.

In any case this point was raised by you, Peter and Demi and we will definitely handle it in v2.

>
> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer.
> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data.
> Needs 32-bits.

The point of the transport is not to transfer data, this should be done using virtqueues.
Transport is only there to allow access to registers, features and configuration, the main transfers are to
be done using virtqueues and i would not expect a network driver to use MMIO registers to write network
packet payloads but to use virtqueues for that (as vsock or net is doing).

So the msg_size here is just to fit transport specific requests.

>
> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY)
> But more below.

Good point, we will check that to make this coherent with other transports.

>
> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate the virtqueues.
> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response?

Equivalent of virtio_mem_read/write is provided by GET/SET CONFIG which allows you to access the configuration area.
So those are the equivalent of mem_read/write.
Does that answer your question ?

> Also, the queue notification message is missing at bus level.

Queue notifications are provided by event_avail and event_used messages.

> But I dont think queue notification via message good idea anyway.

As said in the spec, a bus can handle this using MSI or interrupts and generate the messages in the interface
between the bus and the transport. It is not an hard requirement to have those done using messages transfered and
the bus can do what it wants, but it must inform the transport using a message.

> We rather need a more higher-level message for virtqueues.
> Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding.
> Without this msg transport is incomplete.

I am not completely understanding what you mean here, are your concerns covered by the event messages ?

>
> 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in

We went back and forth on the naming, msg_id could be misunderstood as message opcode.
I guess we will have to find the best consensus here.

>
> 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request for reads.

This was pointed out in an other mail and i agree here.
I will investigate how we could make this a feature bit which i think could be a good solution.

>
> 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses.

Set vqueue has an index to specify the virtqueue index but you have to specific all fields in one go that is true.
Do you need a solution where you could set some fields to a specific value to say "keep current" and only update part of the vqueue configuration ?

> This requires caching all the values on the driver side before sending the transport request.
> I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface.
> and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings.
> It may require more plumbing, but it is cleaner interface when a new transport binding is created.

Admin queue useable with the message transport but I would be interested to understand exactly
the model you are referring to with create/destroy model.
Could you elaborate a bit so that i could understand what messages you would expect and how this would work ?

>
> 8. We should have the message response header for req-resp method with status code.
> (even though they are plain MMIO writes for pci).
> As opposed to that, proposed messages are doing composite tasks, hence we need the status response.

You want an error code in the header, is that what you mean ?
We went into this discussion while designing this, the main issue is that most drivers could not handle an
error as this is something not possible in MMIO or PCI so we decided instead to say that in case of error
a valid answer must be generated (all 0 when doing get_config for example).

>
> 9. virtio stack cannot handle hotplug devices today, at least Linux.
> So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough.
> A graceful method is needed.

Hotplug could be handled, remove is more problematic in Linux (you could declare a new device instance
at any time and this should work).

Could you give more details on what a graceful method would need ?

Regards
Bertrand

>
>> We are working on a few reusable bus implementations:
>>
>> * virtio-msg-ffa based on Arm FF-A interface for use between:
>>  * normal world and secure world
>>  * host and VM or VM to VM
>>  * Can be used w/ or with out a hypervisor
>>  * Any Hypervisor that implements FF-A can be used
>>  * We have this working with pKVM and Xen
>>  * We have this working with Trusty and OP-TEE
>>
>> * virtio-msg-amp for use between heterogenous systems
>>  * The main processors and its co-processors on an AMP SOC
>>  * Two or more systems connected via PCIe
>>  * Minimal requirements: bi-directional interrupts and
>>    at least one shared memory area
>>  * hvac-demo has 2 demos of this
>>  * This is working on two hardware platforms
>>
>> * virtio-msg-loopback for userspace implemented devices
>>  * Allows user space to provide devices to its own kernel
>>  * This is similar to fuse, cuse or loopback block devices but for virtio
>>  * hvac-demo has a demo
>>
>> We also anticipate a few more:
>>
>> * virtio-msg-xen specific to Xen
>>  * Usable on any Xen system (including x86 where FF-A does not exist)
>>  * Using Xen events and page grants
>>
>> * virtio-msg over admin virtqueues
>>  * This allows any virtio-pci device that supports admin virtqueues to also
>>    support a virtio-msg bus that supports sub devices
>>  * [We are looking for collaborators for this work]
>>
>> Changes since RFC2:
>>
>> Spec Functional:
>> * Made the common message header 8 bytes and added a token for optional use
>>  when parallel outstanding requests are possible
>> * Made the 8 byte fields align to 8 byte offsets.
>>  This effects the {SET,GET}_VQUEUE messages
>> * Many conformance cases have been tightened.
>>
>> Spec Editorial:
>> * Major restructure to better align with virtio spec
>>  * Conformance model now matches other transports
>>  * Use C structures to define messages instead of tables
>> * Added a section to describe how responses are matched to requests
>>  This includes the use of the new token field
>> * Redefined / better described error handling between transport and bus
>>  layers to eliminate the need for the bus to generate fake response messages
>> * Included editorial feedback from RFC2
>>
>> Eco-system:
>> * Added virtio-msg-loopback demo
>> * Arm has published the first draft of the virtio-msg over FFA spec [6]
>> * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE
>>  secure world
>> * LKML RFC has been sent [7]
>> * QEMU RFC has been sent [8]
>>
>> This is the first non-RFC patch series. The known short comings have been
>> addressed. We ask for review in earnest on this series and thank you for
>> any feedback you can provide.
>>
>> Background info and work in progress implementations:
>> * HVAC project page with intro slides [1]
>> * HVAC demo repo w/ instructions in README.md [2]
>> * Kernel w/ virtio-msg common level and ffa support [3]
>> * QEMU w/ support for one form of virtio-msg-amp [4]
>> * Portable RTOS library w/ one form of virtio-msg-amp [5]
>>
>> In addition to the QEMU system based demos in the hvac-demo repo, we also have
>> two hardware systems running:
>> * AMD x86 + AMD Arm Versal connected via PCIe
>> * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
>>
>> Please note that although the demos work, a few have not been aligned
>> with this version of the spec.
>>
>> [1]
>> https://linaro.atlassian.net/wiki/spaces/HVAC/overview
>> 05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6390
>> 50419790817373%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIs
>> IldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JnCO8QgAIiyGAprBLkprUd8A3162wCFpmGic3JKoHXo%3D&reserved=0
>> [2] https://github.com/wmamills/hvac-
>> demo&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7
>> C0%7C639050419790868944%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj
>> oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XeBuS8Hr%2FDETKfSyidKKGoGq6OTUcOYoeSGDouUMcf4%3D&reserved=0
>> [3]
>> https://git.kernel.org/pub/scm/linux/kernel/git/vireshk%252
>> Flinux.git%2Flog%2F%3Fh%3Dvirtio%2Fmsg&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d1
>> 5727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419790913867%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIw
>> LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PuLq68Y0ZNCYbfrgDZlipfm44xHZp1%2F%2
>> BLu%2FaKQBBB%2Fo%3D&reserved=0
>> [4] https://github.com/edgarigl/qemu/commits/edgar/virtio-
>> msg-
>> rfc%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%
>> 7C0%7C639050419790960971%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFO
>> IjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xyowKD9KG3IasQTiYCzfCRGqCLSF%2Fa1g4%2F2%2ByKb1OoA%3D&reserved=0
>> [5] https://github.com/arnopo/open-amp/commits/virtio-
>> msg%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0
>> %7C0%7C639050419791007707%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF
>> OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=69OwO2wXi0%2FEhDDSR2uLG2iis5RYzI%2FLnlfkcRYao9o%3D&reserved=0
>> [6] https://documentation-/
>> service.arm.com%2Fstatic%2F68f647791134f773ab3f0a7c&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d9
>> 5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419791052848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy
>> dWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=cD5VqKnTuIFqlN2XV3j%2FHQ
>> Wvsz8UVe5PM0ZjMU3o0sM%3D&reserved=0
>> [7]
>> https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linar
>> o.org%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C
>> 0%7C0%7C639050419791093596%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsI
>> kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=NmhmL%2BuyVqnpHT5r9EfI2uzPf9LSXLga7bUw3FO5x4w%3D&reserved=0
>> [8] https://mail.gnu.org/archive/html/qemu-devel/2025-
>> 10%2Fmsg07438.html&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd
>> 9ccc17a%7C0%7C0%7C639050419791139232%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJ
>> XaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=osI1FttOO9%2B01e%2B5EtwnYe%2Fl1%2BIvFjH%2Blyz2ipM3RM
>> Q%3D&reserved=0
>>
>> Bertrand Marquis (2):
>>  virtio-msg: add new command for bus normative
>>  virtio-msg: add conformance entries in conformance chapter
>>
>> Bill Mills (2):
>>  virtio-msg: Add virtio-msg, a message based virtio transport layer
>>  virtio-msg: link virtio-msg content
>>
>> commands.tex      |    3 +-
>> conformance.tex   |  105 ++-
>> content.tex       |    1 +
>> transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++
>> 4 files changed, 1746 insertions(+), 3 deletions(-)
>> create mode 100644 transport-msg.tex
>>
>> --
>> 2.34.1
>>
>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.