RE: [PATCH v1 0/4] virtio-msg transport layer

> From: Bertrand Marquis <[email protected]>
> Sent: 25 February 2026 08:24 PM
> 
> Hi,
> 
> > On 25 Feb 2026, at 15:49, Michael S. Tsirkin <[email protected]> wrote:
> >
> > On Wed, Feb 25, 2026 at 02:45:35PM +0000, Parav Pandit wrote:
> >>
> >>> From: Bertrand Marquis <[email protected]>
> >>> Sent: 25 February 2026 04:06 PM
> >>>
> >>> Hi Parav,
> >>>
> >>>> On 25 Feb 2026, at 11:24, Parav Pandit <[email protected]> wrote:
> >>>>
> >>>>>
> >>>>> From: Manivannan Sadhasivam <[email protected]>
> >>>>> Sent: 25 February 2026 03:37 PM
> >>>>>
> >>>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote:
> >>>>>> Hi Manivannan,
> >>>>>>
> >>>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <[email protected]> wrote:
> >>>>>>>
> >>>>>>> Hi Bertrand,
> >>>>>>>
> >>>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote:
> >>>>>>>> Hi Parav,
> >>>>>>>>
> >>>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <[email protected]> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> From: Michael S. Tsirkin <[email protected]>
> >>>>>>>>>> Sent: 20 February 2026 05:25 AM
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote:
> >>>>>>>>>>> Hi Bill,
> >>>>>>>>>>>
> >>>>>>>>>>>> From: Bill Mills <[email protected]>
> >>>>>>>>>>>> Sent: 26 January 2026 10:02 PM
> >>>>>>>>>>>>
> >>>>>>>>>>>> This series adds the virtio-msg transport layer.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in
> >>>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one
> >>>>>>>>>>>> more transport that performs its transport layer operations by sending and
> >>>>>>>>>>>> receiving messages.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers:
> >>>>>>>>>>>> * common / device level
> >>>>>>>>>>>> * bus level
> >>>>>>>>>>>>
> >>>>>>>>>>>> The common / device level defines the messages exchanged between the driver
> >>>>>>>>>>>> and a device. This common part should lead to a common driver holding most
> >>>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations.
> >>>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport
> >>>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device
> >>>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the
> >>>>>>>>>>>> main focus of this version of the patch series.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines
> >>>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and
> >>>>>>>>>>>> receive operations. A number of bus implementations are envisioned,
> >>>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations
> >>>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card
> >>>>>>>>>>>> and its driver.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations
> >>>>>>>>>>>> doing the same thing in different ways for no good reason. However the
> >>>>>>>>>>>> different environments will require different things. Instead of trying to
> >>>>>>>>>>>> anticipate all needs and provide something very abstract, we think
> >>>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time,
> >>>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to
> >>>>>>>>>>>> standardize a bus level message for that.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I would review more, had first round of sparse review.
> >>>>>>>>>>> Please find few comments/questions below.
> >>>>>>>>>>
> >>>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and
> >>>>>>>>>> then add performance features on top as appropriate.
> >>>>>>>>> Sounds good. Simple but complete is needed.
> >>>>>>>>
> >>>>>>>> Agree.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> So one way to address some of these comments is to show how
> >>>>>>>>>> they can be addressed with a feature bit down the road.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header.
> >>>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices.
> >>>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field.
> >>>>>>>>>>>
> >>>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer.
> >>>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data.
> >>>>>>>>>>> Needs 32-bits.
> >>>>>>>>>>>
> >>>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY)
> >>>>>>>>>>> But more below.
> >>>>>>>>>>>
> >>>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses
> to
> >>>>> operate
> >>>>>>>>>> the virtqueues.
> >>>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response?
> >>>>>>>>>>
> >>>>>>>>>> surely this can be an optional transport feature bit.
> >>>>>>>>>>
> >>>>>>>>> How is this optional?
> >>>>>>>>
> >>>>>>>> As said in a previous mail, we have messages already for that.
> >>>>>>>> Please confirm if that answer your question.
> >>>>>>>>
> >>>>>>>>> How can one implement a transport without defining the basic data transfer semantics?
> >>>>>>>>
> >>>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O.
> >>>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue.
> >>>>>>>>
> >>>>>>>
> >>>>>>> I'd love to have this transport over PCI because it addresses the shortcomings
> >>>>>>> of the existing PCI transport which just assumes that every config space access\
> >>>>>>> is trap and emulate.
> >>>>>>
> >>>>>> Agree and AMD did exactly that in their demonstrator.
> >>>>>> I will give you answers here as i know them but Edgar will probably give you more
> >>>>>> details (and probably fix my mistakes).
> >>>>>>
> >>>>>>>
> >>>>>>> But that being said, I somewhat agree with Parav that we should define the bus
> >>>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For
> >>>>>>> instance, if we try to use this transport over PCI, we've got questions like:
> >>>>>>>
> >>>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with
> >>>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs?
> >>>>>>
> >>>>>> One bus is appearing as one pci device with its own Vendor ID,
> >>>>>>
> >>>>>
> >>>>> What should be the 'own Vendor ID' here?
> >>>>>
> >>>>> The existing virtio-pci driver binds to all devices with the Vendor ID of
> >>>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own
> >>>>> VID for exposing the Virtio devices? That would mean, the drivers on the host
> >>>>> need update as well, which will not scale.
> >>>>>
> >>>>> It would be good if the existing virtio-pci devices can use this new transport
> >>>>> with only device side modifications.
> >>>>>
> >>>>>>>
> >>>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config
> >>>>>>> space or through some other means?
> >>>>>>
> >>>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI
> >>>>>> device (ending up being memory shared between both sides)
> >>>>>>
> >>>>>
> >>>>> What should be the BAR number and size?
> >>>>>
> >>>>>>>
> >>>>>>> 3. How the notification be delivered from the device to the host? Through
> >>>>>>> INT-X/MSI/MSI-X or even polling?
> >>>>>>
> >>>>>> Notifications are delivered through MSI.
> >>>>>>
> >>>>>
> >>>>> So no INT-X or MSI-X? Why so?
> >>>>>
> >>>>> Anyhow, my objective is not to get answers for my above questions here in this
> >>>>> thread, but to state the reality that it would be hard for us to make use of
> >>>>> this new transport without defining the bus implementation.
> >>>>>
> >>>> +1 to most of the points that Manivannan explained.
> >>>>
> >>>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet
> >>> another interface for _Everything_ that already exists.
> >>>> and device is still have to implement all the existing things because the device does not know which driver will operate.
> >>>>
> >>>> And that too some register based inefficient interface.
> >>>> Just to reset the device one needs to fully setup the new message interface but device still have to be working.
> >>>> That defeats the whole purpose of reset_1 and reset_2 in the device.
> >>>>
> >>>> This does not bring anything better for the PCI devices at all.
> >>>>
> >>>> A transport binding should be defined for the bus binding.
> >>>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are.
> >>>>
> >>>> If we are creating something brand-new, for PCI the only thing needed is:
> >>>> 1. Reset the device
> >>>> 2. Create an admin virtqueue
> >>>> 3. Transport everything needed through this virtqueue including features, configs, control.
> >>>>
> >>>> And this will work for any other bus or msg based too given only contract needed is to creating the aq.
> >>>
> >>> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain.
> >>>
> >>> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI".
> >>>
> >>> When the virtio-msg bus over PCI it will communicate through this device with an external
> >>> system connected through the PCI bus.
> >>> The driver will enumerate virtio devices available behind this bus and register them so that
> >>> the corresponding virtio drivers are probed for them.
> >>> All virtio-msg messages required to communicate with those devices will be transferred through
> >>> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the
> >>> virtqueues with all the devices on the bus.
> >>>
> >>> So the PCI device is not one virtio device but one bus behind which there can be many devices.
> >>>
> >>> Is this making the concept a bit clearer ?
> >>>
> >> Yes. This makes a lot of sense now.
> >>
> >> This is a virtio-msg-transport device that needs its own device id in the table.
> >> And its binding to the PCI transport.
> >
> > ok. how about an rfc of that idea on the list?
> 
> 
> I will let Edgar answer on this.
> 
Sure.
Also please explain how is this different than SR-IOV devices?
On a PCI device there are some child devices exposed using some virtio-msg bus transport.
And all of them are going to share same PCI BDF.
These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either.
So what would this virtio-msg-transport device achieve?

If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story.
But underlying transport being PCI, its back to square one for the originally described motivation.

> Cheers
> Bertrand
> 
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy
> the information in any medium. Thank you.