Re: [PATCH v1 0/4] virtio-msg transport layer

Demi Marie Obenour <[email protected]> Wed, 25 Feb 2026 10:23:12 -0500
Newsgroups	dev.linux.lists.virtio-comment
Message-ID	<[email protected]>
On 2/25/26 10:00, Parav Pandit wrote:
> 
>> From: Bertrand Marquis <[email protected]>
>> Sent: 25 February 2026 08:24 PM
>>
>> Hi,
>>
>>> On 25 Feb 2026, at 15:49, Michael S. Tsirkin <[email protected]> wrote:
>>>
>>> On Wed, Feb 25, 2026 at 02:45:35PM +0000, Parav Pandit wrote:
>>>>
>>>>> From: Bertrand Marquis <[email protected]>
>>>>> Sent: 25 February 2026 04:06 PM
>>>>>
>>>>> Hi Parav,
>>>>>
>>>>>> On 25 Feb 2026, at 11:24, Parav Pandit <[email protected]> wrote:
>>>>>>
>>>>>>>
>>>>>>> From: Manivannan Sadhasivam <[email protected]>
>>>>>>> Sent: 25 February 2026 03:37 PM
>>>>>>>
>>>>>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote:
>>>>>>>> Hi Manivannan,
>>>>>>>>
>>>>>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <[email protected]> wrote:
>>>>>>>>>
>>>>>>>>> Hi Bertrand,
>>>>>>>>>
>>>>>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote:
>>>>>>>>>> Hi Parav,
>>>>>>>>>>
>>>>>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <[email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> From: Michael S. Tsirkin <[email protected]>
>>>>>>>>>>>> Sent: 20 February 2026 05:25 AM
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote:
>>>>>>>>>>>>> Hi Bill,
>>>>>>>>>>>>>
>>>>>>>>>>>>>> From: Bill Mills <[email protected]>
>>>>>>>>>>>>>> Sent: 26 January 2026 10:02 PM
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This series adds the virtio-msg transport layer.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in
>>>>>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one
>>>>>>>>>>>>>> more transport that performs its transport layer operations by sending and
>>>>>>>>>>>>>> receiving messages.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers:
>>>>>>>>>>>>>> * common / device level
>>>>>>>>>>>>>> * bus level
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The common / device level defines the messages exchanged between the driver
>>>>>>>>>>>>>> and a device. This common part should lead to a common driver holding most
>>>>>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations.
>>>>>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport
>>>>>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device
>>>>>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the
>>>>>>>>>>>>>> main focus of this version of the patch series.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines
>>>>>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and
>>>>>>>>>>>>>> receive operations. A number of bus implementations are envisioned,
>>>>>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations
>>>>>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card
>>>>>>>>>>>>>> and its driver.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations
>>>>>>>>>>>>>> doing the same thing in different ways for no good reason. However the
>>>>>>>>>>>>>> different environments will require different things. Instead of trying to
>>>>>>>>>>>>>> anticipate all needs and provide something very abstract, we think
>>>>>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time,
>>>>>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to
>>>>>>>>>>>>>> standardize a bus level message for that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would review more, had first round of sparse review.
>>>>>>>>>>>>> Please find few comments/questions below.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and
>>>>>>>>>>>> then add performance features on top as appropriate.
>>>>>>>>>>> Sounds good. Simple but complete is needed.
>>>>>>>>>>
>>>>>>>>>> Agree.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> So one way to address some of these comments is to show how
>>>>>>>>>>>> they can be addressed with a feature bit down the road.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header.
>>>>>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices.
>>>>>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer.
>>>>>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data.
>>>>>>>>>>>>> Needs 32-bits.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY)
>>>>>>>>>>>>> But more below.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses
>> to
>>>>>>> operate
>>>>>>>>>>>> the virtqueues.
>>>>>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response?
>>>>>>>>>>>>
>>>>>>>>>>>> surely this can be an optional transport feature bit.
>>>>>>>>>>>>
>>>>>>>>>>> How is this optional?
>>>>>>>>>>
>>>>>>>>>> As said in a previous mail, we have messages already for that.
>>>>>>>>>> Please confirm if that answer your question.
>>>>>>>>>>
>>>>>>>>>>> How can one implement a transport without defining the basic data transfer semantics?
>>>>>>>>>>
>>>>>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O.
>>>>>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'd love to have this transport over PCI because it addresses the shortcomings
>>>>>>>>> of the existing PCI transport which just assumes that every config space access\
>>>>>>>>> is trap and emulate.
>>>>>>>>
>>>>>>>> Agree and AMD did exactly that in their demonstrator.
>>>>>>>> I will give you answers here as i know them but Edgar will probably give you more
>>>>>>>> details (and probably fix my mistakes).
>>>>>>>>
>>>>>>>>>
>>>>>>>>> But that being said, I somewhat agree with Parav that we should define the bus
>>>>>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For
>>>>>>>>> instance, if we try to use this transport over PCI, we've got questions like:
>>>>>>>>>
>>>>>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with
>>>>>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs?
>>>>>>>>
>>>>>>>> One bus is appearing as one pci device with its own Vendor ID,
>>>>>>>>
>>>>>>>
>>>>>>> What should be the 'own Vendor ID' here?
>>>>>>>
>>>>>>> The existing virtio-pci driver binds to all devices with the Vendor ID of
>>>>>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own
>>>>>>> VID for exposing the Virtio devices? That would mean, the drivers on the host
>>>>>>> need update as well, which will not scale.
>>>>>>>
>>>>>>> It would be good if the existing virtio-pci devices can use this new transport
>>>>>>> with only device side modifications.
>>>>>>>
>>>>>>>>>
>>>>>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config
>>>>>>>>> space or through some other means?
>>>>>>>>
>>>>>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI
>>>>>>>> device (ending up being memory shared between both sides)
>>>>>>>>
>>>>>>>
>>>>>>> What should be the BAR number and size?
>>>>>>>
>>>>>>>>>
>>>>>>>>> 3. How the notification be delivered from the device to the host? Through
>>>>>>>>> INT-X/MSI/MSI-X or even polling?
>>>>>>>>
>>>>>>>> Notifications are delivered through MSI.
>>>>>>>>
>>>>>>>
>>>>>>> So no INT-X or MSI-X? Why so?
>>>>>>>
>>>>>>> Anyhow, my objective is not to get answers for my above questions here in this
>>>>>>> thread, but to state the reality that it would be hard for us to make use of
>>>>>>> this new transport without defining the bus implementation.
>>>>>>>
>>>>>> +1 to most of the points that Manivannan explained.
>>>>>>
>>>>>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet
>>>>> another interface for _Everything_ that already exists.
>>>>>> and device is still have to implement all the existing things because the device does not know which driver will operate.
>>>>>>
>>>>>> And that too some register based inefficient interface.
>>>>>> Just to reset the device one needs to fully setup the new message interface but device still have to be working.
>>>>>> That defeats the whole purpose of reset_1 and reset_2 in the device.
>>>>>>
>>>>>> This does not bring anything better for the PCI devices at all.
>>>>>>
>>>>>> A transport binding should be defined for the bus binding.
>>>>>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are.
>>>>>>
>>>>>> If we are creating something brand-new, for PCI the only thing needed is:
>>>>>> 1. Reset the device
>>>>>> 2. Create an admin virtqueue
>>>>>> 3. Transport everything needed through this virtqueue including features, configs, control.
>>>>>>
>>>>>> And this will work for any other bus or msg based too given only contract needed is to creating the aq.
>>>>>
>>>>> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain.
>>>>>
>>>>> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI".
>>>>>
>>>>> When the virtio-msg bus over PCI it will communicate through this device with an external
>>>>> system connected through the PCI bus.
>>>>> The driver will enumerate virtio devices available behind this bus and register them so that
>>>>> the corresponding virtio drivers are probed for them.
>>>>> All virtio-msg messages required to communicate with those devices will be transferred through
>>>>> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the
>>>>> virtqueues with all the devices on the bus.
>>>>>
>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices.
>>>>>
>>>>> Is this making the concept a bit clearer ?
>>>>>
>>>> Yes. This makes a lot of sense now.
>>>>
>>>> This is a virtio-msg-transport device that needs its own device id in the table.
>>>> And its binding to the PCI transport.
>>>
>>> ok. how about an rfc of that idea on the list?
>>
>>
>> I will let Edgar answer on this.
>>
> Sure.
> Also please explain how is this different than SR-IOV devices?
> On a PCI device there are some child devices exposed using some virtio-msg bus transport.
> And all of them are going to share same PCI BDF.
> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either.
> So what would this virtio-msg-transport device achieve?

Since the message transport is more peer-to-peer, would it make
sense to use this to allow VMs to implement virtio devices for use
by other VMs?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
OpenPGP_0xB288B55FFF9C22C1.asc (application/pgp-keys, 7 KB)
OpenPGP_signature.asc (application/pgp-signature, 833 B)