Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer

Hi Demi,

> On 24 Feb 2026, at 18:46, Demi Marie Obenour <[email protected]> wrote:
>
> On 2/24/26 12:20, Bertrand Marquis wrote:
>> Hi Demi,
>>
>>> On 24 Feb 2026, at 18:14, Demi Marie Obenour <[email protected]> wrote:
>>>
>>> On 1/26/26 11:32, Bill Mills wrote:
>>>> Add a new transport layer that is based on messages.
>>>>
>>>> This transport layer still uses virtqueues as the other transport layers do
>>>> but implements transport layer operations by sending and receiving messages
>>>> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci.
>>>>
>>>> This transport is useful when the device and driver are both implemented in
>>>> software but the trap and emulate operations of virtio-mmio and virtio-pci
>>>> can not be used.
>>>>
>>>> This transport is intended to be used in many situations, including:
>>>> * between a host processor and its co-processors
>>>> * between two different systems (not SMP) connected via PCIe
>>>> * between normal and secure worlds
>>>> * host to vm
>>>> * vm to vm
>>>
>>> (snip)
>>>
>>>> +As described in Sections
>>>> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications}
>>>> +and
>>>> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications},
>>>> +runtime notifications (\msgref{EVENT_AVAIL}, \msgref{EVENT_USED}, and
>>>> +\msgref{EVENT_CONFIG}) may be delivered in-band or via equivalent out-of-band
>>>> +mechanisms.
>>>
>>> Sending these notifications out of band should almost always be more
>>> efficient.  Under Xen, one should use per-virtqueue, per-direction
>>> event channels.  KVM has irqfd and ioeventfd, and the loopback
>>> transport can use per-virtqueue eventfds or io_uring queues.  I would
>>> be quite surprised if FF-A and PCI did not have similar mechanisms
>>> that could be used.
>>>
>>> I recommend removing these from the spec and expecting buses to
>>> provide their own notification mechanism.
>>
>> Busses can have more efficient way and can map that other real interrupts or xen events
>> but the bus has to create a fake message back to the transport to keep the interface
>> coherent.
>>
>> In ffa case we do use notifications but we still use messages as those can be more
>> efficient in some cases (indirect messages) and we have a limited number of notifications.
>> In ffa we have a also a transfer system using a FIFO which can hold several events at a time
>> preventing several ping/pong.
>>
>> We do let the door open so that a bus can choose how to do this but we keep the message
>> (even fake) between the bus and the transport to have a transport independent and handling
>> the OS specific way of signaling the drivers.
>>
>> Does that answer your question ?
>
> I expect that most implementations will have a single queue in each
> direction for bus messages.  This means that sending bus messages will
> require taking a lock.  Sending virtqueue notifications out of band,
> without a corresponding bus message, will avoid taking the lock and
> therefore significantly improve scalability on multi-core systems.

FF-A implementation using FIFOs for transfer has been designed to use
2 lock-less queues (one in each direction) which should prevent those
kind of issues.

In any case the transport does not limit what you can do in any way. In
his design for system to system, Bill has designed something where
messages are transfered using queues but events are transferred using
out of band notifications and some extra info stored in a shared memory.

> For instance, a Xen transport can use per-virtqueue event channels
> and a KVM or MSHV transport can use per-virtqueue irqfds/ioeventfds.

It is definitely possible and the transport is not enforcing or denying any
way to do and is mentioning out of band:
- in Driver notifications: "the bus may translate EVENT_AVAIL into out-of-band
  or deliver in-band; it MUST relay in-band if it does not translate; it MUST NOT
  drop EVENT_AVAIL unless polling/equivalent is arranged."
- in Device notifications: "the bus may forward in-band or synthesize from other
  signals; it MUST forward (or equivalent) to the driver."

We even have a bus normative to say that NOTIFICATION_DATA should not
be negotiated if out of band notification is used.

But I agree we could benefit from a more explicit mention ...

>
> I strongly recommend language like the following:
>
> EVENT_CONFIG, EVENT_USED, and EVENT_AVAIL MAY be sent in-band or
> out-of-band.  The specification of a given bus MUST specify which
> is used.  If a bus specifies that out-of-band notifications are to
> be used, it MUST document how to send them.

so I will include something like that in the v2 of the spec.

Cheers
Bertrand

> --
> Sincerely,
> Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc>


IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.