Re: [PATCH v7] virtio-vsock: Add support for multi devices

> > From: Xuewei Niu <[email protected]>
> > Sent: 19 June 2025 08:57 AM
> > 
> > Hi Parav,
> > 
> > Could you please take a look at the diagram in [1]?
> > 
> > IIUC, for VM0, there are two groups, and for VM1, there is only one group.
> > Am I right? If yes, I think the group concept is reasonable but we don't need
> > at this time.
> > 
> > I think the first thing is to figure out how to pick the right group.
> > 
> > Standard socket doesn't provide a way to access the group information.
> > 
> > Source and destination are from `bind()` and `connect()`, respectively. If we
> > don't call `bind()`, only the destination is known.
> > 
> > However, only destination is not enough to find the group. For example, the
> > well-known CIDs (e.g. 2) are valid for all groups.
> > 
> > 1:
> > https://lore.kernel.org/virtio-comment/[email protected]/
> 
> Right. Sock addressing scheme is naïve presently to select the group. Not sure when/how you or others plan to do.
> This is transport layer problem to solve (not to confuse with transport = pci/mmio etc).
> 
> However, at device level, we should have the construct of grouping.
> Without this construct, all devices will be part of single group and one will not be able to build the group concept later.
> So even if you don't need it explicitly now, grouping the device is what you need when connect() is called.
> 
> So I was imagining a relatively simple scheme:
> For example, virtio device level, some kind of group id is present.
> So two devices which has same group id, are part of single group.
> An example group id format can be a UUID.
> 
> And this is completely optional for devices to implement.
> Generic enough and usable beyond just vsock device in other use cases we discussed in past.

Fair enough.

@Stefano, could you please take a look at this? I'd love to have some input
from you.

A brief summary of the idea is: The config space will be extended to
include a group id. The devices with the same group id are considered to be
in the same group.

Thanks,
Xuewei
 
> > 
> > > > From: Xuewei Niu <[email protected]>
> > > > Sent: Monday, June 16, 2025 4:26 PM
> > > >
> > > > > > From: Xuewei Niu <[email protected]>
> > > > > > Sent: Monday, June 16, 2025 2:30 PM
> > > > > >
> > > > > > > > From: Xuewei Niu <[email protected]>
> > > > > > > > Sent: Monday, June 16, 2025 1:48 PM
> > > > > > > >
> > > > > > > > Hi, Parav.
> > > > > > > >
> > > > > > > > Thanks for your detailed comments.
> > > > > > > >
> > > > > > > > > Hi Xuewei,
> > > > > > > > >
> > > > > > > > > > From: Xuewei Niu <[email protected]>
> > > > > > > > > > Sent: Monday, May 19, 2025 3:08 PM
> > > > > > > > > >
> > > > > > > > > > > On Sat, Apr 12, 2025 at 10:28:25PM +0800, Xuewei Niu wrote:
> > > > > > > > > > > > This patch brings a new feature, called "multi
> > > > > > > > > > > > devices", to the virtio vsock. It introduces a
> > > > "VIRTIO_VSOCK_F_MULTI_DEVICES"
> > > > > > > > > > > > feature bit, and a "device_order" field to the
> > > > > > > > > > > > config for the virtio
> > > > > > vsock.
> > > > > > > > > > > >
> > > > > > > > > > > > == Motivition ==
> > > > > > > > > > > >
> > > > > > > > > > > > Vsock is a lightweight and widely used data exchange
> > > > > > > > > > > > mechanism between host and guest.
> > > > > > > > >
> > > > > > > > > Even though it is the current use, the specification does
> > > > > > > > > not prevent its usage
> > > > > > > > between two guests via a host.
> > > > > > > > > So we should not assume such guest <-> host communication
> > > > > > > > > as the only
> > > > > > > > case to add new feature.
> > > > > > > > >
> > > > > > > > > For example, in the spec only must requirement is that
> > > > > > > > > src_cid ==
> > > > > > > > config.guest_cid.
> > > > > > > > > Dst_cid can be anything, it need not be well known 0x2 (for the
> > host).
> > > > > > > >
> > > > > > > > Yes.
> > > > > > > >
> > > > > > > > > With this flexibility in the spec, one can connect vsock
> > > > > > > > > devices with multiple
> > > > > > > > different backends.
> > > > > > > > >
> > > > > > > > > For example,
> > > > > > > > > QEMU can insert one vsock device for VM to HV communication.
> > > > > > > >
> > > > > > > > It is also able to communicate with other devices on the same host,
> > i.e.
> > > > > > > > VM-to-VM.
> > > > > > > >
> > > > > > > > > A real PCI device can insert one voscket device for
> > > > > > > > > VM-to-VM
> > > > > > > > communication bypassing a full TCP/IP stack.
> > > > > > > >
> > > > > > > > AFAIK, all devices are implemented in software. Is it a real PCI HW
> > device?
> > > > > > > >
> > > > > > > Yes. virto PCI devices are implemented as hw or as vdpa for
> > > > > > > many years now by
> > > > > > cloud operators and by NIC vendors.
> > > > > >
> > > > > > Thanks for your confirmation.
> > > > > >
> > > > > > > > > This means there are two different backends.
> > > > > > > > > And these two devices should not be grouped in the use
> > > > > > > > > case you
> > > > > > described.
> > > > > > > >
> > > > > > > > I think one group is enough for all use cases. It is
> > > > > > > > required that CIDs are unique in global, i.e. all backends.
> > > > > > > >
> > > > > > > > As your example, let me assume there are two VMs
> > > > > > > >
> > > > > > > > 1. VM0 (two vsock backends)
> > > > > > > >     1.1 device0 (cid=3, default), backend is host kernel (vhost-vsock);
> > > > > > > >     1.2 device1 (cid=4), backend is HV (virtio-vsock).
> > > > > > > > 2. VM1
> > > > > > > >     2.1 device2 (cid=5, default), backend is host kernel (vhost-vsock).
> > > > > > > >
> > > > > > > > The device0 is able to do VM0-HOST (src_cid=3, dst_cid=2)
> > > > > > > > and
> > > > > > > > VM0-VM1 (src_cid=3, dst_cid=5) communicatation, while the
> > > > > > > > device1 is only able to do VM0-HV (src_cid=4, dst_cid=2)
> > > > communicatation.
> > > > > > > >
> > > > > > > In example 1.1 and 1.2 no devices are grouped.
> > > > > > > Your proposal of this patch wants to group the two devices and
> > > > > > > pick one of
> > > > > > them as default device.
> > > > > >
> > > > > > Yes. I just wonder if it is possible to have more than one groups in one
> > guest?
> > > > > >
> > > > > For example
> > > > > Group_1: two devices dev0 and dev1, implemented as PCI HW devices.
> > > > > Group_2: two devices by QEMU SW implemented as sw backend.
> > > > >
> > > > > All the 4 devices has _F bit indicating they can be grouped.
> > > > > But there is no indication that they are part of which group.
> > > > > And hence the guest VM driver is in dark on how to forward
> > > > > requests without
> > > > the bind() call.
> > > >
> > > > I see. Thanks!
> > > >
> > > > My idea is that there is only one default device, no matter how many
> > > > types of backends are. If the users intend to use other devices, `bind()` call
> > is required.
> > > >
> > > > For example, we set `dev0` as the default device:
> > > >
> > > > 1. Do not call `bind()`: use dev0;
> > > > 2. Call `bind(${dev0_cid})`: use dev1; ...
> > > > 5. Call `bind(${dev4_cid})`: use dev4;
> > > >
> > > > Even though we introduce the group concept, if we don't call
> > > > `bind()`, how does driver know which group to use? If the driver
> > > > recoginizes the dst_cid, it can use the group to find the device,
> > > > then the things will be complicated. The driver needs to know the
> > relationship between the dst_cid and the group.
> > > >
> > > Based on the dst_cid picking the right vscock group would be needed. This
> > is vsock level issue at driver level.
> > > Driver would need enough hints or encoding or of dst_cid or something
> > else.
> > >
> > > So even though we miss vsock level construct, it should be the reason to not
> > group the devices.
> > > As both attempt to solve issue at different level.
> > >
> > > > WDYT?
> > > >
> > > > Thanks,
> > > > Xuewei
> > > >
> > > > > > > If device0 and device1 are inserted to the VM0 with the
> > > > > > > feature bit you
> > > > > > suggested, the guest things that they are part of the same group.
> > > > > >
> > > > > > In this patch, we don't allow to insert devices without the
> > > > > > feature bit if there are already devices with the feature bit.
> > > > > >
> > > > > In above example of two groups, all the 4 devices spread across
> > > > > two groups has
> > > > the feature bit set.
> > > > > Yet, they cannot be grouped correctly.
> > > > > Driver driving blind thinks that all 4 devices are part of the same group.
> > > > >
> > > > > > As a result, there can be either multiple devices with the
> > > > > > feature bit or just a single device.
> > > > > >
> > > > >
> > > > > > > When bind() call is not done, host sw does not know which
> > > > > > > device to pick up
> > > > > > between device0 and device1 when binding the devices.
> > > > > >
> > > > > > Are you referring to an vsock application on the host? If yes,
> > > > > > "host sw" is able to pick up one device according to the dst
> > > > > > cid. For example, pick up
> > > > > > device0 if `connect(3)` is called.
> > > > > >
> > > > > > Please be aware that the "host sw" can not pick up device1,
> > > > > > since its device is not in the host kernel.
> > > > > >
> > > > > > > > In a word, a tuple identifies a connection.
> > > > > > > >
> > > > > > > > "Refuse to connect" will be raised if the device1 attempts
> > > > > > > > to connect to the device2. "They are not in the same group"
> > > > > > > > is a reasonable
> > > > > > explaination.
> > > > > > > > Am I right?
> > > > > > > >
> > > > > > > During bind call, one needs to select the device when the
> > > > > > > devices are coming
> > > > > > from multiple different backends.
> > > > > >
> > > > > > Yes.
> > > > > >
> > > > > > Thanks,
> > > > > > Xuewei