Re: [PATCH v7] virtio-vsock: Add support for multi devices

> > From: Xuewei Niu <[email protected]>
> > Sent: Monday, June 16, 2025 2:30 PM
> > 
> > > > From: Xuewei Niu <[email protected]>
> > > > Sent: Monday, June 16, 2025 1:48 PM
> > > >
> > > > Hi, Parav.
> > > >
> > > > Thanks for your detailed comments.
> > > >
> > > > > Hi Xuewei,
> > > > >
> > > > > > From: Xuewei Niu <[email protected]>
> > > > > > Sent: Monday, May 19, 2025 3:08 PM
> > > > > >
> > > > > > > On Sat, Apr 12, 2025 at 10:28:25PM +0800, Xuewei Niu wrote:
> > > > > > > > This patch brings a new feature, called "multi devices", to
> > > > > > > > the virtio vsock. It introduces a "VIRTIO_VSOCK_F_MULTI_DEVICES"
> > > > > > > > feature bit, and a "device_order" field to the config for the virtio
> > vsock.
> > > > > > > >
> > > > > > > > == Motivition ==
> > > > > > > >
> > > > > > > > Vsock is a lightweight and widely used data exchange
> > > > > > > > mechanism between host and guest.
> > > > >
> > > > > Even though it is the current use, the specification does not
> > > > > prevent its usage
> > > > between two guests via a host.
> > > > > So we should not assume such guest <-> host communication as the
> > > > > only
> > > > case to add new feature.
> > > > >
> > > > > For example, in the spec only must requirement is that src_cid ==
> > > > config.guest_cid.
> > > > > Dst_cid can be anything, it need not be well known 0x2 (for the host).
> > > >
> > > > Yes.
> > > >
> > > > > With this flexibility in the spec, one can connect vsock devices
> > > > > with multiple
> > > > different backends.
> > > > >
> > > > > For example,
> > > > > QEMU can insert one vsock device for VM to HV communication.
> > > >
> > > > It is also able to communicate with other devices on the same host, i.e.
> > > > VM-to-VM.
> > > >
> > > > > A real PCI device can insert one voscket device for VM-to-VM
> > > > communication bypassing a full TCP/IP stack.
> > > >
> > > > AFAIK, all devices are implemented in software. Is it a real PCI HW device?
> > > >
> > > Yes. virto PCI devices are implemented as hw or as vdpa for many years now by
> > cloud operators and by NIC vendors.
> > 
> > Thanks for your confirmation.
> > 
> > > > > This means there are two different backends.
> > > > > And these two devices should not be grouped in the use case you
> > described.
> > > >
> > > > I think one group is enough for all use cases. It is required that
> > > > CIDs are unique in global, i.e. all backends.
> > > >
> > > > As your example, let me assume there are two VMs
> > > >
> > > > 1. VM0 (two vsock backends)
> > > >     1.1 device0 (cid=3, default), backend is host kernel (vhost-vsock);
> > > >     1.2 device1 (cid=4), backend is HV (virtio-vsock).
> > > > 2. VM1
> > > >     2.1 device2 (cid=5, default), backend is host kernel (vhost-vsock).
> > > >
> > > > The device0 is able to do VM0-HOST (src_cid=3, dst_cid=2) and
> > > > VM0-VM1 (src_cid=3, dst_cid=5) communicatation, while the device1 is
> > > > only able to do VM0-HV (src_cid=4, dst_cid=2) communicatation.
> > > >
> > > In example 1.1 and 1.2 no devices are grouped.
> > > Your proposal of this patch wants to group the two devices and pick one of
> > them as default device.
> > 
> > Yes. I just wonder if it is possible to have more than one groups in one guest?
> > 
> For example
> Group_1: two devices dev0 and dev1, implemented as PCI HW devices.
> Group_2: two devices by QEMU SW implemented as sw backend.
> 
> All the 4 devices has _F bit indicating they can be grouped.
> But there is no indication that they are part of which group.
> And hence the guest VM driver is in dark on how to forward requests without the bind() call.

I see. Thanks!

My idea is that there is only one default device, no matter how many types
of backends are. If the users intend to use other devices, `bind()` call
is required.

For example, we set `dev0` as the default device:

1. Do not call `bind()`: use dev0;
2. Call `bind(${dev0_cid})`: use dev1;
...
5. Call `bind(${dev4_cid})`: use dev4;

Even though we introduce the group concept, if we don't call `bind()`, how
does driver know which group to use? If the driver recoginizes the dst_cid,
it can use the group to find the device, then the things will be
complicated. The driver needs to know the relationship between the dst_cid
and the group.

WDYT?

Thanks,
Xuewei

> > > If device0 and device1 are inserted to the VM0 with the feature bit you
> > suggested, the guest things that they are part of the same group.
> > 
> > In this patch, we don't allow to insert devices without the feature bit if there are
> > already devices with the feature bit.
> > 
> In above example of two groups, all the 4 devices spread across two groups has the feature bit set.
> Yet, they cannot be grouped correctly.
> Driver driving blind thinks that all 4 devices are part of the same group.
> 
> > As a result, there can be either multiple devices with the feature bit or just a
> > single device.
> > 
> 
> > > When bind() call is not done, host sw does not know which device to pick up
> > between device0 and device1 when binding the devices.
> > 
> > Are you referring to an vsock application on the host? If yes, "host sw" is able to
> > pick up one device according to the dst cid. For example, pick up
> > device0 if `connect(3)` is called.
> > 
> > Please be aware that the "host sw" can not pick up device1, since its device is not
> > in the host kernel.
> > 
> > > > In a word, a tuple identifies a connection.
> > > >
> > > > "Refuse to connect" will be raised if the device1 attempts to
> > > > connect to the device2. "They are not in the same group" is a reasonable
> > explaination.
> > > > Am I right?
> > > >
> > > During bind call, one needs to select the device when the devices are coming
> > from multiple different backends.
> > 
> > Yes.
> > 
> > Thanks,
> > Xuewei