Re: [PATCH v7] virtio-vsock: Add support for multi devices

Hi Parav,

Could you please take a look at the diagram in [1]?

IIUC, for VM0, there are two groups, and for VM1, there is only one group.
Am I right? If yes, I think the group concept is reasonable but we don't
need at this time.

I think the first thing is to figure out how to pick the right group.

Standard socket doesn't provide a way to access the group information.

Source and destination are from `bind()` and `connect()`, respectively. If
we don't call `bind()`, only the destination is known.

However, only destination is not enough to find the group. For example, the
well-known CIDs (e.g. 2) are valid for all groups.

1: https://lore.kernel.org/virtio-comment/[email protected]/

> > From: Xuewei Niu <[email protected]>
> > Sent: Monday, June 16, 2025 4:26 PM
> > 
> > > > From: Xuewei Niu <[email protected]>
> > > > Sent: Monday, June 16, 2025 2:30 PM
> > > >
> > > > > > From: Xuewei Niu <[email protected]>
> > > > > > Sent: Monday, June 16, 2025 1:48 PM
> > > > > >
> > > > > > Hi, Parav.
> > > > > >
> > > > > > Thanks for your detailed comments.
> > > > > >
> > > > > > > Hi Xuewei,
> > > > > > >
> > > > > > > > From: Xuewei Niu <[email protected]>
> > > > > > > > Sent: Monday, May 19, 2025 3:08 PM
> > > > > > > >
> > > > > > > > > On Sat, Apr 12, 2025 at 10:28:25PM +0800, Xuewei Niu wrote:
> > > > > > > > > > This patch brings a new feature, called "multi devices",
> > > > > > > > > > to the virtio vsock. It introduces a
> > "VIRTIO_VSOCK_F_MULTI_DEVICES"
> > > > > > > > > > feature bit, and a "device_order" field to the config
> > > > > > > > > > for the virtio
> > > > vsock.
> > > > > > > > > >
> > > > > > > > > > == Motivition ==
> > > > > > > > > >
> > > > > > > > > > Vsock is a lightweight and widely used data exchange
> > > > > > > > > > mechanism between host and guest.
> > > > > > >
> > > > > > > Even though it is the current use, the specification does not
> > > > > > > prevent its usage
> > > > > > between two guests via a host.
> > > > > > > So we should not assume such guest <-> host communication as
> > > > > > > the only
> > > > > > case to add new feature.
> > > > > > >
> > > > > > > For example, in the spec only must requirement is that src_cid
> > > > > > > ==
> > > > > > config.guest_cid.
> > > > > > > Dst_cid can be anything, it need not be well known 0x2 (for the host).
> > > > > >
> > > > > > Yes.
> > > > > >
> > > > > > > With this flexibility in the spec, one can connect vsock
> > > > > > > devices with multiple
> > > > > > different backends.
> > > > > > >
> > > > > > > For example,
> > > > > > > QEMU can insert one vsock device for VM to HV communication.
> > > > > >
> > > > > > It is also able to communicate with other devices on the same host, i.e.
> > > > > > VM-to-VM.
> > > > > >
> > > > > > > A real PCI device can insert one voscket device for VM-to-VM
> > > > > > communication bypassing a full TCP/IP stack.
> > > > > >
> > > > > > AFAIK, all devices are implemented in software. Is it a real PCI HW device?
> > > > > >
> > > > > Yes. virto PCI devices are implemented as hw or as vdpa for many
> > > > > years now by
> > > > cloud operators and by NIC vendors.
> > > >
> > > > Thanks for your confirmation.
> > > >
> > > > > > > This means there are two different backends.
> > > > > > > And these two devices should not be grouped in the use case
> > > > > > > you
> > > > described.
> > > > > >
> > > > > > I think one group is enough for all use cases. It is required
> > > > > > that CIDs are unique in global, i.e. all backends.
> > > > > >
> > > > > > As your example, let me assume there are two VMs
> > > > > >
> > > > > > 1. VM0 (two vsock backends)
> > > > > >     1.1 device0 (cid=3, default), backend is host kernel (vhost-vsock);
> > > > > >     1.2 device1 (cid=4), backend is HV (virtio-vsock).
> > > > > > 2. VM1
> > > > > >     2.1 device2 (cid=5, default), backend is host kernel (vhost-vsock).
> > > > > >
> > > > > > The device0 is able to do VM0-HOST (src_cid=3, dst_cid=2) and
> > > > > > VM0-VM1 (src_cid=3, dst_cid=5) communicatation, while the
> > > > > > device1 is only able to do VM0-HV (src_cid=4, dst_cid=2)
> > communicatation.
> > > > > >
> > > > > In example 1.1 and 1.2 no devices are grouped.
> > > > > Your proposal of this patch wants to group the two devices and
> > > > > pick one of
> > > > them as default device.
> > > >
> > > > Yes. I just wonder if it is possible to have more than one groups in one guest?
> > > >
> > > For example
> > > Group_1: two devices dev0 and dev1, implemented as PCI HW devices.
> > > Group_2: two devices by QEMU SW implemented as sw backend.
> > >
> > > All the 4 devices has _F bit indicating they can be grouped.
> > > But there is no indication that they are part of which group.
> > > And hence the guest VM driver is in dark on how to forward requests without
> > the bind() call.
> > 
> > I see. Thanks!
> > 
> > My idea is that there is only one default device, no matter how many types of
> > backends are. If the users intend to use other devices, `bind()` call is required.
> > 
> > For example, we set `dev0` as the default device:
> > 
> > 1. Do not call `bind()`: use dev0;
> > 2. Call `bind(${dev0_cid})`: use dev1;
> > ...
> > 5. Call `bind(${dev4_cid})`: use dev4;
> > 
> > Even though we introduce the group concept, if we don't call `bind()`, how does
> > driver know which group to use? If the driver recoginizes the dst_cid, it can use
> > the group to find the device, then the things will be complicated. The driver
> > needs to know the relationship between the dst_cid and the group.
> >
> Based on the dst_cid picking the right vscock group would be needed. This is vsock level issue at driver level.
> Driver would need enough hints or encoding or of dst_cid or something else.
> 
> So even though we miss vsock level construct, it should be the reason to not group the devices.
> As both attempt to solve issue at different level.
> 
> > WDYT?
> > 
> > Thanks,
> > Xuewei
> > 
> > > > > If device0 and device1 are inserted to the VM0 with the feature
> > > > > bit you
> > > > suggested, the guest things that they are part of the same group.
> > > >
> > > > In this patch, we don't allow to insert devices without the feature
> > > > bit if there are already devices with the feature bit.
> > > >
> > > In above example of two groups, all the 4 devices spread across two groups has
> > the feature bit set.
> > > Yet, they cannot be grouped correctly.
> > > Driver driving blind thinks that all 4 devices are part of the same group.
> > >
> > > > As a result, there can be either multiple devices with the feature
> > > > bit or just a single device.
> > > >
> > >
> > > > > When bind() call is not done, host sw does not know which device
> > > > > to pick up
> > > > between device0 and device1 when binding the devices.
> > > >
> > > > Are you referring to an vsock application on the host? If yes, "host
> > > > sw" is able to pick up one device according to the dst cid. For
> > > > example, pick up
> > > > device0 if `connect(3)` is called.
> > > >
> > > > Please be aware that the "host sw" can not pick up device1, since
> > > > its device is not in the host kernel.
> > > >
> > > > > > In a word, a tuple identifies a connection.
> > > > > >
> > > > > > "Refuse to connect" will be raised if the device1 attempts to
> > > > > > connect to the device2. "They are not in the same group" is a
> > > > > > reasonable
> > > > explaination.
> > > > > > Am I right?
> > > > > >
> > > > > During bind call, one needs to select the device when the devices
> > > > > are coming
> > > > from multiple different backends.
> > > >
> > > > Yes.
> > > >
> > > > Thanks,
> > > > Xuewei