Re: [PATCH v7] virtio-vsock: Add support for multi devices

> > From: Xuewei Niu <[email protected]>
> > Sent: 19 June 2025 10:41 AM
> > 
> > > > From: Xuewei Niu <[email protected]>
> > > > Sent: 19 June 2025 08:57 AM
> > > >
> > > > Hi Parav,
> > > >
> > > > Could you please take a look at the diagram in [1]?
> > > >
> > > > IIUC, for VM0, there are two groups, and for VM1, there is only one group.
> > > > Am I right? If yes, I think the group concept is reasonable but we
> > > > don't need at this time.
> > > >
> > > > I think the first thing is to figure out how to pick the right group.
> > > >
> > > > Standard socket doesn't provide a way to access the group information.
> > > >
> > > > Source and destination are from `bind()` and `connect()`,
> > > > respectively. If we don't call `bind()`, only the destination is known.
> > > >
> > > > However, only destination is not enough to find the group. For
> > > > example, the well-known CIDs (e.g. 2) are valid for all groups.
> > > >
> > > > 1:
> > > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flo
> > > > re.kernel.org%2Fvirtio-comment%2F20250618095139.1412138-1-
> > niuxuewei.
> > > >
> > nxw%40antgroup.com%2F&data=05%7C02%7Cparav%40nvidia.com%7C9d4a
> > 762f95
> > > >
> > 49433be00208ddaeefa4ba%7C43083d15727340c1b7db39efd9ccc17a%7C0%7
> > C0%7C
> > > >
> > 638859066500269242%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOn
> > RydWUsI
> > > >
> > lYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D
> > %
> > > >
> > 7C0%7C%7C%7C&sdata=ANv4q7h1Z1Jw5AyypDRE9brIbeyt4siy4g4hiYf8Lis%3
> > D&re
> > > > served=0
> > >
> > > Right. Sock addressing scheme is naïve presently to select the group. Not
> > sure when/how you or others plan to do.
> > > This is transport layer problem to solve (not to confuse with transport =
> > pci/mmio etc).
> > >
> > > However, at device level, we should have the construct of grouping.
> > > Without this construct, all devices will be part of single group and one will
> > not be able to build the group concept later.
> > > So even if you don't need it explicitly now, grouping the device is what you
> > need when connect() is called.
> > >
> > > So I was imagining a relatively simple scheme:
> > > For example, virtio device level, some kind of group id is present.
> > > So two devices which has same group id, are part of single group.
> > > An example group id format can be a UUID.
> > >
> > > And this is completely optional for devices to implement.
> > > Generic enough and usable beyond just vsock device in other use cases we
> > discussed in past.
> > 
> > Fair enough.
> > 
> > @Stefano, could you please take a look at this? I'd love to have some input
> > from you.
> > 
> > A brief summary of the idea is: The config space will be extended to include a
> > group id. 
> UUIDs are long even though they are read only.
> And config space is readable only after feature bits are negotiated.
> 
> I didn't think enough if the driver needs to know early enough when creating the 'struct virtio_device' with two legs of virtio_pci_dev * in it.
> Need some more thoughts on it.

I think it is okay to do so. It is used in socket layer. By that time, the
driver will be able to access the config space.

Thanks,
Xuewei

> > The devices with the same group id are considered to be in the
> > same group.
> >
> Sounds good.
>  
> > Thanks,
> > Xuewei
> > 
> > > >
> > > > > > From: Xuewei Niu <[email protected]>
> > > > > > Sent: Monday, June 16, 2025 4:26 PM
> > > > > >
> > > > > > > > From: Xuewei Niu <[email protected]>
> > > > > > > > Sent: Monday, June 16, 2025 2:30 PM
> > > > > > > >
> > > > > > > > > > From: Xuewei Niu <[email protected]>
> > > > > > > > > > Sent: Monday, June 16, 2025 1:48 PM
> > > > > > > > > >
> > > > > > > > > > Hi, Parav.
> > > > > > > > > >
> > > > > > > > > > Thanks for your detailed comments.
> > > > > > > > > >
> > > > > > > > > > > Hi Xuewei,
> > > > > > > > > > >
> > > > > > > > > > > > From: Xuewei Niu <[email protected]>
> > > > > > > > > > > > Sent: Monday, May 19, 2025 3:08 PM
> > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Apr 12, 2025 at 10:28:25PM +0800, Xuewei Niu
> > wrote:
> > > > > > > > > > > > > > This patch brings a new feature, called "multi
> > > > > > > > > > > > > > devices", to the virtio vsock. It introduces a
> > > > > > "VIRTIO_VSOCK_F_MULTI_DEVICES"
> > > > > > > > > > > > > > feature bit, and a "device_order" field to the
> > > > > > > > > > > > > > config for the virtio
> > > > > > > > vsock.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > == Motivition ==
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Vsock is a lightweight and widely used data
> > > > > > > > > > > > > > exchange mechanism between host and guest.
> > > > > > > > > > >
> > > > > > > > > > > Even though it is the current use, the specification
> > > > > > > > > > > does not prevent its usage
> > > > > > > > > > between two guests via a host.
> > > > > > > > > > > So we should not assume such guest <-> host
> > > > > > > > > > > communication as the only
> > > > > > > > > > case to add new feature.
> > > > > > > > > > >
> > > > > > > > > > > For example, in the spec only must requirement is that
> > > > > > > > > > > src_cid ==
> > > > > > > > > > config.guest_cid.
> > > > > > > > > > > Dst_cid can be anything, it need not be well known 0x2
> > > > > > > > > > > (for the
> > > > host).
> > > > > > > > > >
> > > > > > > > > > Yes.
> > > > > > > > > >
> > > > > > > > > > > With this flexibility in the spec, one can connect
> > > > > > > > > > > vsock devices with multiple
> > > > > > > > > > different backends.
> > > > > > > > > > >
> > > > > > > > > > > For example,
> > > > > > > > > > > QEMU can insert one vsock device for VM to HV
> > communication.
> > > > > > > > > >
> > > > > > > > > > It is also able to communicate with other devices on the
> > > > > > > > > > same host,
> > > > i.e.
> > > > > > > > > > VM-to-VM.
> > > > > > > > > >
> > > > > > > > > > > A real PCI device can insert one voscket device for
> > > > > > > > > > > VM-to-VM
> > > > > > > > > > communication bypassing a full TCP/IP stack.
> > > > > > > > > >
> > > > > > > > > > AFAIK, all devices are implemented in software. Is it a
> > > > > > > > > > real PCI HW
> > > > device?
> > > > > > > > > >
> > > > > > > > > Yes. virto PCI devices are implemented as hw or as vdpa
> > > > > > > > > for many years now by
> > > > > > > > cloud operators and by NIC vendors.
> > > > > > > >
> > > > > > > > Thanks for your confirmation.
> > > > > > > >
> > > > > > > > > > > This means there are two different backends.
> > > > > > > > > > > And these two devices should not be grouped in the use
> > > > > > > > > > > case you
> > > > > > > > described.
> > > > > > > > > >
> > > > > > > > > > I think one group is enough for all use cases. It is
> > > > > > > > > > required that CIDs are unique in global, i.e. all backends.
> > > > > > > > > >
> > > > > > > > > > As your example, let me assume there are two VMs
> > > > > > > > > >
> > > > > > > > > > 1. VM0 (two vsock backends)
> > > > > > > > > >     1.1 device0 (cid=3, default), backend is host kernel (vhost-
> > vsock);
> > > > > > > > > >     1.2 device1 (cid=4), backend is HV (virtio-vsock).
> > > > > > > > > > 2. VM1
> > > > > > > > > >     2.1 device2 (cid=5, default), backend is host kernel (vhost-
> > vsock).
> > > > > > > > > >
> > > > > > > > > > The device0 is able to do VM0-HOST (src_cid=3,
> > > > > > > > > > dst_cid=2) and
> > > > > > > > > > VM0-VM1 (src_cid=3, dst_cid=5) communicatation, while
> > > > > > > > > > the
> > > > > > > > > > device1 is only able to do VM0-HV (src_cid=4, dst_cid=2)
> > > > > > communicatation.
> > > > > > > > > >
> > > > > > > > > In example 1.1 and 1.2 no devices are grouped.
> > > > > > > > > Your proposal of this patch wants to group the two devices
> > > > > > > > > and pick one of
> > > > > > > > them as default device.
> > > > > > > >
> > > > > > > > Yes. I just wonder if it is possible to have more than one
> > > > > > > > groups in one
> > > > guest?
> > > > > > > >
> > > > > > > For example
> > > > > > > Group_1: two devices dev0 and dev1, implemented as PCI HW
> > devices.
> > > > > > > Group_2: two devices by QEMU SW implemented as sw backend.
> > > > > > >
> > > > > > > All the 4 devices has _F bit indicating they can be grouped.
> > > > > > > But there is no indication that they are part of which group.
> > > > > > > And hence the guest VM driver is in dark on how to forward
> > > > > > > requests without
> > > > > > the bind() call.
> > > > > >
> > > > > > I see. Thanks!
> > > > > >
> > > > > > My idea is that there is only one default device, no matter how
> > > > > > many types of backends are. If the users intend to use other
> > > > > > devices, `bind()` call
> > > > is required.
> > > > > >
> > > > > > For example, we set `dev0` as the default device:
> > > > > >
> > > > > > 1. Do not call `bind()`: use dev0; 2. Call `bind(${dev0_cid})`:
> > > > > > use dev1; ...
> > > > > > 5. Call `bind(${dev4_cid})`: use dev4;
> > > > > >
> > > > > > Even though we introduce the group concept, if we don't call
> > > > > > `bind()`, how does driver know which group to use? If the driver
> > > > > > recoginizes the dst_cid, it can use the group to find the
> > > > > > device, then the things will be complicated. The driver needs to
> > > > > > know the
> > > > relationship between the dst_cid and the group.
> > > > > >
> > > > > Based on the dst_cid picking the right vscock group would be
> > > > > needed. This
> > > > is vsock level issue at driver level.
> > > > > Driver would need enough hints or encoding or of dst_cid or
> > > > > something
> > > > else.
> > > > >
> > > > > So even though we miss vsock level construct, it should be the
> > > > > reason to not
> > > > group the devices.
> > > > > As both attempt to solve issue at different level.
> > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Thanks,
> > > > > > Xuewei
> > > > > >
> > > > > > > > > If device0 and device1 are inserted to the VM0 with the
> > > > > > > > > feature bit you
> > > > > > > > suggested, the guest things that they are part of the same group.
> > > > > > > >
> > > > > > > > In this patch, we don't allow to insert devices without the
> > > > > > > > feature bit if there are already devices with the feature bit.
> > > > > > > >
> > > > > > > In above example of two groups, all the 4 devices spread
> > > > > > > across two groups has
> > > > > > the feature bit set.
> > > > > > > Yet, they cannot be grouped correctly.
> > > > > > > Driver driving blind thinks that all 4 devices are part of the same
> > group.
> > > > > > >
> > > > > > > > As a result, there can be either multiple devices with the
> > > > > > > > feature bit or just a single device.
> > > > > > > >
> > > > > > >
> > > > > > > > > When bind() call is not done, host sw does not know which
> > > > > > > > > device to pick up
> > > > > > > > between device0 and device1 when binding the devices.
> > > > > > > >
> > > > > > > > Are you referring to an vsock application on the host? If
> > > > > > > > yes, "host sw" is able to pick up one device according to
> > > > > > > > the dst cid. For example, pick up
> > > > > > > > device0 if `connect(3)` is called.
> > > > > > > >
> > > > > > > > Please be aware that the "host sw" can not pick up device1,
> > > > > > > > since its device is not in the host kernel.
> > > > > > > >
> > > > > > > > > > In a word, a tuple identifies a connection.
> > > > > > > > > >
> > > > > > > > > > "Refuse to connect" will be raised if the device1
> > > > > > > > > > attempts to connect to the device2. "They are not in the same
> > group"
> > > > > > > > > > is a reasonable
> > > > > > > > explaination.
> > > > > > > > > > Am I right?
> > > > > > > > > >
> > > > > > > > > During bind call, one needs to select the device when the
> > > > > > > > > devices are coming
> > > > > > > > from multiple different backends.
> > > > > > > >
> > > > > > > > Yes.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Xuewei