Re: [PATCH v7] virtio-vsock: Add support for multi devices
Jason Wang <[email protected]>
| Newsgroups | dev.linux.lists.virtio-comment |
|---|---|
| Message-ID | <CACGkMEscxnpdcFRkoLgow8qUKoKUSJM+j9Q1LG8HR5-aVVqxeg@mail.gmail.com> |
On Mon, Jun 23, 2025 at 5:47 PM Xuewei Niu <[email protected]> wrote: > > > On Thu, Jun 19, 2025 at 10:42 AM Xuewei Niu <[email protected]> wrote: > > > > > > > On Wed, Jun 18, 2025 at 5:51 PM Xuewei Niu <[email protected]> wrote: > > > > > > > > > > > On Wed, Jun 18, 2025 at 1:40 PM Xuewei Niu <[email protected]> wrote: > > > > > > > > > > > > > > > On Wed, Jun 18, 2025 at 10:47 AM Xuewei Niu <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > On Tue, Jun 17, 2025 at 3:46 PM Xuewei Niu <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > Resend, because it isn’t listed in the mailing list due to my mistake. > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 16, 2025 at 4:38 PM Stefano Garzarella <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 16 Jun 2025 at 10:29, Xuewei Niu <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 13, 2025 at 4:46 PM Xuewei Niu <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 13 Jun 2025 at 06:57, Xuewei Niu <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Apr 12, 2025 at 10:39 PM Xuewei Niu <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This patch brings a new feature, called "multi devices", to the virtio > > > > > > > > > > > > > > > > > > > > vsock. It introduces a "VIRTIO_VSOCK_F_MULTI_DEVICES" feature bit, and a > > > > > > > > > > > > > > > > > > > > "device_order" field to the config for the virtio vsock. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > == Motivition == > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Vsock is a lightweight and widely used data exchange mechanism between host > > > > > > > > > > > > > > > > > > > > and guest. Currently, the virtio-vsock only supports one device, resulting > > > > > > > > > > > > > > > > > > > > in the inability to enable more than one backend. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wonder which part of the spec forbids more than one device. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > No. The spec, however, is designed for a single device, and lacks some > > > > > > > > > > > > > > > > > > specifications for multiple devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example, we should have a mechanism to select a device from all to > > > > > > > > > > > > > > > > > > communicate with a peer. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wonder if this is a: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) mechanism that needs to be mandated by the device > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) a policy that is allowed to be tweaked by the user as TCP/IP did > > > > > > > > > > > > > > > > > > > > > > > > > > > > Not allowed in the current version. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (Note anyhow the driver can override what the device suggests...) > > > > > > > > > > > > > > > > > > > > > > > > > > I think we should follow what we described in the spec: > > > > > > > > > > > > > "The virtio socket device is a zero-configuration socket communications device." > > > > > > > > > > > > > > > > > > > > > > > > We probably need to define "configuration" first. > > > > > > > > > > > > > > > > > > > > > > > > For example, if it means zero configuration from the user, it does not > > > > > > > > > > > > conflict with 2), the driver can use its own algorithm to elect a > > > > > > > > > > > > "default" device. > > > > > > > > > > > > > > > > > > > > > > IMHO, it can be done, but it is not the current design. > > > > > > > > > > > > > > > > > > > > Well, you need at least explain the advantages or why you choose to do this. > > > > > > > > > > > > > > > > > > I listed in the previous message. Maybe it is not clear enough. I'll try to > > > > > > > > > explain it again. > > > > > > > > > > > > > > > > > > I think people should pick up the device by a `bind()` call, which takes a > > > > > > > > > CID as an argument. > > > > > > > > > > > > > > > > > > Generally, the device is picked up by the source CID, which is achieved > > > > > > > > > through a `bind()` call. > > > > > > > > > > > > > > > > > > The default device, which is equivalent to the current single device, is > > > > > > > > > used to be compatible with the existing applications. > > > > > > > > > > > > > > > > > > Apart from that, the default device is used to communicate with hypervisor > > > > > > > > > for some init works, such as gathering information about other vsock > > > > > > > > > devices. > > > > > > > > > > > > > > > > > > To summarize, users must do `bind()` call explicitly to select the desired > > > > > > > > > device for non-HV-VM communications. > > > > > > > > > > > > > > > > So if I understand correctly you need a way to select the default when > > > > > > > > bind() is not called? > > > > > > > > > > > > > > Yes. > > > > > > > > > > > > > > > > > > I prefer to set the default device for VM-HV communication to do some init > > > > > > > > > > > work. I don't think people have a strong need for this. > > > > > > > > > > > > > > > > > > > > > > > Another perspective, making decisions in guests may be even more > > > > > > > > > > > > helpful for the case where the device is not trusted > > > > > > > > > > > > > > > > > > > > > > How does the guest realize the vsock device is not trusted? > > > > > > > > > > > > > > > > > > > > There're various ways to build trust (for example device attestation) > > > > > > > > > > and more might come in the future. > > > > > > > > > > > > > > > > > > > > > The guest only > > > > > > > > > > > knows the information from its config space, which is provided by the host. > > > > > > > > > > > > > > > > > > > > The way to build trust is probably beyond the scope of virtio, but it > > > > > > > > > > is something we need to consider. > > > > > > > > > > > > > > > > > > Agree with you. If it comes in the future, I think the driver should have > > > > > > > > > the ability to make decisions. > > > > > > > > > > > > > > > > > > > > > > > > Actually, I meant the way to build trust via virtio is something that > > > > > > > > needs to be considered. But now we have other ways to build trust. > > > > > > > > That would result a situlation: > > > > > > > > > > > > > > > > 1) "default" vsock device is not trusted but other might > > > > > > > > > > > > > > I think this topic might be beyond the scope of this patch. > > > > > > > > > > > > > > With the current version, there is only one device supported, which can be > > > > > > > considered as the "default". We don't have a mechanism to say "we don't > > > > > > > trust you", right? > > > > > > > > > > > > No. I meant we don't have it in the virtio core but we already have it > > > > > > in other layers (for example the transport layer). > > > > > > > > > > Are you referring to virito-vsock transport layer? > > > > > > > > Yes, for example the PCI layer. > > > > > > > > > > > > > > > > That is, it is assumed that we trust the device provided > > > > > > > by the hypervisor. > > > > > > > > > > > > > > This patch is for multiple devices support, based on the same assumption. I > > > > > > > think trust is a good point to consider, but perhaps we have to address it > > > > > > > in the follow-up patches. > > > > > > > > > > > > My point is not about how to build trust, it's about letting the > > > > > > driver decide by itself in some cases. > > > > > > > > > > > > A better way might be something like: > > > > > > > > > > > > "The device_order is a hint for the driver to select a default vsock > > > > > > device. Device MAY choose ...." > > > > > > > > > > Okay, I'll do that. > > > > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > 2) two device claims that they are all "default" > > > > > > > > > > > > > > > > This means anyhow we need a decision from the driver side so the > > > > > > > > device side order seems to be useless here. > > > > > > > > > > > > > > There is only one default device. The device with the lowest device_order > > > > > > > is considered the default. It is not allowed to have the same device_order. > > > > > > > > > > > > Who can forbid two same device_order? Note that in various security > > > > > > models, hypervisors are not trusted at all. > > > > > > > > > > The driver does. Indeed, the driver is able to deny devices if they violate > > > > > the spec. > > > > > > > > Yes, that's the point, anyhow driver need to do the decision. > > > > > > > > > > > > > > > > > > > > > > So, IMO the guest (driver) should not be allowed to change anything. > > > > > > > > > > > > > E.g. Right now it's not allowed to change the CID assigned by the host (device). > > > > > > > > > > > > > > > > > > > > > > > > A dumb question when having two cids (cid1 and cid2) in the same > > > > > > > > > > > > guest, what happens if src=cid1 and dst=cid2? > > > > > > > > > > > > > > > > > > > > > > The packets will be directed to another application, if any, on the same guest. > > > > > > > > > > > > > > > > > > > > Ok, so in your example, vhost-user-vsock should route those packets > > > > > > > > > > back to kernel vsock? > > > > > > > > > > > > > > > > > > It depends. Two cases are possible: > > > > > > > > > > > > > > > > > > 1. dev0(cid1) and dev1(cid2) are from the same type of backend, e.g. > > > > > > > > > vhost-vsock, so the packets will be routed back as you described. > > > > > > > > > 2. dev0(cid1) from vhost-user-vsock, and dev1(cid2) from vhost-vsock, the > > > > > > > > > packets will be dropped if vhost-user-vsock device can't find a device > > > > > > > > > whose cid is cid2. > > > > > > > > > > > > > > > > Well, this means the behavior depends on the implementation which is not good. > > > > > > > > > > > > > > No, it is not. It depends on whether the device can find a proper > > > > > > > target device based on the dst_cid. > > > > > > > > > > > > This sounds really weird, two cids belong to the same guest. So guests > > > > > > should expect that the two vsock devic can talk to each other? > > > > > > > > > > Yes, a little bit complicated. > > > > > > > > > > Two constraints should be applied: > > > > > > > > > > 1. No CID conflicts within the driver; > > > > > 2. No CID conflicts within the address space. > > > > > > > > > > Here is a diagram to illustrate the situation where it does not violate the > > > > > constraints: > > > > > > > > > > ┌─ kernel─(as0)────────────────────────────────────────────────────┐ > > > > > │ ┌───────────┐ ┌───────────┐ ┌───────────┐│ > > > > > │ │ dev0(cid0)│ │ dev2(cid1)│ │ dev3(cid2)││ > > > > > │ └───┬───────┘ └───┬───────┘ └────────┬──┘│ > > > > > └───────┼──────────────────────────────────┼───────────────────┼───┘ > > > > > ┌──────┼───────────────────────┐ ┌──────┼───────────────────┼───┐ > > > > > │ │ ┌───────────┐ │ │ │ ┌───────────┐ │ │ > > > > > │ └──► dev0(cid0)│ │ │ └──► dev2(cid1)│ │ │ > > > > > │ └───────────┘ │ │ └───────────┘ │ │ > > > > > │ ┌───────────┐ │ │ ┌───────────┐ │ │ > > > > > │ ┌──► dev1(cid1)│ │ │ │ dev3(cid2)◄────┘ │ > > > > > │ │ └───────────┘ VM0│ │ └───────────┘ VM1│ > > > > > └──────┼───────────────────────┘ └──────────────────────────────┘ > > > > > vhost-user-vsock > > > > > ┌────────┼──────────────────────────────────────────────────────────┐ > > > > > │ ┌────┼──────┐ │ > > > > > │ │ dev1(cid1)│ │ > > > > > │ └───────────┘ │ > > > > > └─userapp─(as1)─────────────────────────────────────────────────────┘ > > > > > > > > > > - VM0 > > > > > - dev0 > > > > > - dst_cid = 2 (well-known cid): connect to host; > > > > > - dst_cid = cid1: connect to dev2 (they are in the same as0); > > > > > - dst_cid = cid2: connect to dev3; > > > > > - dev1 > > > > > - dst_cid = 2 (well-known cid): connect to userapp; > > > > > - dst_cid = cid0: failure (no cid0 is available in as1, even though > > > > > cid1 is available in the VM0); > > > > > - dst_cid = cid1: connect to dev0; > > > > > - VM1 > > > > > - dev2 > > > > > - dst_cid = 2 (well-known cid): connect to host; > > > > > - dst_cid = cid0: connect to dev0; > > > > > - dst_cid = cid2: connect to dev3; > > > > > - dev3: skip the same as dev2. > > > > > > > > > > So back to your question, my answer is that it depends on the address > > > > > space. Hope it could be helpful. > > > > > > > > This brings an interesting question, for example if vm0 tries to > > > > connect to vm1, how does it know which device it needs to use (lacking > > > > the concept like switch/route/address announcing etc...)? > > > > > > The HV should maintain a table for that. The guest things firstly > > > communicate with the HV, through the default device, to know which device > > > to use. > > > > It's still not clear to me how things work. For example, we had a > > guest1 with two cid 4 ("default"),5 another guest2 with one cid 6. You > > meant guest1 needs to ask the host to know about which device is > > connected to 6? Or actually any device in guest1 can be used to > > connected to guest2? > > I think it is application-level, like micro services? Here are some > diagrams to illustrate the process: > > Step1: app0 asks the host: "I want to connect to app1 inside VM1"; > Step2: host replies: "Please use source cid=5, and destination cid=6" This sounds like arp anyhow. > > +────────+ +───────────────────────────────+ +──────────────────────────────+ > │ host │ │+───────────────+ VM0 │ │ VM1 │ > │service │◀─step1──┤│dev0(cid=4,def)│──┐ │ │ │ > +────────+ │+───────────────+ │ │ │ │ > │+───────────────+ │ +───────+│ │+───────────────+ +──────+│ > ││ dev1(cid=5) │ └──│ app0 ││ ││dev1(cid=6,def)│ │ app1 ││ > │+───────────────+ +───────+│ │+───────────────+ +──────+│ > +───────────────────────────────+ +──────────────────────────────+ > > Step3: app0 binds (5, -1), and connects to (6, {PORT}) to establish a > connection. > > +────────+ +───────────────────────────────+ +──────────────────────────────+ > │ host │ │+───────────────+ VM0 │ │ VM1 │ > │service │ ││dev0(cid=4,def)│ │ │ │ > +────────+ │+───────────────+ │ │ │ > │+──────────────bind(5, -1)────+│ │+───────────────+ +──────+│ > ┌─┤│ dev1(cid=5) │◀────│ app0 ││ ││dev1(cid=6,def)│─────▶ app1 ││ > │ │+───────────────+ +───────+│ │+───────────────+ +──────+│ > │ +───────────────────────────────+ +────────▲─────────────────────+ > │ │ > └───────────────connect(6,PORT)─────────────────┘ > > This is a little bit complicated, but it is dynamic. Another way is to > assign a fixed CID for app1, Who did the assignment here? > so app0 can always connect to it without > needing to ask for the host service. Basically, I wonder if the above needs to be part of the spec or not and why. If not, we should not bother here. Thanks > > Thanks, > Xuewei > > > Thanks > > > > > > > > Thanks, > > > Xuewei > > > >