[SPDK] Re: The difference between vhost-nvme and vhost-blk

what i meat is that :
in vhost-user:
1. the guestOS should put an (vhost) request in the virtqueue 
2. then the spdk polling discover this request 
3. spdk should put an nvme request  in the actual device and knock the door bell.
this is two data(not the actual data but the request struct itself).


in vfio-user:
1. the guestOS put an (nvme) request struct in the DRAM region
2. the spdk discovers this and then what ? still needs to inform the nvme physical device  right? 
this is still two data copy(DMA) in the manner of nvme request struct.








At 2022-03-08 22:02:48, "Thanos Makatos" <thanos.makatos(a)nutanix.com> wrote:
>> -----Original Message-----
>> From: Liu, Xiaodong <xiaodong.liu(a)intel.com>
>> Sent: 08 March 2022 14:00
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> Subject: [SPDK] Re: The difference between vhost-nvme and vhost-blk
>> 
>> No, Vhost-user will just do once data DMA from Guest DRAM region to
>> device buffer.
>
>Same for vfio-user, the guest has shared its memory to SPDK so the physical device can access that memory directly.
>
>> 
>> -----Original Message-----
>> From: zbhhbz <zbhhbz(a)yeah.net>
>> Sent: Tuesday, March 8, 2022 9:54 PM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> Subject: [SPDK] Re: The difference between vhost-nvme and vhost-blk
>> 
>> 
>> 
>> 
>> ok, thanks,
>> one more follow up question:
>> isn't this leads to double DMA of data? one from guest to DRAM region and
>> one from DRAM region to device buffer.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> At 2022-03-08 21:31:44, "Thanos Makatos" <thanos.makatos(a)nutanix.com>
>> wrote:
>> >> -----Original Message-----
>> >> From: zbhhbz <zbhhbz(a)yeah.net>
>> >> Sent: 08 March 2022 13:20
>> >> To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> >> Subject: [SPDK] Re: The difference between vhost-nvme and vhost-blk
>> >>
>> >>
>> >>
>> >>
>> >> thanks, this helps a lot.
>> >>
>> >>
>> >> about the vfio-user,
>> >>
>> >>
>> >> I understand that in the vfio-user,
>> >> the guestOS can issue DMA read/write to a "pcie space " of a virtual
>> >> device, but I'm confused:
>> >> 1. does the guestOS issue DMA read/write to region on actual physical
>> >> device or just a DRAM region?
>> >
>> >A DRAM region.
>> >
>> >>      if the guestOS directly access the physical device(DMA,IOMMU),
>> >> where does spdk stands?
>> >> 2. why do vfio-user need a socket, what kind of data does the socket
>> carries?
>> >
>> >The vfio-user protocol allows a device to be emulated outside QEMU (the
>> vfio-user client), in a separate process (the vfio-user server, SPDK running
>> the nvmf/vfio-user target in our case). The UNIX domain socket is used
>> between QEMU and SPDK for initial device setup, virtual IRQs, and other
>> infrequent operations.
>> >
>> >> 3. Does the vfio-user look like the vhost-user except for direct DMA
>> >> access instead of shared memory communication?
>> >
>> >vfio-user allows any kind of device to be emulated in a separate process
>> (even non-PCI), while vhost-user is mainly for VirtIO devices.
>> >
>> >>
>> >>
>> >> thanks
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> At 2022-03-08 17:40:12, "Thanos Makatos"
>> <thanos.makatos(a)nutanix.com>
>> >> wrote:
>> >> >> -----Original Message-----
>> >> >> From: zbhhbz <zbhhbz(a)yeah.net>
>> >> >> Sent: 08 March 2022 09:29
>> >> >> To: spdk <spdk(a)lists.01.org>
>> >> >> Subject: [SPDK] Re: The difference between vhost-nvme and
>> >> >> vhost-blk
>> >> >>
>> >> >> thanks, follow up questions:
>> >> >> 1. if I still use Vhost-user-blk-pci in qemu and with nvme ssd(bdev), i
>> cant'
>> >> >> have access to the nvme feature in guest(shadow door bell). is
>> >> >> that
>> >> correct?
>> >> >
>> >> >Correct.
>> >> >
>> >> >> 2. in the vfio-user solution, does the interrupt sending from the
>> >> ssd(nvme)
>> >> >> go through the qemu/kvm? or it go straight to the guest kernel?
>> >> >
>> >> >It doesn't go to QEMU/KVM. It depends on how you've set it up in
>> >> >SPDK: it
>> >> can either go the host kernel or to SPDK.
>> >> >
>> >> >> 3. when will the vfio-user be available? does kvm have same delima
>> here?
>> >> >
>> >> >vfio-user is under review in QEMU, so we can't predict when it will
>> >> >be
>> >> accepted upstream. This doesn't mean you can't use it though, have a
>> >> look
>> >> here: https://urldefense.proofpoint.com/v2/url?u=https-
>> >> 3A__github.com_nutanix_libvfio-
>> >>
>> 2Duser_blob_master_docs_spdk.md&d=DwIGaQ&c=s883GpUCOChKOHiocY
>> >>
>> tGcg&r=XTpYsh5Ps2zJvtw6ogtti46atk736SI4vgsJiUKIyDE&m=5xzcHgA4AEwEk
>> >> wrWO3u9vZt5Fmg2QcvFmOxjMr4NL5Usz4o_2HIoCisBygQ9-
>> >> D1Q&s=k2t1I2xi9Och5KIeiaXyo2wo9wFbhIyALcMUph5xLK0&e=
>> >> >
>> >> >>
>> >> >>
>> >> >> thank you very much!
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> ---- 回复的原邮件 ----
>> >> >> | 发件人 | Liu, Changpeng<changpeng.liu(a)intel.com> |
>> >> >> | 日期 | 2022年03月08日 16:33 |
>> >> >> | 收件人 | Storage Performance Development Kit<spdk(a)lists.01.org>
>> |
>> >> >> | 抄送至 | |
>> >> >> | 主题 | [SPDK] Re: The difference between vhost-nvme and vhost-
>> blk
>> >> >> | |
>> >> >> Previously SPDK extended QEMU with a separate driver to enable
>> >> >> vhost- nvme, this driver was not accepted by QEMU, so now we
>> >> >> support emulated NVMe with a
>> >> new
>> >> >> solution
>> >> >> "vfio-user", again the driver for supporting this is still under
>> >> >> code review
>> >> of
>> >> >> QEMU community,
>> >> >> but SPDK already supports this.
>> >> >>
>> >> >> > -----Original Message-----
>> >> >> > From: zbhhbz(a)yeah.net <zbhhbz(a)yeah.net>
>> >> >> > Sent: Tuesday, March 8, 2022 4:29 PM
>> >> >> > To: spdk(a)lists.01.org
>> >> >> > Subject: [SPDK] The difference between vhost-nvme and vhost-blk
>> >> >> >
>> >> >> > Could someone help me understand the difference between vhost-
>> >> nvme
>> >> >> and
>> >> >> > vhost-blk?
>> >> >> > The online doc only shows there is vhost-user-blk-pci, why not
>> >> >> > vhost-
>> >> user-
>> >> >> nvme-
>> >> >> > pci?
>> >> >> > There is little doc fined in github/spdk and the qemu itself
>> >> >> > doesn't help
>> >> >> either
>> >> >> > Thanks!
>> >> >> > _______________________________________________
>> >> >> > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an
>> >> >> > email to spdk-leave(a)lists.01.org
>> >> >> _______________________________________________
>> >> >> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an
>> >> >> email to spdk-leave(a)lists.01.org
>> >> >> _______________________________________________
>> >> >> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an
>> >> >> email to spdk-leave(a)lists.01.org
>> >> >_______________________________________________
>> >> >SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email
>> >> >to spdk-leave(a)lists.01.org
>> >> _______________________________________________
>> >> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email
>> >> to spdk-leave(a)lists.01.org
>> >_______________________________________________
>> >SPDK mailing list -- spdk(a)lists.01.org
>> >To unsubscribe send an email to spdk-leave(a)lists.01.org
>> _______________________________________________
>> SPDK mailing list -- spdk(a)lists.01.org
>> To unsubscribe send an email to spdk-leave(a)lists.01.org
>> _______________________________________________
>> SPDK mailing list -- spdk(a)lists.01.org
>> To unsubscribe send an email to spdk-leave(a)lists.01.org
>_______________________________________________
>SPDK mailing list -- spdk(a)lists.01.org
>To unsubscribe send an email to spdk-leave(a)lists.01.org