[SPDK] Re: vhost is destroyed many times when starting vhost device

From the virtio specification, it's OK if the driver only use 1 IO queues, even you specified the num-queue to a bigger value.
BIOS will only use 1 IO queue, so the vhost target still doesn't know when it can start the device.

I tested with QEMU 4.1.50, I don't see the issue you mentioned the amplification factor is 2.


> -----Original Message-----
> From: Li Feng <fengli(a)smartx.com>
> Sent: Sunday, April 26, 2020 1:17 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Kyle Zhang
> <kyle(a)smartx.com>
> Subject: [SPDK] Re: vhost is destroyed many times when starting vhost device
> 
> Comment in-line.
> 
> Thanks,
> 
> Feng Li
> 
> Liu, Changpeng <changpeng.liu(a)intel.com> 于2020年4月26日周日 上午
> 8:46写道：
> 
> >
> > Hi Feng,
> >
> > The issue you mentioned does exist, but it's not related with number of disk,
> but with "num-queues" parameter input from
> > QEMU, in my test environment, if I used 8 IO queues, the vhost_blk_start() will
> be called 9 times.  Note that the BIOS will
> > use 1 IO queue so vhost_blk_start() is called 1 time during the BIOS stage.
> 
> Thanks for your clarification of the relationship with num-queues.
> In my previous test, the num-queues is equal to 4.
> The amplification factor is nearly 2.
> 
> Even set num-queues = 1, the stop/start device and
> vhost_blk_start/stop is still been called 2 times.
> The following log is about num-queues =1.
> At the time of showing grub menu, the event is:
> start_device
> vhost_blk_start
> io_setup
> start_device
> stop_device
> vhost_blk_stop
> start_device
> vhost_blk_start
> io_destroy
> io_setup
> 
> After starting the kernel:
> stop_device
> vhost_blk_stop
> stop_device
> io_destroy
> start_device
> vhost_blk_start
> io_setup
> stop_device
> vhost_blk_stop
> start_device
> vhost_blk_start
> io_destroy
> io_setup
> start_device
> 
> 
> My qemu is master branch of upstream.
> 
> >
> > In the vhost target side, we don't know the exactly number of queues used for
> this device, so we will start the device once
> > got one valid IO queue, and restart the device when got a new IO queue.
> That's the reason why the start/stop device will
> > be called multiple times.
> 
> Could the vhost target have a chance to implement the multi-queue
> without start/stop the device?
> Currently, when the io-queue num is up, the call times of the backend
> bdev constructor/destructor will increase.
> 
> What do you think my patch works?
> 
> >
> > > -----Original Message-----
> > > From: Li Feng <fengli(a)smartx.com>
> > > Sent: Thursday, April 23, 2020 8:30 PM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: [SPDK] vhost is destroyed many times when starting vhost device
> > >
> > > Hi,
> > >
> > > In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> > > destroy_device when some conditions meet.
> > >
> > > I tested and found when a vhost-blk setup, the vhost device will be
> > > destroyed 9 times.
> > > And the bdev(no matter which type) will be affected. I'm using the aio
> > > bdev, and the io_setup/io_destroy will be called 9 times when a disk
> > > is inserted to a VM.
> > >
> > > My concern is:
> > > Is there any solution to reduce the call times(at least: disks nums * 9)?
> > > The call will slow down the VM boot time if VM has multiple disks.
> > > There are some  related logs:
> > >
> > > Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> > > [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> > > --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> > > --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid2004865 ]
> > > EAL: No available hugepages reported in hugepages-1048576kB
> > > app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> > > reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > >
> > > Thanks,
> > >
> > > Feng Li
> > >
> > > --
> > > The SmartX email address is only for business purpose. Any sent message
> > > that is not related to the business is not authorized or permitted by
> > > SmartX.
> > > 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出
> 的
> > > 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
> > >
> > > _______________________________________________
> > > SPDK mailing list -- spdk(a)lists.01.org
> > > To unsubscribe send an email to spdk-leave(a)lists.01.org
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org
> 
> --
> The SmartX email address is only for business purpose. Any sent message
> that is not related to the business is not authorized or permitted by
> SmartX.
> 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的
> 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
> 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org