[SPDK] Re: vhost is destroyed many times when starting vhost device

I use QEMU: QEMU emulator version 4.2.92 (v5.0.0-rc2-32-g618b649d06-dirty)

Anyway, when there is more than 1 IO queue, the start/stop device will
be called many times.
If there is no way to reduce the start/stop device calls, I think that
reduce the bdev start/stop call is a good option.
Thanks,

Feng Li

Liu, Changpeng <changpeng.liu(a)intel.com> 于2020年4月26日周日 下午3:22写道：
>
> From the virtio specification, it's OK if the driver only use 1 IO queues, even you specified the num-queue to a bigger value.
> BIOS will only use 1 IO queue, so the vhost target still doesn't know when it can start the device.
>
> I tested with QEMU 4.1.50, I don't see the issue you mentioned the amplification factor is 2.
>
>
> > -----Original Message-----
> > From: Li Feng <fengli(a)smartx.com>
> > Sent: Sunday, April 26, 2020 1:17 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>; Kyle Zhang
> > <kyle(a)smartx.com>
> > Subject: [SPDK] Re: vhost is destroyed many times when starting vhost device
> >
> > Comment in-line.
> >
> > Thanks,
> >
> > Feng Li
> >
> > Liu, Changpeng <changpeng.liu(a)intel.com> 于2020年4月26日周日 上午
> > 8:46写道：
> >
> > >
> > > Hi Feng,
> > >
> > > The issue you mentioned does exist, but it's not related with number of disk,
> > but with "num-queues" parameter input from
> > > QEMU, in my test environment, if I used 8 IO queues, the vhost_blk_start() will
> > be called 9 times.  Note that the BIOS will
> > > use 1 IO queue so vhost_blk_start() is called 1 time during the BIOS stage.
> >
> > Thanks for your clarification of the relationship with num-queues.
> > In my previous test, the num-queues is equal to 4.
> > The amplification factor is nearly 2.
> >
> > Even set num-queues = 1, the stop/start device and
> > vhost_blk_start/stop is still been called 2 times.
> > The following log is about num-queues =1.
> > At the time of showing grub menu, the event is:
> > start_device
> > vhost_blk_start
> > io_setup
> > start_device
> > stop_device
> > vhost_blk_stop
> > start_device
> > vhost_blk_start
> > io_destroy
> > io_setup
> >
> > After starting the kernel:
> > stop_device
> > vhost_blk_stop
> > stop_device
> > io_destroy
> > start_device
> > vhost_blk_start
> > io_setup
> > stop_device
> > vhost_blk_stop
> > start_device
> > vhost_blk_start
> > io_destroy
> > io_setup
> > start_device
> >
> >
> > My qemu is master branch of upstream.
> >
> > >
> > > In the vhost target side, we don't know the exactly number of queues used for
> > this device, so we will start the device once
> > > got one valid IO queue, and restart the device when got a new IO queue.
> > That's the reason why the start/stop device will
> > > be called multiple times.
> >
> > Could the vhost target have a chance to implement the multi-queue
> > without start/stop the device?
> > Currently, when the io-queue num is up, the call times of the backend
> > bdev constructor/destructor will increase.
> >
> > What do you think my patch works?
> >
> > >
> > > > -----Original Message-----
> > > > From: Li Feng <fengli(a)smartx.com>
> > > > Sent: Thursday, April 23, 2020 8:30 PM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: [SPDK] vhost is destroyed many times when starting vhost device
> > > >
> > > > Hi,
> > > >
> > > > In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> > > > destroy_device when some conditions meet.
> > > >
> > > > I tested and found when a vhost-blk setup, the vhost device will be
> > > > destroyed 9 times.
> > > > And the bdev(no matter which type) will be affected. I'm using the aio
> > > > bdev, and the io_setup/io_destroy will be called 9 times when a disk
> > > > is inserted to a VM.
> > > >
> > > > My concern is:
> > > > Is there any solution to reduce the call times(at least: disks nums * 9)?
> > > > The call will slow down the VM boot time if VM has multiple disks.
> > > > There are some  related logs:
> > > >
> > > > Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> > > > [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> > > > --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> > > > --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> > > > --file-prefix=spdk_pid2004865 ]
> > > > EAL: No available hugepages reported in hugepages-1048576kB
> > > > app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> > > > reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > > >
> > > > Thanks,
> > > >
> > > > Feng Li
> > > >
> > > > --
> > > > The SmartX email address is only for business purpose. Any sent message
> > > > that is not related to the business is not authorized or permitted by
> > > > SmartX.
> > > > 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出
> > 的
> > > > 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
> > > >
> > > > _______________________________________________
> > > > SPDK mailing list -- spdk(a)lists.01.org
> > > > To unsubscribe send an email to spdk-leave(a)lists.01.org
> > > _______________________________________________
> > > SPDK mailing list -- spdk(a)lists.01.org
> > > To unsubscribe send an email to spdk-leave(a)lists.01.org
> >
> > --
> > The SmartX email address is only for business purpose. Any sent message
> > that is not related to the business is not authorized or permitted by
> > SmartX.
> > 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的
> > 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
> >
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org

-- 
The SmartX email address is only for business purpose. Any sent message 
that is not related to the business is not authorized or permitted by 
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.