[SPDK] Re: vhost is destroyed many times when starting vhost device

Comment in-line.

Thanks,

Feng Li

Liu, Changpeng <changpeng.liu(a)intel.com> 于2020年4月26日周日 上午8:46写道：

>
> Hi Feng,
>
> The issue you mentioned does exist, but it's not related with number of disk, but with "num-queues" parameter input from
> QEMU, in my test environment, if I used 8 IO queues, the vhost_blk_start() will be called 9 times.  Note that the BIOS will
> use 1 IO queue so vhost_blk_start() is called 1 time during the BIOS stage.

Thanks for your clarification of the relationship with num-queues.
In my previous test, the num-queues is equal to 4.
The amplification factor is nearly 2.

Even set num-queues = 1, the stop/start device and
vhost_blk_start/stop is still been called 2 times.
The following log is about num-queues =1.
At the time of showing grub menu, the event is:
start_device
vhost_blk_start
io_setup
start_device
stop_device
vhost_blk_stop
start_device
vhost_blk_start
io_destroy
io_setup

After starting the kernel:
stop_device
vhost_blk_stop
stop_device
io_destroy
start_device
vhost_blk_start
io_setup
stop_device
vhost_blk_stop
start_device
vhost_blk_start
io_destroy
io_setup
start_device


My qemu is master branch of upstream.

>
> In the vhost target side, we don't know the exactly number of queues used for this device, so we will start the device once
> got one valid IO queue, and restart the device when got a new IO queue.  That's the reason why the start/stop device will
> be called multiple times.

Could the vhost target have a chance to implement the multi-queue
without start/stop the device?
Currently, when the io-queue num is up, the call times of the backend
bdev constructor/destructor will increase.

What do you think my patch works?

>
> > -----Original Message-----
> > From: Li Feng <fengli(a)smartx.com>
> > Sent: Thursday, April 23, 2020 8:30 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] vhost is destroyed many times when starting vhost device
> >
> > Hi,
> >
> > In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> > destroy_device when some conditions meet.
> >
> > I tested and found when a vhost-blk setup, the vhost device will be
> > destroyed 9 times.
> > And the bdev(no matter which type) will be affected. I'm using the aio
> > bdev, and the io_setup/io_destroy will be called 9 times when a disk
> > is inserted to a VM.
> >
> > My concern is:
> > Is there any solution to reduce the call times(at least: disks nums * 9)?
> > The call will slow down the VM boot time if VM has multiple disks.
> > There are some  related logs:
> >
> > Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> > [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> > --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> > --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> > --file-prefix=spdk_pid2004865 ]
> > EAL: No available hugepages reported in hugepages-1048576kB
> > app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> > reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*:  call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*:  call io_setup
> >
> > Thanks,
> >
> > Feng Li
> >
> > --
> > The SmartX email address is only for business purpose. Any sent message
> > that is not related to the business is not authorized or permitted by
> > SmartX.
> > 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的
> > 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
> >
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org

-- 
The SmartX email address is only for business purpose. Any sent message 
that is not related to the business is not authorized or permitted by 
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.