[SPDK] Re: thoughts of vhost-user disks + RDMA

Hi Feng,

Have you read the following link?
https://community.mellanox.com/s/article/howto-increase-memory-size-used-by-mellanox-adapters
https://www.rdmamojo.com/2012/09/07/ibv_reg_mr/

You might already tried though.

I'm non-experts and please wait until experts start this week 🙂

Thanks,
Shuhei


________________________________
差出人: Li Feng <fengli(a)smartx.com>
送信日時: 2020年2月16日 17:35
宛先: Storage Performance Development Kit <spdk(a)lists.01.org>; Kyle Zhang <kyle(a)smartx.com>
件名: [SPDK] thoughts of vhost-user disks + RDMA

Hi experts,

I have questions about getting data from the vhost-user and sending
out through RDMA with zero-copy.

When using vhost-user-blk/scsi, all VM's memory range is mapped to target.

For example, if a VM has a very big memory, like 100G, and it connects
to 10 vhost-user-blk disks.
Then the vhost-user target needs mmap 10*100G virtual memory.
If the data needs to transfer to another host through RDMA, and don't
want to copy the data out of vhost-user, you should register the all
virtual memory of vhost-user as RDMA MR at initialization. Because you
don't know the virtio data is put in which memory range, and only
register part of virtual memory.  right?
There will have so big range of memory that needs to be registered.
With some tests, the `ibv_reg_mr` call will make the host hang when
registering serval TB virtual memory.


If registering MR at the IO path, the performance is not acceptable.

Any ideas about this issue if don't want to copy data?

Thanks,

Feng Li

--
The SmartX email address is only for business purpose. Any sent message
that is not related to the business is not authorized or permitted by
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org