[SPDK] Re: thoughts of vhost-user disks + RDMA

Hi Shuhei,
Thanks for your replay.

The link gives an option to increase the memory usage, that could delay
the  hang of calling 'ibv_reg_mr',
Maybe I need a better solution that cut off the big memory map, replaced
with a smaller region. Looks like it's hard.

Thanks,
Feng Li


松本周平 / MATSUMOTO，SHUUHEI <shuhei.matsumoto.xt(a)hitachi.com> 于2020年2月17日周一
上午6:43写道：

> Hi Feng,
>
> Have you read the following link?
>
> https://community.mellanox.com/s/article/howto-increase-memory-size-used-by-mellanox-adapters
> https://www.rdmamojo.com/2012/09/07/ibv_reg_mr/
>
> You might already tried though.
>
> I'm non-experts and please wait until experts start this week 🙂
>
> Thanks,
> Shuhei
>
>
> ________________________________
> 差出人: Li Feng <fengli(a)smartx.com>
> 送信日時: 2020年2月16日 17:35
> 宛先: Storage Performance Development Kit <spdk(a)lists.01.org>; Kyle Zhang <
> kyle(a)smartx.com>
> 件名: [SPDK] thoughts of vhost-user disks + RDMA
>
> Hi experts,
>
> I have questions about getting data from the vhost-user and sending
> out through RDMA with zero-copy.
>
> When using vhost-user-blk/scsi, all VM's memory range is mapped to target.
>
> For example, if a VM has a very big memory, like 100G, and it connects
> to 10 vhost-user-blk disks.
> Then the vhost-user target needs mmap 10*100G virtual memory.
> If the data needs to transfer to another host through RDMA, and don't
> want to copy the data out of vhost-user, you should register the all
> virtual memory of vhost-user as RDMA MR at initialization. Because you
> don't know the virtio data is put in which memory range, and only
> register part of virtual memory.  right?
> There will have so big range of memory that needs to be registered.
> With some tests, the `ibv_reg_mr` call will make the host hang when
> registering serval TB virtual memory.
>
>
> If registering MR at the IO path, the performance is not acceptable.
>
> Any ideas about this issue if don't want to copy data?
>
> Thanks,
>
> Feng Li
>
> --
> The SmartX email address is only for business purpose. Any sent message
> that is not related to the business is not authorized or permitted by
> SmartX.
> 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
>
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>

-- 
The SmartX email address is only for business purpose. Any sent message 
that is not related to the business is not authorized or permitted by 
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.