[SPDK] Re: SPDK Thread and Nvmf QPair migration

Hi Hari,

My work on dynamic SPDK thread scheduling has been suspended due to other higher priority works,
but I will restart it once other works have been done.
I will restart it from automatic SPDK thread scheduler by using the current SPDK thread statistics.
The target date is not clear yet.

Of course it's very welcome if anyone can do it.

Ben may provide better feedback.

Thanks,
Shuhei


________________________________
差出人: 松本周平 / MATSUMOTO，SHUUHEI <shuhei.matsumoto.xt(a)hitachi.com>
送信日時: 2020年5月18日 7:51
宛先: spdk(a)lists.01.org <spdk(a)lists.01.org>
件名: Re: [SPDK] SPDK Thread and Nvmf QPair migration

Hi Hari,

About migrating a SPDK thread to a different system thread, we support spdk_set_cpumask() API now.
spdk_set_cpumask() requires the event framework layer to implment SPDK_THREAD_OP_RESCHED operation.

spdk_set_cpumask() is similar with Linux sched_setaffinity().

I hope lib/event/reactor.c will give some hint if you don't use reactor but implement your own scheduler.

Even if you use reactor, I admit there is few documentation about this feature. I want to write it though.
Besides, SPDK reactor not have dynamic SPDK thread scheduler yet.
So we have to use RPC and do manually SPDK thread migration.


About migrating a qair to a different poll group, NVMf layer does not support it.
One of the primary reason will be that mapping qpair to pollgroup is fixed for RDMA transport, e.g., due to Shared Receive Queue.

Thanks,
Shuhei

________________________________
差出人: Hari Kadayam <harihara.kadayam(a)gmail.com>
送信日時: 2020年5月17日 7:11
宛先: spdk(a)lists.01.org <spdk(a)lists.01.org>
件名: [SPDK] SPDK Thread and Nvmf QPair migration

We are trying to add SPDK to our existing async based framework and would like to do at spdk_thread level instead of reactor. I think we can run multiple spdk_threads in a single system thread by polling each thread in a round robin fashion. However, how do we move the spdk_thread into a different system thread, should we pause and pass the object to other thread and resume? What happens to the NVMf poll group which is associated in that thread. Are they movable to a different system thread? My understanding is that nvmf q-pair once associated needs to stick that poll group. Is that the same for system threads as well (meaning does it use any thread local variables) which stops them from moving to a different thread.

Thanks
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org