[SPDK] Re: SPDK Thread and Nvmf QPair migration

Hi Hari,

About migrating a SPDK thread to a different system thread, we support spdk_set_cpumask() API now.
spdk_set_cpumask() requires the event framework layer to implment SPDK_THREAD_OP_RESCHED operation.

spdk_set_cpumask() is similar with Linux sched_setaffinity().

I hope lib/event/reactor.c will give some hint if you don't use reactor but implement your own scheduler.

Even if you use reactor, I admit there is few documentation about this feature. I want to write it though.
Besides, SPDK reactor not have dynamic SPDK thread scheduler yet.
So we have to use RPC and do manually SPDK thread migration.


About migrating a qair to a different poll group, NVMf layer does not support it.
One of the primary reason will be that mapping qpair to pollgroup is fixed for RDMA transport, e.g., due to Shared Receive Queue.

Thanks,
Shuhei

________________________________
差出人: Hari Kadayam <harihara.kadayam(a)gmail.com>
送信日時: 2020年5月17日 7:11
宛先: spdk(a)lists.01.org <spdk(a)lists.01.org>
件名: [SPDK] SPDK Thread and Nvmf QPair migration

We are trying to add SPDK to our existing async based framework and would like to do at spdk_thread level instead of reactor. I think we can run multiple spdk_threads in a single system thread by polling each thread in a round robin fashion. However, how do we move the spdk_thread into a different system thread, should we pause and pass the object to other thread and resume? What happens to the NVMf poll group which is associated in that thread. Are they movable to a different system thread? My understanding is that nvmf q-pair once associated needs to stick that poll group. Is that the same for system threads as well (meaning does it use any thread local variables) which stops them from moving to a different thread.

Thanks
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org