[SPDK] Re: SPDK Thread and Nvmf QPair migration

Hi Hari,

> -----Original Message-----
> From: Hari Kadayam <harihara.kadayam(a)gmail.com>
> Sent: Saturday, May 16, 2020 3:11 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] SPDK Thread and Nvmf QPair migration
> 
> We are trying to add SPDK to our existing async based framework and would like
> to do at spdk_thread level instead of reactor. I think we can run multiple
> spdk_threads in a single system thread by polling each thread in a round robin
> fashion. However, how do we move the spdk_thread into a different system
> thread, should we pause and pass the object to other thread and resume? What
> happens to the NVMf poll group which is associated in that thread. Are they
> movable to a different system thread? My understanding is that nvmf q-pair
> once associated needs to stick that poll group. Is that the same for system
> threads as well (meaning does it use any thread local variables) which stops
> them from moving to a different thread.

Since you have your own async framework, it's best to not use the SPDK framework (lib/event) like you indicated.
You can simply incorporate the SPDK lightweight thread library (lib/thread) directly with your current framework
instead. All of the code in the nvmf library only runs within the context of an SPDK lightweight thread
(struct spdk_thread). And a lightweight thread only is running when your framework calls spdk_thread_poll
on that particular thread. Therefore, if no system thread is currently calling spdk_thread_poll on the lightweight
thread in question, you are free to move it to another system thread via message passing or however your
framework wants to do it. You do not need any additional coordination at the lib/nvmf level - all of the code
associated with that thread will only execute when your framework calls spdk_thread_poll(), so if you don't
call that during the move, then nothing is running.

Shuhei and I have been doing some work to dynamically reschedule threads for the applications SPDK ships,
which all rely on the SPDK application framework. You can use that code as an example (all in lib/event). But
the specifics of how the thread library will need to be integrated into your particular framework depend heavily
on the design of your framework. Fortunately the thread library's requirements are very small - a few function
callbacks - so I think it'll be a quick task. Certainly in the few instances where we've tried to integrate it with
an outside framework, it ended up only requiring ~100 lines of code.

Feel free to ask questions if you hit any issues.

Thanks,
Ben

> 
> Thanks
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org