[SPDK] Re: Query on scaling SPDK threads

Hi Lokesh,

There currently is no way to dynamically change the size of the message pool or the cache size in the thread library.  I would be open to adding runtime initialization parameters for these fields though if you are interested in working on that.

The numbers were selected a long time back based on a maximum CPU core count of 256 cores, and no more than 1 spdk_thread per core.  This may need revisiting though, because we may create multiple spdk_threads per core when running multiple front-end protocols (i.e. vhost + nvmf + iscsi) at once.

I would expect no issues with just allocating more messages in the message pool.

SPDK gets best scaling when there is just one POSIX pthread running per core.  The SPDK application framework (lib/event) is based on this model using the spdk_reactor.  The SPDK libraries may create multiple spdk_threads per core, but these are lightweight thread abstractions so they scale very well.  If you are creating 300 threads in your application, the scaling will depend a lot on whether those are pthreads or spdk_threads, and how many cores you are allocating for those threads to execute.

-Jim


On 9/19/21, 1:26 AM, "lokesharo(a)gmail.com" <lokesharo(a)gmail.com> wrote:

    Hello

    In the SPDK source code, the message pool is of size 262143 ( taken from _thread_lib_init() ) and the per thread cache size is SPDK_MSG_MEMPOOL_CACHE_SIZE (1024). What this means is the first 255 threads created will have the cache size as 1024 and after that the threads will be created but the cache will be NULL and the threads have to refer the global pool to get the msge object to send the messages.

    In my application, we create around 300 threads during init. So, 256th thread onward, global pool is referenced for the msge object. 
    Now, when the program is run, some of the threads that got 1024 entries in the cache while creation (i.e. threads between 0 - 255) remain unused. What this means is that some of the entries from the global pool remain unused always and the threads (from 256 - 300) are not able to get the free entries from the global pool and this leads to spdk_thread_send_msg() failing for the threads in range of 256 - 300.

    I have the following doubts here:
    1. How do we scale the number of threads in the SPDK environment with the hardcoded value of cache size and pool size.
    2. Why is the pool size set to 262143 and cache size to 1024. Is there some logical explanation behind the numbers?
    3. If the user changes the above two numbers, are there any issues with it, be it performance or memory or any thing else?
    4. Is there any maximum limit that the SPDK suggests for the scaling of threads? If yes, what is the scaling model that SPDK suggests the user should implement?

    Thanks
    Lokesh
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org