[SPDK] Re: "nvmf_create_transport" fails with 16 cores

Hi Charlie,

Two definitions in tcp.c

#define SPDK_NVMF_TCP_DEFAULT_NUM_SHARED_BUFFERS 511
#define SPDK_NVMF_TCP_DEFAULT_BUFFER_CACHE_SIZE 32

If you create 16 cores, then the total buffer size for different polling groups are:  32*16 = 512 (512 > 511),  

The hint is : 

" tcp.c: 604:nvmf_tcp_create: *ERROR*: Please specify a larger number of shared buffers "

So when you create your transport with TCP,

You need to enlarge the total shared buffer size by -n such as -n 1024,
nvmf_create_transport -t TCP -n 1014

 then this issue will not happen, this is not a bug.




Best Regards,
Ziye Yang

-----Original Message-----
From: Luse, Paul E <paul.e.luse(a)intel.com> 
Sent: Tuesday, January 19, 2021 5:19 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: "nvmf_create_transport" fails with 16 cores

Oh gotcha ☺  Sure, it’s a holiday in the US today and I’m on my way out the door.  If you don’t get an answer by the time I logon tomorrow morning I’ll do that and see what happens on this end.  Thanks!!

-Paul

From: Charlie Li <charlie.li(a)gmail.com>
Date: Monday, January 18, 2021 at 2:11 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: "nvmf_create_transport" fails with 16 cores Thanks Paul, I have more than 16 cores. I don't have any problem starting the target with 16 or more cores.
But when I try to create a TCP transport after the target is started, it fails. Can you try "sudo ./scripts/rpc.py nvmf_create_transport -t TCP" on your system?

Charlie


On Mon, Jan 18, 2021 at 2:57 PM Luse, Paul E <paul.e.luse(a)intel.com> wrote:

> This might sound like a dumb question but how many physical cores do 
> you have? I just fire up the target on 32 w/o problems:
>
> sudo ./build/bin/nvmf_tgt -m 0xffffffff
>
> …
>
> [2021-01-18 15:56:41.042423] app.c: 536:spdk_app_start: *NOTICE*: 
> Total cores available: 32
> [2021-01-18 15:56:41.205684] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 1
> [2021-01-18 15:56:41.206424] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 2
> [2021-01-18 15:56:41.206963] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 3
> [2021-01-18 15:56:41.207589] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 4
> [2021-01-18 15:56:41.208210] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 5
> [2021-01-18 15:56:41.208828] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 6
> [2021-01-18 15:56:41.209481] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 7
> [2021-01-18 15:56:41.210092] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 8
> [2021-01-18 15:56:41.210723] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 9
> [2021-01-18 15:56:41.211361] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 10
> [2021-01-18 15:56:41.212071] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 11
> [2021-01-18 15:56:41.212882] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 12
> [2021-01-18 15:56:41.213586] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 13
> [2021-01-18 15:56:41.214310] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 14
> [2021-01-18 15:56:41.215053] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 15
> [2021-01-18 15:56:41.215987] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 16
> [2021-01-18 15:56:41.216847] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 17
> [2021-01-18 15:56:41.217799] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 18
> [2021-01-18 15:56:41.218646] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 19
> [2021-01-18 15:56:41.219347] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 20
> [2021-01-18 15:56:41.220023] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 21
> [2021-01-18 15:56:41.220869] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 22
> [2021-01-18 15:56:41.221797] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 23
> [2021-01-18 15:56:41.222801] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 24
> [2021-01-18 15:56:41.223822] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 25
> [2021-01-18 15:56:41.224711] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 26
> [2021-01-18 15:56:41.225398] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 27
> [2021-01-18 15:56:41.226399] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 28
> [2021-01-18 15:56:41.227232] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 29
> [2021-01-18 15:56:41.228311] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 30
> [2021-01-18 15:56:41.229336] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 31
> [2021-01-18 15:56:41.230389] reactor.c: 699:reactor_run: *NOTICE*: 
> Reactor started on core 0
> [2021-01-18 15:56:41.231103] accel_engine.c:
> 692:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized 
> to use software engine.
>
> From: Charlie Li <charlie.li(a)gmail.com>
> Date: Monday, January 18, 2021 at 1:47 PM
> To: spdk(a)lists.01.org <spdk(a)lists.01.org>
> Subject: [SPDK] "nvmf_create_transport" fails with 16 cores Hello,
>
> I am using SPDK 20.10 - "Starting SPDK v20.10 git sha1 e5d26ecc2 / 
> DPDK
> 20.08.0 initialization..."
>
> If I start SPDK NVMe-oF target with 15 cores - "sudo 
> ./build/bin/nvmf_tgt -m 0x7FFF", "sudo ./scripts/rpc.py 
> nvmf_create_transport -t TCP" works fine.
>
> If I start SPDK NVMe-oF target with 16 cores - "sudo 
> ./build/bin/nvmf_tgt -m 0xFFFF", "sudo ./scripts/rpc.py 
> nvmf_create_transport -t TCP" fails with the following message
>
> request:
> {
>   "trtype": "TCP",
>   "c2h_success": true,
>   "no_wr_batching": false,
>   "method": "nvmf_create_transport",
>   "req_id": 1
> }
> Got JSON-RPC error response
> response:
> {
>   "code": -32603,
>   "message": "Transport type 'TCP' create failed"
> }
>
> Is there any way to use 16 or more cores?
>
> Thanks,
> Charlie
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to 
> spdk-leave(a)lists.01.org 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to 
> spdk-leave(a)lists.01.org
>
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org