[SPDK] Re: "nvmf_create_transport" fails with 16 cores

Thanks Ziye for your insight. It makes sense!

Charlie

On Mon, Jan 18, 2021 at 7:28 PM Yang, Ziye <ziye.yang(a)intel.com> wrote:

> Hi Charlie,
>
> Two definitions in tcp.c
>
> #define SPDK_NVMF_TCP_DEFAULT_NUM_SHARED_BUFFERS 511
> #define SPDK_NVMF_TCP_DEFAULT_BUFFER_CACHE_SIZE 32
>
> If you create 16 cores, then the total buffer size for different polling
> groups are:  32*16 = 512 (512 > 511),
>
> The hint is :
>
> " tcp.c: 604:nvmf_tcp_create: *ERROR*: Please specify a larger number of
> shared buffers "
>
> So when you create your transport with TCP,
>
> You need to enlarge the total shared buffer size by -n such as -n 1024,
> nvmf_create_transport -t TCP -n 1014
>
>  then this issue will not happen, this is not a bug.
>
>
>
>
> Best Regards,
> Ziye Yang
>
> -----Original Message-----
> From: Luse, Paul E <paul.e.luse(a)intel.com>
> Sent: Tuesday, January 19, 2021 5:19 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Re: "nvmf_create_transport" fails with 16 cores
>
> Oh gotcha ☺  Sure, it’s a holiday in the US today and I’m on my way out
> the door.  If you don’t get an answer by the time I logon tomorrow morning
> I’ll do that and see what happens on this end.  Thanks!!
>
> -Paul
>
> From: Charlie Li <charlie.li(a)gmail.com>
> Date: Monday, January 18, 2021 at 2:11 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Re: "nvmf_create_transport" fails with 16 cores Thanks
> Paul, I have more than 16 cores. I don't have any problem starting the
> target with 16 or more cores.
> But when I try to create a TCP transport after the target is started, it
> fails. Can you try "sudo ./scripts/rpc.py nvmf_create_transport -t TCP" on
> your system?
>
> Charlie
>
>
> On Mon, Jan 18, 2021 at 2:57 PM Luse, Paul E <paul.e.luse(a)intel.com>
> wrote:
>
> > This might sound like a dumb question but how many physical cores do
> > you have? I just fire up the target on 32 w/o problems:
> >
> > sudo ./build/bin/nvmf_tgt -m 0xffffffff
> >
> > …
> >
> > [2021-01-18 15:56:41.042423] app.c: 536:spdk_app_start: *NOTICE*:
> > Total cores available: 32
> > [2021-01-18 15:56:41.205684] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 1
> > [2021-01-18 15:56:41.206424] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 2
> > [2021-01-18 15:56:41.206963] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 3
> > [2021-01-18 15:56:41.207589] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 4
> > [2021-01-18 15:56:41.208210] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 5
> > [2021-01-18 15:56:41.208828] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 6
> > [2021-01-18 15:56:41.209481] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 7
> > [2021-01-18 15:56:41.210092] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 8
> > [2021-01-18 15:56:41.210723] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 9
> > [2021-01-18 15:56:41.211361] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 10
> > [2021-01-18 15:56:41.212071] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 11
> > [2021-01-18 15:56:41.212882] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 12
> > [2021-01-18 15:56:41.213586] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 13
> > [2021-01-18 15:56:41.214310] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 14
> > [2021-01-18 15:56:41.215053] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 15
> > [2021-01-18 15:56:41.215987] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 16
> > [2021-01-18 15:56:41.216847] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 17
> > [2021-01-18 15:56:41.217799] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 18
> > [2021-01-18 15:56:41.218646] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 19
> > [2021-01-18 15:56:41.219347] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 20
> > [2021-01-18 15:56:41.220023] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 21
> > [2021-01-18 15:56:41.220869] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 22
> > [2021-01-18 15:56:41.221797] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 23
> > [2021-01-18 15:56:41.222801] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 24
> > [2021-01-18 15:56:41.223822] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 25
> > [2021-01-18 15:56:41.224711] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 26
> > [2021-01-18 15:56:41.225398] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 27
> > [2021-01-18 15:56:41.226399] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 28
> > [2021-01-18 15:56:41.227232] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 29
> > [2021-01-18 15:56:41.228311] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 30
> > [2021-01-18 15:56:41.229336] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 31
> > [2021-01-18 15:56:41.230389] reactor.c: 699:reactor_run: *NOTICE*:
> > Reactor started on core 0
> > [2021-01-18 15:56:41.231103] accel_engine.c:
> > 692:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized
> > to use software engine.
> >
> > From: Charlie Li <charlie.li(a)gmail.com>
> > Date: Monday, January 18, 2021 at 1:47 PM
> > To: spdk(a)lists.01.org <spdk(a)lists.01.org>
> > Subject: [SPDK] "nvmf_create_transport" fails with 16 cores Hello,
> >
> > I am using SPDK 20.10 - "Starting SPDK v20.10 git sha1 e5d26ecc2 /
> > DPDK
> > 20.08.0 initialization..."
> >
> > If I start SPDK NVMe-oF target with 15 cores - "sudo
> > ./build/bin/nvmf_tgt -m 0x7FFF", "sudo ./scripts/rpc.py
> > nvmf_create_transport -t TCP" works fine.
> >
> > If I start SPDK NVMe-oF target with 16 cores - "sudo
> > ./build/bin/nvmf_tgt -m 0xFFFF", "sudo ./scripts/rpc.py
> > nvmf_create_transport -t TCP" fails with the following message
> >
> > request:
> > {
> >   "trtype": "TCP",
> >   "c2h_success": true,
> >   "no_wr_batching": false,
> >   "method": "nvmf_create_transport",
> >   "req_id": 1
> > }
> > Got JSON-RPC error response
> > response:
> > {
> >   "code": -32603,
> >   "message": "Transport type 'TCP' create failed"
> > }
> >
> > Is there any way to use 16 or more cores?
> >
> > Thanks,
> > Charlie
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to
> > spdk-leave(a)lists.01.org
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to
> > spdk-leave(a)lists.01.org
> >
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>