[SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target

Hi Eric,

SPDK will start polled-mode threads on each of the cores specified in your coremask.  So for coremask 0xF0, it would start polled-mode threads on cores 4, 5, 6 and 7.  The number of cores in your coremask really depends on your system and what you want to test - if you give the application more cores to run on, you'll be able to get better overall performance and throughput.

If you don't specify a coremask, it will just use core 0 only.  That's fine for just getting something up and running to play around with.

-Jim


On 10/16/20, 7:13 AM, "Curtin, Eric" <Eric.Curtin(a)dell.com> wrote:

    Thanks guys, I got it working. You were right I was not running as root. The other steps I had to perform in case anyone else ever gets curious (I used 127.0.0.1). The only think I don't understand is the coremasks I used 0xf0 and 0x10, I just guessed those based on random examples I saw online. How do I know what coremasks I want to pick?

    mkdir -p /dev/hugepages
    mountpoint -q /dev/hugepages || mount -t hugetlbfs nodev /dev/hugepages
    echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
    build/bin/nvmf_tgt -m 0xf

    In another terminal:

    scripts/rpc.py nvmf_create_transport -t TCP
    scripts/rpc.py bdev_malloc_create -b Malloc0 200 512
    scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1
    scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0
    scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 127.0.0.1 -s 4420
    build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:127.0.0.1 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' -q 128 -o 4096 -w write -t 100 -c 0x10

    Mise le meas/Regards,

    Eric Curtin
    Software Engineer

    Ovens Campus,
    Cork,
    Ireland

    Dell EMC

    Next planned absences: NA
    ________________________________
    From: Rui Chang <Rui.Chang(a)arm.com>
    Sent: Thursday 15 October 2020 02:49
    To: Curtin, Eric <Eric.Curtin(a)dell.com>
    Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
    Subject: RE: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target


    [EXTERNAL EMAIL]

    Yeah, it also works on my virtualbox 6.1 with Win10/X86_64 host and Ubuntu 20.4 guest.

    Regards,
    Rui

    -----Original Message-----
    From: Luse, Paul E <paul.e.luse(a)intel.com>
    Sent: Thursday, October 15, 2020 07:48
    To: Storage Performance Development Kit <spdk(a)lists.01.org>; Rui Chang <Rui.Chang(a)arm.com>
    Cc: spdk(a)lists.01.org
    Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target

    Hi Eric,

    Yeah, run with sudo.  I get the same thing when I run w/o, it's DPDK unable to get the PA with privilege.

    Thx
    Paul

    From: Luse, Paul E <paul.e.luse(a)intel.com>
    Date: Wednesday, October 14, 2020 at 4:43 PM
    To: Storage Performance Development Kit <spdk(a)lists.01.org>, Rui Chang <Rui.Chang(a)arm.com>
    Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
    Subject: Re: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target Hi Eric,

    Bummer that's not working.  It works OK for my with vbox 6.1 on a mac and ubuntu 1901 and the output says iova-mode=pa. Did you change any settings on your VM and try simple stuf like adding memory and/or running with one core (how many cores did you give your VM?).  Also are you running privelaged?

    Thx
    Paul

    peluse(a)pels:~/spdk$ sudo ./build/bin/nvmf_tgt
    [2020-10-14 23:38:50.539538] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
    [2020-10-14 23:38:50.540128] [ DPDK EAL parameters: [2020-10-14 23:38:50.540346] nvmf [2020-10-14 23:38:50.540556] --no-shconf [2020-10-14 23:38:50.540755] -c 0x1 [2020-10-14 23:38:50.540957] --log-level=lib.eal:6 [2020-10-14 23:38:50.541153] --log-level=lib.cryptodev:5 [2020-10-14 23:38:50.541342] --log-level=user1:6 [2020-10-14 23:38:50.541566] --iova-mode=pa [2020-10-14 23:38:50.541763] --base-virtaddr=0x200000000000 [2020-10-14 23:38:50.541957] --match-allocations [2020-10-14 23:38:50.542290] --file-prefix=spdk_pid918 [2020-10-14 23:38:50.542544] ]
    EAL: No legacy callbacks, legacy socket not created
    [2020-10-14 23:38:50.552172] app.c: 667:spdk_app_start: *NOTICE*: Total cores available: 1

    From: Curtin, Eric <Eric.Curtin(a)dell.com>
    Date: Wednesday, October 14, 2020 at 1:56 AM
    To: Rui Chang <Rui.Chang(a)arm.com>
    Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
    Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target How would I get my CPU mask on Linux? I'm running on a Virtualbox VM. I also used 0xf as a CPU mask and this was my output:

    ./build/bin/nvmf_tgt -m 0xf
    [2020-10-14 09:47:07.650399] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
    [2020-10-14 09:47:07.651075] [ DPDK EAL parameters: [2020-10-14 09:47:07.651092] nvmf [2020-10-14 09:47:07.651126] --no-shconf [2020-10-14 09:47:07.651132] -c 0xf [2020-10-14 09:47:07.651138] --log-level=lib.eal:6 [2020-10-14 09:47:07.651144] --log-level=lib.cryptodev:5 [2020-10-14 09:47:07.651150] --log-level=user1:6 [2020-10-14 09:47:07.651160] --iova-mode=pa [2020-10-14 09:47:07.651166] --base-virtaddr=0x200000000000 [2020-10-14 09:47:07.651172] --match-allocations [2020-10-14 09:47:07.651181] --file-prefix=spdk_pid702834 [2020-10-14 09:47:07.651187] ]
    EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
    EAL: Cannot use IOVA as 'PA' since physical addresses are not available
    [2020-10-14 09:47:07.659984] init.c: 577:spdk_env_init: *ERROR*: Failed to initialize DPDK
    [2020-10-14 09:47:07.659999] app.c: 535:app_setup_env: *ERROR*: Unable to initialize SPDK env

    Regards,

    Eric Curtin
    Software Engineer

    Ovens Campus,
    Cork,
    Ireland

    Dell EMC

    Next planned absences: NA
    ________________________________
    From: Rui Chang <Rui.Chang(a)arm.com>
    Sent: Wednesday 14 October 2020 08:50
    To: Curtin, Eric <Eric.Curtin(a)dell.com>
    Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
    Subject: RE: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target


    [EXTERNAL EMAIL]

    Hi Eric



    Refer to this page for HOWTO:

    https://spdk.io/doc/nvmf.html



    You said you want to run on one machine, suppose your machine's IP is 8.8.8.6



      1.  Build your SPDK

    git clone https://github.com/spdk/spdk spdk

    cd spdk

    git submodule update --init

    ./configure

    make



      1.  Run:



    Start target:

                    ./build/bin/nvmf_tgt -m 0xf          (-m gives the CPU mask. Normally you can make it on the same CPU socket as network card to get better performance)



    On another console, run rpc comamnds to do configuration:

    ./scripts/rpc.py nvmf_create_transport -t TCP      (there are lots of additional options you can use if you wish)

    ./scripts/rpc.py bdev_malloc_create -b Malloc0 200 512

    ./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

    ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0

    ./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 8.8.8.6 -s 4420                 (suppose 8.8.8.6 is the IP you want the target to listen on)



    Run io:

    ./build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:8.8.8.6 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'  -q 128 -o 4096  -w write -t 100 -c 0xf0  (-c give the cpu mask for perf tool, you'd better make it different with nvmf_tgt to get better performance)



    Regards,

    Rui





    -----Original Message-----

    From: eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com> <eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com>>

    Sent: Thursday, October 1, 2020 3:23 PM

    To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

    Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target



    Hi Guys,



    I'm just getting started with SPDK and wanted to create a fake ramdisk NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on a single Linux instance for development purposes.



    Anybody know how to set up a machine like this for NVMe/TCP development?

    _______________________________________________

    SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

    To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-leave(a)lists.01.org>

    IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org