[SPDK] Re: SPDK and RocksDB integration -- setup problem

Hi Karol.

We tried this and there's no difference between your suggested changed and the prior one.

Are we sure that everything works correctly with  the latest version of SPDK?  It looks like the RocksDB code was originally integrated and supported with SPDK v19.01.x

It appears nothing has changed since then.  

ssan-rx2560-03:rocksdb(spdk-v5.14.3) > git logg -3
* f43f85564 2019-05-03  (HEAD -> spdk-v5.14.3, origin/spdk-v5.14.3) Add SpdkEnv C API [ Jim Harris / youngtack.jin(a)circuitblvd.com ]
* e28188774 2018-09-04  Add SPDK BlobFS integration. [ Jim Harris / changpeng.liu(a)intel.com ]
* 626550343 2018-08-21  (tag: rocksdb-5.14.3) Bump version to 5.14.3 and update HISTORY [ Yi Wu / yiwu(a)fb.com ]

ssan-rx2560-03:spdk(v19.01.x) > git logg -3
* 3f5e32adc 2019-05-01  (HEAD -> v19.01.x, origin/v19.01.x) test/rocksdb: add rocksdb_commit_id file [ Jim Harris / james.r.harris(a)intel.com ]
* 089585c8d 2019-05-01  rocksdb: use C++ constructor for global channel [ Jim Harris / james.r.harris(a)intel.com ]
* fe3a2c4dc 2019-05-01  test/rocksdb: suppress leak reports on thread local ctx [ Jim Harris / james.r.harris(a)intel.com ]

Should we try this using SPDK v19.01.x?

/John

[root(a)ssan-rx2560-03 spdk]# scripts/gen_nvme.sh ----json-with-subsystems
0000:84:00.0 ....
{
"subsystem": "bdev",
"config": [
{
"method": "bdev_nvme_attach_controller",
"params": {
"trtype": "PCIe",
"name":"Nvme0",
"traddr":"0000:84:00.0"
}
}
]
}

[root(a)ssan-rx2560-03 spdk]# scripts/gen_nvme.sh
0000:84:00.0 ....
{
"subsystem": "bdev",
"config": [
{
"method": "bdev_nvme_attach_controller",
"params": {
"trtype": "PCIe",
"name":"Nvme0",
"traddr":"0000:84:00.0"
}
}
]
}



On 12/17/20, 4:31 AM, "Latecki, Karol" <karol.latecki(a)intel.com> wrote:

    Hey John!
    Could you try "scripts/gen_nvme.sh ----json-with-subsystems" and let me know if this worked?

    Karol

    -----Original Message-----
    From: Meneghini, John <John.Meneghini(a)netapp.com>
    Sent: Wednesday, December 16, 2020 7:42 PM
    To: spdk(a)lists.01.org
    Cc: Meneghini, John <John.Meneghini(a)netapp.com>; Lalsangi, Raj <Raj.Lalsangi(a)netapp.com>
    Subject: [SPDK] SPDK and RocksDB integration -- setup problem

    I have the followed the instructions here -- https://spdk.io/doc/blobfs.html to setup RockDB integration with SPDK environment. And I’m using top of the tree SPDK from github.

    Here is the output from running scripts/gen_nvme.sh:


    [lalsangi(a)spdk]$ scripts/gen_nvme.sh

    0000:84:00.0 ....

    {

    "subsystem": "bdev",

    "config": [

    {

    "method": "bdev_nvme_attach_controller",

    "params": {

    "trtype": "PCIe",

    "name":"Nvme0",

    "traddr":"0000:84:00.0"

    }

    }

    ]

    }

    And the output from scripts/setup.sh: (with some tracing)


    [root(a)spdk]# HUGEMEM=5120 scripts/setup.sh

    START ...>

    ----------

    TARGET_USER .. defined.

    lalsangi ..

    OS is Linux! .. config

    configure_linux_pci ..

    vfio-pci ..

    bdf related stuff ..

    0000:84:00.0 (8086 0953): no driver -> vfio-pci

    8086 0953

    0000:84:00.0

    1

    end of config_linux_pci ..

    config_linux_pci .. DONE.

    -------------------------

    hugetblfs_mounts ..

    2560

    Setting 2560 in /proc/sys/vm/nr_hugepages ..

    /dev/hugepages

    ...............

    Target user is: lalsangi

    Password:

    MEMLOCK_AMNT = 65536 ..

    "lalsangi" user memlock limit: 64 MB



    This is the maximum amount of memory you will be

    able to use with DPDK and VFIO if run as user "lalsangi".

    To change this, please adjust limits.conf memlock limit for user "lalsangi".

    [root(a)spdk]#

    And finally, the problem when I run ‘test/blobfs/mkfs/mkfs /usr/local/etc/spdk/rocksdb.json Nvme0n1: (with some tracing) (I tried with different bdev names -- e.g., Nvme0 -- still same problem)


    [root(a)ssan-rx2560-03 spdk]# test/blobfs/mkfs/mkfs /usr/local/etc/spdk/rocksdb.json Nvme0n1

    [2020-12-14 21:30:05.201555] Launching mkfs, config_file: /usr/local/etc/spdk/rocksdb.json

    [2020-12-14 21:30:05.201832] Starting SPDK v21.01-pre git sha1 602b134fa / DPDK 20.08.0 initialization...

    [2020-12-14 21:30:05.201879] [ DPDK EAL parameters: [2020-12-14 21:30:05.201906] spdk_mkfs [2020-12-14 21:30:05.201928] --no-shconf [2020-12-14 21:30:05.201948] -c 0x3 [2020-12-14 21:30:05.201970] --log-level=lib.eal:6 [2020-12-14 21:30:05.201992] --log-level=lib.cryptodev:5 [2020-12-14 21:30:05.202012] --log-level=user1:6 [2020-12-14 21:30:05.202031] --base-virtaddr=0x200000000000 [2020-12-14 21:30:05.202051] --match-allocations [2020-12-14 21:30:05.202071] --file-prefix=spdk_pid103642 [2020-12-14 21:30:05.202092] ]

    EAL: No available hugepages reported in hugepages-1048576kB

    EAL: No legacy callbacks, legacy socket not created

    [2020-12-14 21:30:05.250379] app.c: 465:spdk_app_start: *NOTICE*: Total cores available: 2

    [2020-12-14 21:30:05.250411] Total cores available: 2

    [2020-12-14 21:30:05.250538] spdk_reactor_init ..

    [2020-12-14 21:30:05.347656] Reactor init done ..

    [2020-12-14 21:30:05.347710] cpuset_set_cpu done ..

    [2020-12-14 21:30:05.347719] spdk_thread_create init done ..

    [2020-12-14 21:30:05.348539] spdk_mempool_get_bulk done ..

    [2020-12-14 21:30:05.348556] Allowing new thread app_thread ..

    [2020-12-14 21:30:05.348568] Ran the thread function app_thread ..

    [2020-12-14 21:30:05.461055] send msg to run bootstrap_fn ..

    [2020-12-14 21:30:05.461181] reactor.c: 701:reactor_run: *NOTICE*: Reactor started on core 1

    [2020-12-14 21:30:05.461184] spdk_thread_create init done ..

    [2020-12-14 21:30:05.462004] spdk_mempool_get_bulk done ..

    [2020-12-14 21:30:05.462035] Allowing new thread reactor_1 ..

    [2020-12-14 21:30:05.462045] Ran the thread function reactor_1 ..

    [2020-12-14 21:30:05.462052] reactor.c: 701:reactor_run: *NOTICE*: Reactor started on core 0

    [2020-12-14 21:30:05.462080] Load json config /usr/local/etc/spdk/rocksdb.json ..Inside spdk_json_parse ..

    Inside spdk_json_parse ..

    [2020-12-14 21:30:05.462126] Read the json config file /usr/local/etc/spdk/rocksdb.json ..

    [2020-12-14 21:30:05.462155] Find the key_name: subsystem ..

    [2020-12-14 21:30:05.462164] parsing token  #??..

    [2020-12-14 21:30:05.462172] key  #??, value 0??

    [2020-12-14 21:30:05.462180] json_config.c: 597:spdk_app_json_config_load: *WARNING*: No 'subsystems' key JSON configuration file.

    [2020-12-14 21:30:05.462524] accel_engine.c: 692:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.

    Initializing filesystem on bdev Nvme0n1...[2020-12-14 21:30:05.527735] bdev.c:5562:spdk_bdev_open_ext: *NOTICE*: Currently unable to find bdev with name: Nvme0n1



    Failed to initialize filesystem on bdev Nvme0n1...done.

    First, I noticed that the code is expecting ‘subsystems’ key_name, but the gen_nvme.sh() created the .json with the key ‘subsystem’.  I changed the code to match the key_name. But still the same issue exists.

    Based on the above, it seems that either I’m making a silly mistake, or something basic is not working.

    Any hints / suggestions on debugging this problem is highly appreciated!

    Or should I stick with code bits with particular tags for SPDK and RocksDB integration?

    Thanks,

    John Meneghini
    ONTAP SAN Target Architect
    978-930-3519 (cell)
    johnm(a)netapp.com



    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org