[SPDK] Re: SPDK and RocksDB integration -- setup problem
Latecki, Karol <karol.latecki at intel.com>
| Newsgroups | dev.linux.lists.spdk |
|---|---|
| Message-ID | <BY5PR11MB3942EC741725597F944B34A9E7C40@BY5PR11MB3942.namprd11.prod.outlook.com> |
Hey John,
I was able to follow the steps from documentation on spdk.io, with my suggested change. Please take a look at the output below, as I'm no blobfs/rocksdb expert, but it looks fine to me.
I think in your first e-mail you mentioned changing "subsystem" to "subsystems" in .json file. It's not the same what --json-with-subsystems option is doing.
I am using today's SPDK master branch and rocksdb from documentation for this reproduction.
(alternatively, you can just clone rocksdb and just checkout the commit specified in spdk/test/blobfs/rocksdb/rocksdb_commit_id, this should work too)
SPDK:
9ae4346504 (HEAD -> master, origin/master, origin/HEAD) nvmf/fc: Cleanup fc delete connection code.
Git submodule update --init
./configure
Make -j32
ROCKSDB:
Cd ../
git clone -b spdk-v5.14.3 https://github.com/spdk/rocksdb.git
cd rocksdb
make db_bench -j32 EXTRA_CXXFLAGS='-Wno-deprecated-copy -Wno-pessimizing-move -Wno-error=stringop-truncation' SPDK_DIR=/home/vagrant/spdk_repo/spdk
(extra flags needed on my system)
Back to SPDK:
cd spdk
sudo -E HUGEMEM=5120 scripts/setup.sh
sudo -E scripts/gen_nvme.sh --json-with-subsystems > bdev.conf.1 ("good" config)
sudo -E scripts/gen_nvme.sh > bdev.conf.2 ("bad" config)
[vagrant(a)fedora31 spdk]$ sudo -E test/blobfs/mkfs/mkfs bdev.conf.2 Nvme0n1
[2020-12-17 15:19:10.389473] Starting SPDK v21.01-pre git sha1 9ae4346504 / DPDK 20.08.0 initialization...
[2020-12-17 15:19:10.389678] [ DPDK EAL parameters: [2020-12-17 15:19:10.389706] spdk_mkfs [2020-12-17 15:19:10.389719] --no-shconf [2020-12-17 15:19:10.389737] -c 0x3 [2020-12-17 15:19:10.389751] --log-level=lib.eal:6 [2020-12-17 15:19:10.389765] --log-level=lib.cryptodev:5 [2020-12-17 15:19:10.389778] --log-level=user1:6 [2020-12-17 15:19:10.389792] --iova-mode=pa [2020-12-17 15:19:10.389805] --base-virtaddr=0x200000000000 [2020-12-17 15:19:10.389819] --match-allocations [2020-12-17 15:19:10.389833] --file-prefix=spdk_pid207739 [2020-12-17 15:19:10.389846] ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: No legacy callbacks, legacy socket not created
[2020-12-17 15:19:10.511290] app.c: 537:spdk_app_start: *NOTICE*: Total cores available: 2
[2020-12-17 15:19:10.833856] reactor.c: 699:reactor_run: *NOTICE*: Reactor started on core 1
[2020-12-17 15:19:10.835315] reactor.c: 699:reactor_run: *NOTICE*: Reactor started on core 0
[2020-12-17 15:19:10.835482] json_config.c: 594:spdk_app_json_config_load: *WARNING*: No 'subsystems' key JSON configuration file.
[2020-12-17 15:19:10.836813] accel_engine.c: 692:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
Initializing filesystem on bdev Nvme0n1...[2020-12-17 15:19:10.949637] bdev.c:5581:spdk_bdev_open_ext: *NOTICE*: Currently unable to find bdev with name: Nvme0n1
Failed to initialize filesystem on bdev Nvme0n1...done.
[vagrant(a)fedora31 spdk]$ sudo -E test/blobfs/mkfs/mkfs bdev.conf.1 Nvme0n1
[2020-12-17 15:19:29.490919] Starting SPDK v21.01-pre git sha1 9ae4346504 / DPDK 20.08.0 initialization...
[2020-12-17 15:19:29.491321] [ DPDK EAL parameters: [2020-12-17 15:19:29.491392] spdk_mkfs [2020-12-17 15:19:29.491429] --no-shconf [2020-12-17 15:19:29.491460] -c 0x3 [2020-12-17 15:19:29.491490] --log-level=lib.eal:6 [2020-12-17 15:19:29.491519] --log-level=lib.cryptodev:5 [2020-12-17 15:19:29.491548] --log-level=user1:6 [2020-12-17 15:19:29.491577] --iova-mode=pa [2020-12-17 15:19:29.491606] --base-virtaddr=0x200000000000 [2020-12-17 15:19:29.491634] --match-allocations [2020-12-17 15:19:29.491663] --file-prefix=spdk_pid207745 [2020-12-17 15:19:29.491692] ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: No legacy callbacks, legacy socket not created
[2020-12-17 15:19:29.610871] app.c: 537:spdk_app_start: *NOTICE*: Total cores available: 2
[2020-12-17 15:19:29.932216] reactor.c: 699:reactor_run: *NOTICE*: Reactor started on core 1
[2020-12-17 15:19:29.933645] reactor.c: 699:reactor_run: *NOTICE*: Reactor started on core 0
[2020-12-17 15:19:29.934556] accel_engine.c: 692:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
[2020-12-17 15:19:30.064969] nvme_qpair.c: 238:nvme_admin_qpair_print_command: *NOTICE*: SET FEATURES ASYNC EVENT CONFIGURATION cid:23 cdw10:0000000b PRP1 0x0 PRP2 0x0
[2020-12-17 15:19:30.065066] nvme_qpair.c: 451:spdk_nvme_print_completion: *NOTICE*: INVALID FIELD (00/02) qid:0 cid:23 cdw0:0 sqhd:0004 p:1 m:0 dnr:1
[2020-12-17 15:19:30.065096] nvme_ctrlr.c:2614:nvme_ctrlr_configure_aer_done: *NOTICE*: nvme_ctrlr_configure_aer failed!
Initializing filesystem on bdev Nvme0n1...done.
-----Original Message-----
From: Meneghini, John <John.Meneghini(a)netapp.com>
Sent: Thursday, December 17, 2020 3:46 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Cc: Lalsangi, Raj <Raj.Lalsangi(a)netapp.com>; Meneghini, John <John.Meneghini(a)netapp.com>
Subject: [SPDK] Re: SPDK and RocksDB integration -- setup problem
Hi Karol.
We tried this and there's no difference between your suggested changed and the prior one.
Are we sure that everything works correctly with the latest version of SPDK? It looks like the RocksDB code was originally integrated and supported with SPDK v19.01.x
It appears nothing has changed since then.
ssan-rx2560-03:rocksdb(spdk-v5.14.3) > git logg -3
* f43f85564 2019-05-03 (HEAD -> spdk-v5.14.3, origin/spdk-v5.14.3) Add SpdkEnv C API [ Jim Harris / youngtack.jin(a)circuitblvd.com ]
* e28188774 2018-09-04 Add SPDK BlobFS integration. [ Jim Harris / changpeng.liu(a)intel.com ]
* 626550343 2018-08-21 (tag: rocksdb-5.14.3) Bump version to 5.14.3 and update HISTORY [ Yi Wu / yiwu(a)fb.com ]
ssan-rx2560-03:spdk(v19.01.x) > git logg -3
* 3f5e32adc 2019-05-01 (HEAD -> v19.01.x, origin/v19.01.x) test/rocksdb: add rocksdb_commit_id file [ Jim Harris / james.r.harris(a)intel.com ]
* 089585c8d 2019-05-01 rocksdb: use C++ constructor for global channel [ Jim Harris / james.r.harris(a)intel.com ]
* fe3a2c4dc 2019-05-01 test/rocksdb: suppress leak reports on thread local ctx [ Jim Harris / james.r.harris(a)intel.com ]
Should we try this using SPDK v19.01.x?
/John
[root(a)ssan-rx2560-03 spdk]# scripts/gen_nvme.sh ----json-with-subsystems
0000:84:00.0 ....
{
"subsystem": "bdev",
"config": [
{
"method": "bdev_nvme_attach_controller",
"params": {
"trtype": "PCIe",
"name":"Nvme0",
"traddr":"0000:84:00.0"
}
}
]
}
[root(a)ssan-rx2560-03 spdk]# scripts/gen_nvme.sh
0000:84:00.0 ....
{
"subsystem": "bdev",
"config": [
{
"method": "bdev_nvme_attach_controller",
"params": {
"trtype": "PCIe",
"name":"Nvme0",
"traddr":"0000:84:00.0"
}
}
]
}
On 12/17/20, 4:31 AM, "Latecki, Karol" <karol.latecki(a)intel.com> wrote:
Hey John!
Could you try "scripts/gen_nvme.sh ----json-with-subsystems" and let me know if this worked?
Karol
-----Original Message-----
From: Meneghini, John <John.Meneghini(a)netapp.com>
Sent: Wednesday, December 16, 2020 7:42 PM
To: spdk(a)lists.01.org
Cc: Meneghini, John <John.Meneghini(a)netapp.com>; Lalsangi, Raj <Raj.Lalsangi(a)netapp.com>
Subject: [SPDK] SPDK and RocksDB integration -- setup problem
I have the followed the instructions here -- https://spdk.io/doc/blobfs.html to setup RockDB integration with SPDK environment. And I’m using top of the tree SPDK from github.
Here is the output from running scripts/gen_nvme.sh:
[lalsangi(a)spdk]$ scripts/gen_nvme.sh
0000:84:00.0 ....
{
"subsystem": "bdev",
"config": [
{
"method": "bdev_nvme_attach_controller",
"params": {
"trtype": "PCIe",
"name":"Nvme0",
"traddr":"0000:84:00.0"
}
}
]
}
And the output from scripts/setup.sh: (with some tracing)
[root(a)spdk]# HUGEMEM=5120 scripts/setup.sh
START ...>
----------
TARGET_USER .. defined.
lalsangi ..
OS is Linux! .. config
configure_linux_pci ..
vfio-pci ..
bdf related stuff ..
0000:84:00.0 (8086 0953): no driver -> vfio-pci
8086 0953
0000:84:00.0
1
end of config_linux_pci ..
config_linux_pci .. DONE.
-------------------------
hugetblfs_mounts ..
2560
Setting 2560 in /proc/sys/vm/nr_hugepages ..
/dev/hugepages
...............
Target user is: lalsangi
Password:
MEMLOCK_AMNT = 65536 ..
"lalsangi" user memlock limit: 64 MB
This is the maximum amount of memory you will be
able to use with DPDK and VFIO if run as user "lalsangi".
To change this, please adjust limits.conf memlock limit for user "lalsangi".
[root(a)spdk]#
And finally, the problem when I run ‘test/blobfs/mkfs/mkfs /usr/local/etc/spdk/rocksdb.json Nvme0n1: (with some tracing) (I tried with different bdev names -- e.g., Nvme0 -- still same problem)
[root(a)ssan-rx2560-03 spdk]# test/blobfs/mkfs/mkfs /usr/local/etc/spdk/rocksdb.json Nvme0n1
[2020-12-14 21:30:05.201555] Launching mkfs, config_file: /usr/local/etc/spdk/rocksdb.json
[2020-12-14 21:30:05.201832] Starting SPDK v21.01-pre git sha1 602b134fa / DPDK 20.08.0 initialization...
[2020-12-14 21:30:05.201879] [ DPDK EAL parameters: [2020-12-14 21:30:05.201906] spdk_mkfs [2020-12-14 21:30:05.201928] --no-shconf [2020-12-14 21:30:05.201948] -c 0x3 [2020-12-14 21:30:05.201970] --log-level=lib.eal:6 [2020-12-14 21:30:05.201992] --log-level=lib.cryptodev:5 [2020-12-14 21:30:05.202012] --log-level=user1:6 [2020-12-14 21:30:05.202031] --base-virtaddr=0x200000000000 [2020-12-14 21:30:05.202051] --match-allocations [2020-12-14 21:30:05.202071] --file-prefix=spdk_pid103642 [2020-12-14 21:30:05.202092] ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: No legacy callbacks, legacy socket not created
[2020-12-14 21:30:05.250379] app.c: 465:spdk_app_start: *NOTICE*: Total cores available: 2
[2020-12-14 21:30:05.250411] Total cores available: 2
[2020-12-14 21:30:05.250538] spdk_reactor_init ..
[2020-12-14 21:30:05.347656] Reactor init done ..
[2020-12-14 21:30:05.347710] cpuset_set_cpu done ..
[2020-12-14 21:30:05.347719] spdk_thread_create init done ..
[2020-12-14 21:30:05.348539] spdk_mempool_get_bulk done ..
[2020-12-14 21:30:05.348556] Allowing new thread app_thread ..
[2020-12-14 21:30:05.348568] Ran the thread function app_thread ..
[2020-12-14 21:30:05.461055] send msg to run bootstrap_fn ..
[2020-12-14 21:30:05.461181] reactor.c: 701:reactor_run: *NOTICE*: Reactor started on core 1
[2020-12-14 21:30:05.461184] spdk_thread_create init done ..
[2020-12-14 21:30:05.462004] spdk_mempool_get_bulk done ..
[2020-12-14 21:30:05.462035] Allowing new thread reactor_1 ..
[2020-12-14 21:30:05.462045] Ran the thread function reactor_1 ..
[2020-12-14 21:30:05.462052] reactor.c: 701:reactor_run: *NOTICE*: Reactor started on core 0
[2020-12-14 21:30:05.462080] Load json config /usr/local/etc/spdk/rocksdb.json ..Inside spdk_json_parse ..
Inside spdk_json_parse ..
[2020-12-14 21:30:05.462126] Read the json config file /usr/local/etc/spdk/rocksdb.json ..
[2020-12-14 21:30:05.462155] Find the key_name: subsystem ..
[2020-12-14 21:30:05.462164] parsing token #??..
[2020-12-14 21:30:05.462172] key #??, value 0??
[2020-12-14 21:30:05.462180] json_config.c: 597:spdk_app_json_config_load: *WARNING*: No 'subsystems' key JSON configuration file.
[2020-12-14 21:30:05.462524] accel_engine.c: 692:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
Initializing filesystem on bdev Nvme0n1...[2020-12-14 21:30:05.527735] bdev.c:5562:spdk_bdev_open_ext: *NOTICE*: Currently unable to find bdev with name: Nvme0n1
Failed to initialize filesystem on bdev Nvme0n1...done.
First, I noticed that the code is expecting ‘subsystems’ key_name, but the gen_nvme.sh() created the .json with the key ‘subsystem’. I changed the code to match the key_name. But still the same issue exists.
Based on the above, it seems that either I’m making a silly mistake, or something basic is not working.
Any hints / suggestions on debugging this problem is highly appreciated!
Or should I stick with code bits with particular tags for SPDK and RocksDB integration?
Thanks,
John Meneghini
ONTAP SAN Target Architect
978-930-3519 (cell)
johnm(a)netapp.com
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org