[SPDK] Can rocksdb workloads run in the blobfs of qemu virtual machine?

Hi
I try to test the performance of rocksdb in virtual machine where guest and host nvme driver both work in user mode. 
Running spdk-rocksdb in bare-mental alone works without a problem, but the same configuration fail in VM.
It seems like feasible theoretically, but I don't know if anyone has tried before.

Running configuration:
Guest: 
  - spdk & spdk/rocksdb are mainline version.
  - follow the BlobFS Getting Started Guide.
Host: 
  - spdk 18.04 release
  - spdk/qemu with vhost-user-nvme patch(https://review.gerrithub.io/c/spdk/qemu/+/409094/)
  - follow the vhost-target guide

Error log in Guest :

[root(a)localhost rocksdb-spdk]# ./db_bench --benchmarks="fillrandom" -spdk=/root/spdk/rocksdb.json -spdk_bdev=Nvme0n1
[2021-11-23 20:22:17.631040] Starting SPDK v22.01-pre git sha1 a1014fc / DPDK 21.08.0 initialization...
[2021-11-23 20:22:17.631315] [ DPDK EAL parameters: [2021-11-23 20:22:17.631360] rocksdb [2021-11-23 20:22:17.631391] --no-shconf [2021-11-23 20:22:17.631438] -c 0x1 [2021-11-23 20:22:17.631475] --log-level=lib.eal:6 [2021-11-23 20:22:17.631504] --log-level=lib.cryptodev:5 [2021-11-23 20:22:17.631534] --log-level=user1:6 [2021-11-23 20:22:17.631563] --iova-mode=pa [2021-11-23 20:22:17.631592] --base-virtaddr=0x200000000000 [2021-11-23 20:22:17.631621] --match-allocations [2021-11-23 20:22:17.631649] --file-prefix=spdk_pid12118 [2021-11-23 20:22:17.631678] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2021-11-23 20:22:17.775111] app.c: 543:spdk_app_start: *NOTICE*: Total cores available: 1
[2021-11-23 20:22:17.935764] app.c: 389:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified.
[2021-11-23 20:22:17.935853] app.c: 393:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 12118' to capture a snapshot of events at runtime.
[2021-11-23 20:22:17.935874] app.c: 395:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid12118 for offline analysis/debug.
[2021-11-23 20:22:17.935898] reactor.c: 943:reactor_run: *NOTICE*: Reactor started on core 0
[2021-11-23 20:22:17.936476] accel_engine.c:1012:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
[2021-11-23 20:22:18.079801] nvme_qpair.c: 241:nvme_admin_qpair_print_command: *NOTICE*: SET FEATURES ASYNC EVENT CONFIGURATION cid:23 cdw10:0000000b PRP1 0x0 PRP2 0x0
[2021-11-23 20:22:18.079888] nvme_qpair.c: 456:spdk_nvme_print_completion: *NOTICE*: INVALID FIELD (00/02) qid:0 cid:23 cdw0:0 sqhd:0004 p:1 m:0 dnr:1
[2021-11-23 20:22:18.079911] nvme_ctrlr.c:3185:nvme_ctrlr_configure_aer_done: *NOTICE*: [0000:00:03.0] nvme_ctrlr_configure_aer failed!
using bdev Nvme0n1
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
open error: Invalid argument: Found options incompatible with filesystem: IO error: /tmp/rocksdbtest-0/dbbench/CURRENT: No such file or directory



[root(a)localhost examples]# ./identify 
[2021-11-23 20:27:06.080535] Starting SPDK v22.01-pre git sha1 a1014fc / DPDK 21.08.0 initialization...
[2021-11-23 20:27:06.080773] [ DPDK EAL parameters: [2021-11-23 20:27:06.080824] identify [2021-11-23 20:27:06.080854] --no-shconf [2021-11-23 20:27:06.080902] -c 0x1 [2021-11-23 20:27:06.080937] -n 1 [2021-11-23 20:27:06.080965] -m 0 [2021-11-23 20:27:06.080992] --log-level=lib.eal:6 [2021-11-23 20:27:06.081019] --log-level=lib.cryptodev:5 [2021-11-23 20:27:06.081047] --log-level=user1:6 [2021-11-23 20:27:06.081074] --iova-mode=pa [2021-11-23 20:27:06.081101] --base-virtaddr=0x200000000000 [2021-11-23 20:27:06.081129] --match-allocations [2021-11-23 20:27:06.081168] --file-prefix=spdk_pid12136 [2021-11-23 20:27:06.081196] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2021-11-23 20:27:06.293188] nvme_qpair.c: 241:nvme_admin_qpair_print_command: *NOTICE*: SET FEATURES ASYNC EVENT CONFIGURATION cid:23 cdw10:0000000b PRP1 0x0 PRP2 0x0
[2021-11-23 20:27:06.293340] nvme_qpair.c: 456:spdk_nvme_print_completion: *NOTICE*: INVALID FIELD (00/02) qid:0 cid:23 cdw0:0 sqhd:0004 p:1 m:0 dnr:1
[2021-11-23 20:27:06.293394] nvme_ctrlr.c:3185:nvme_ctrlr_configure_aer_done: *NOTICE*: [0000:00:03.0] nvme_ctrlr_configure_aer failed!
[2021-11-23 20:27:06.293925] nvme_qpair.c: 241:nvme_admin_qpair_print_command: *NOTICE*: GET FEATURES ARBITRATION cid:23 cdw10:00000001 PRP1 0x0 PRP2 0x0
[2021-11-23 20:27:06.293986] nvme_qpair.c: 456:spdk_nvme_print_completion: *NOTICE*: INVALID FIELD (00/02) qid:0 cid:23 cdw0:0 sqhd:0006 p:1 m:0 dnr:1
get_feature(0x01) failed
[2021-11-23 20:27:06.294317] nvme_qpair.c: 241:nvme_admin_qpair_print_command: *NOTICE*: GET FEATURES POWER MANAGEMENT cid:23 cdw10:00000002 PRP1 0x0 PRP2 0x0
[2021-11-23 20:27:06.294374] nvme_qpair.c: 456:spdk_nvme_print_completion: *NOTICE*: INVALID FIELD (00/02) qid:0 cid:23 cdw0:0 sqhd:0007 p:1 m:0 dnr:1
get_feature(0x02) failed
[2021-11-23 20:27:06.294563] nvme_qpair.c: 241:nvme_admin_qpair_print_command: *NOTICE*: GET FEATURES TEMPERATURE THRESHOLD cid:23 cdw10:00000004 PRP1 0x0 PRP2 0x0
[2021-11-23 20:27:06.294618] nvme_qpair.c: 456:spdk_nvme_print_completion: *NOTICE*: INVALID FIELD (00/02) qid:0 cid:23 cdw0:0 sqhd:0008 p:1 m:0 dnr:1
get_feature(0x04) failed
[2021-11-23 20:27:06.295011] nvme_qpair.c: 251:nvme_admin_qpair_print_command: *NOTICE*: GET LOG PAGE (02) qid:0 cid:23 nsid:ffffffff cdw10:000f0001 cdw11:00000000 PRP1 0x1ac3bf000 PRP2 0x0
[2021-11-23 20:27:06.295070] nvme_qpair.c: 456:spdk_nvme_print_completion: *NOTICE*: INVALID OPCODE (00/01) qid:0 cid:23 cdw0:0 sqhd:000a p:1 m:0 dnr:1
get log page failed
[2021-11-23 20:27:06.295159] nvme_qpair.c: 251:nvme_admin_qpair_print_command: *NOTICE*: GET LOG PAGE (02) qid:0 cid:22 nsid:ffffffff cdw10:007f0002 cdw11:00000000 PRP1 0x1ac3be000 PRP2 0x0
[2021-11-23 20:27:06.295207] nvme_qpair.c: 456:spdk_nvme_print_completion: *NOTICE*: INVALID OPCODE (00/01) qid:0 cid:22 cdw0:0 sqhd:000b p:1 m:0 dnr:1
get log page failed
[2021-11-23 20:27:06.295264] nvme_qpair.c: 251:nvme_admin_qpair_print_command: *NOTICE*: GET LOG PAGE (02) qid:0 cid:21 nsid:ffffffff cdw10:007f0003 cdw11:00000000 PRP1 0x1ac3bd000 PRP2 0x0
[2021-11-23 20:27:06.295309] nvme_qpair.c: 456:spdk_nvme_print_completion: *NOTICE*: INVALID OPCODE (00/01) qid:0 cid:21 cdw0:0 sqhd:000c p:1 m:0 dnr:1
get log page failed
=====================================================
NVMe Controller at 0000:00:03.0 [8086:5845]
=====================================================
Controller Capabilities/Features
================================
Vendor ID:                             8086
Subsystem Vendor ID:                   8086
Serial Number:                         NVMe_vhost.2
Model Number:                          SPDK Virtual NVMe Controller
Firmware Version:                      18.04
Recommended Arb Burst:                 6
IEEE OUI Identifier:                   e4 d2 5c
Multi-path I/O
  May have multiple subsystem ports:   No
  May have multiple controllers:       No
  Associated with SR-IOV VF:           No
Max Data Transfer Size:                131072
Max Number of Namespaces:              1
NVMe Specification Version (VS):       1.0
NVMe Specification Version (Identify): 1.0
Maximum Queue Entries:                 256
Contiguous Queues Required:            Yes
Arbitration Mechanisms Supported
  Weighted Round Robin:                Not Supported
  Vendor Specific:                     Not Supported
Reset Timeout:                         500 ms
Doorbell Stride:                       4 bytes
NVM Subsystem Reset:                   Not Supported
Command Sets Supported
  NVM Command Set:                     Supported
Boot Partition:                        Not Supported
Memory Page Size Minimum:              4096 bytes
Memory Page Size Maximum:              4096 bytes
Persistent Memory Region:              Not Supported
Optional Asynchronous Events Supported
  Namespace Attribute Notices:         Not Supported
  Firmware Activation Notices:         Not Supported
128-bit Host Identifier:               Not Supported

Controller Memory Buffer Support
================================
Supported:                             No

Persistent Memory Region Support
================================
Supported:                             No

Admin Command Set Attributes
============================
Security Send/Receive:                 Not Supported
Format NVM:                            Not Supported
Firmware Activate/Download:            Not Supported
Namespace Management:                  Not Supported
Device Self-Test:                      Not Supported
Directives:                            Not Supported
NVMe-MI:                               Not Supported
Virtualization Management:             Not Supported
Doorbell Buffer Config:                Supported
Abort Command Limit:                   1
Async Event Request Limit:             1
Number of Firmware Slots:              N/A
Firmware Slot 1 Read-Only:             N/A
Firmware Update Granularity:           No Information Provided
Per-Namespace SMART Log:               No
Asymmetric Namespace Access Log Page:  Not Supported
Command Effects Log Page:              Not Supported
Get Log Page Extended Data:            Not Supported
Telemetry Log Pages:                   Not Supported
Error Log Page Entries Supported:      1
Keep Alive:                            Not Supported

NVM Command Set Attributes
==========================
Submission Queue Entry Size
  Max:                       64
  Min:                       64
Completion Queue Entry Size
  Max:                       16
  Min:                       16
Number of Namespaces:        1
Compare Command:             Not Supported
Write Uncorrectable Command: Not Supported
Dataset Management Command:  Supported
Write Zeroes Command:        Not Supported
Set Features Save Field:     Not Supported
Reservations:                Not Supported
Timestamp:                   Not Supported
Copy:                        Not Supported
Volatile Write Cache:        Not Present
Atomic Write Unit (Normal):  1
Atomic Write Unit (PFail):   1
Atomic Compare & Write Unit: 1
Fused Compare & Write:       Not Supported
Scatter-Gather List
  SGL Command Set:           Not Supported
  SGL Keyed:                 Not Supported
  SGL Bit Bucket Descriptor: Not Supported
  SGL Metadata Pointer:      Not Supported
  Oversized SGL:             Not Supported
  SGL Metadata Address:      Not Supported
  SGL Offset:                Not Supported
  Transport SGL Data Block:  Not Supported
Replay Protected Memory Block:  Not Supported

Firmware Slot Information
=========================
Active slot:                 0


Error Log
=========

Number of Queues
================
Number of I/O Submission Queues:      1
Number of I/O Completion Queues:      1

Active Namespaces
=================
[2021-11-23 20:27:06.296348] nvme_qpair.c: 241:nvme_admin_qpair_print_command: *NOTICE*: GET FEATURES ERROR_RECOVERY cid:21 cdw10:00000005 PRP1 0x0 PRP2 0x0
[2021-11-23 20:27:06.296404] nvme_qpair.c: 456:spdk_nvme_print_completion: *NOTICE*: INVALID FIELD (00/02) qid:0 cid:21 cdw0:0 sqhd:000d p:1 m:0 dnr:1
get_feature(0x05) failed
Namespace ID:1
Command Set Identifier:                NVM (00h)
Deallocate:                            Supported
Deallocated/Unwritten Error:           Not Supported
Deallocated Read Value:                Unknown
Deallocate in Write Zeroes:            Not Supported
Deallocated Guard Field:               0xFFFF
Flush:                                 Not Supported
Reservation:                           Not Supported
Namespace Sharing Capabilities:        Private
Size (in LBAs):                        937703088 (447GiB)
Capacity (in LBAs):                    937703088 (447GiB)
Utilization (in LBAs):                 937703088 (447GiB)
Thin Provisioning:                     Not Supported
Per-NS Atomic Units:                   No
NGUID/EUI64 Never Reused:              No
Number of LBA Formats:                 1
Current LBA Format:                    LBA Format #00
LBA Format #00: Data Size:   512  Metadata Size:     0