[SPDK] Re: Registering other than NVMe device as a PCI memory buffer

> From: rmbhagwat(a)gmail.com <rmbhagwat(a)gmail.com>
> 
> It looks like in SPDK we can register only handful of devices (nvme, ioat, idxd,
> virtio, vmd).
> Is there any way to register Pure PCI Memory device (say 1 to 8 GB of DDR
> behind the PCIe)?

Is this memory exposed as a PCI BAR? Is there a driver for this device? You have two options:

1) If you have a kernel driver for the device, have that kernel driver expose the memory region as a file descriptor that can be passed to mmap
2) Write a userspace driver against vfio-pci. https://www.kernel.org/doc/Documentation/vfio.txt. This is how SPDK works.

The drivers in SPDK are all userspace drivers written against the vfio interface in Linux (we actually make DPDK calls which then uses vfio-pci). You can probably copy SPDK's lib/ioat or lib/idxd as a good template for writing a simple user space driver. Once the BAR is mapped into the process address space, you can call spdk_mem_register() on the address to configure it for DMA access and use the virtual addresses as input to calls like spdk_nvme_ns_cmd_write().

> My intention is to register the PCI memory and use it as a data buffer which will
> be passed to either spdk_nvme_ctrlr_cmd_io_raw() command or
> spdk_nvme_ns_cmd_read() or spdk_nvme_ns_cmd_write(). Basically instead of
> using Host DDR memory, I wish to use memory on PCIe. While doing so, I do not
> intend to use CMB as my SSDs do not have CMB.
> Please let me know if there is any possibility in SPDK. If yes, what would be the
> procedure?
> Thank you!