[SPDK] How to do mmio correctly when vfio-pci is used?

oscar.huang at microchip.com
Newsgroups dev.linux.lists.spdk
Message-ID <[email protected]>
Hi, Gugus, 
I have a simple SPDK application, which maps a device BAR, and write some data onto the device with spdk_mmio_write_8 function. The code flow is like this:
...
spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size);
addrToWrite = virBarAddr + offset;   //offset is where I'd like to write data in the device memory
spdk_mmio_write_8(addrToWrite,  data);
...

The code has been working with uio_pci_generic. But recently when I switched to vfio-pci (iommu enabled), I got a segmentation fault the program was executing that spdk_mmio_write_8 function. 

If I do the following, the program can run:

...
spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size);
physAddr = phys +offset;
int  fd = open("/dev/mem", O_RDWR | O_SYNC);
void *map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, physAddr & ~MAP_MASK);
addrToWrite = (uint64_t)map_base + (physAddr & MAP_MASK);
spdk_mmio_write_8(addrToWrite, data);
...

I don't understand why it is so. Basically I need a routine to map device physical address to process virtual address, and then call spdk_mmio_write_x functions. It looks like the virtual address from spdk_pci_device_map_bar cannot be used. I noticed the virtual address from spdk_pci_device_map_bar was 0x2010008000, but the one by mmap was 0x7ffff7fef000, very different. 

Is there a unified mmio approach that works for both uio_pci_generic and vfio_pci?
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.