[SPDK] How to do mmio correctly when vfio-pci is used?
oscar.huang at microchip.com
| Newsgroups | dev.linux.lists.spdk |
|---|---|
| Message-ID | <[email protected]> |
Hi, Gugus,
I have a simple SPDK application, which maps a device BAR, and write some data onto the device with spdk_mmio_write_8 function. The code flow is like this:
...
spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size);
addrToWrite = virBarAddr + offset; //offset is where I'd like to write data in the device memory
spdk_mmio_write_8(addrToWrite, data);
...
The code has been working with uio_pci_generic. But recently when I switched to vfio-pci (iommu enabled), I got a segmentation fault the program was executing that spdk_mmio_write_8 function.
If I do the following, the program can run:
...
spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size);
physAddr = phys +offset;
int fd = open("/dev/mem", O_RDWR | O_SYNC);
void *map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, physAddr & ~MAP_MASK);
addrToWrite = (uint64_t)map_base + (physAddr & MAP_MASK);
spdk_mmio_write_8(addrToWrite, data);
...
I don't understand why it is so. Basically I need a routine to map device physical address to process virtual address, and then call spdk_mmio_write_x functions. It looks like the virtual address from spdk_pci_device_map_bar cannot be used. I noticed the virtual address from spdk_pci_device_map_bar was 0x2010008000, but the one by mmap was 0x7ffff7fef000, very different.
Is there a unified mmio approach that works for both uio_pci_generic and vfio_pci?