[SPDK] Re: How to do mmio correctly when vfio-pci is used?

Hi Oscar,

Do you have an SPDK_PCI_DRIVER_REGISTER defined for your device?  This ensures the driver is properly registered with DPDK so that associated devices are configured correctly.  If not, lib/env_dpdk/ioat.c provides a suitable example.  This enables a unified mmio approach for SPDK.

-Jim


On 9/17/20, 12:17 AM, "oscar.huang(a)microchip.com" <oscar.huang(a)microchip.com> wrote:

    Hi, Gugus, 
    I have a simple SPDK application, which maps a device BAR, and write some data onto the device with spdk_mmio_write_8 function. The code flow is like this:
    ...
    spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size);
    addrToWrite = virBarAddr + offset;   //offset is where I'd like to write data in the device memory
    spdk_mmio_write_8(addrToWrite,  data);
    ...

    The code has been working with uio_pci_generic. But recently when I switched to vfio-pci (iommu enabled), I got a segmentation fault the program was executing that spdk_mmio_write_8 function. 

    If I do the following, the program can run:

    ...
    spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size);
    physAddr = phys +offset;
    int  fd = open("/dev/mem", O_RDWR | O_SYNC);
    void *map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, physAddr & ~MAP_MASK);
    addrToWrite = (uint64_t)map_base + (physAddr & MAP_MASK);
    spdk_mmio_write_8(addrToWrite, data);
    ...

    I don't understand why it is so. Basically I need a routine to map device physical address to process virtual address, and then call spdk_mmio_write_x functions. It looks like the virtual address from spdk_pci_device_map_bar cannot be used. I noticed the virtual address from spdk_pci_device_map_bar was 0x2010008000, but the one by mmap was 0x7ffff7fef000, very different. 

    Is there a unified mmio approach that works for both uio_pci_generic and vfio_pci?
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org