[MODERATED] EPT/IOMMU shattering on SNB

Hello,

Testing has revealed an issue with the use of NX superpages in
combination with in-flight DMA.

It is possible that this is a bug in Xen, but we can only reproduce it
on SandyBridge E5 parts.

The scenario is running a graphics workload (Haven benchmark) in a VM
(Win7) with GPU Passthrough (Nvidia Quatro card in this case).  The EPT
tables are shared with the IOMMU.

What we observe is that userspace starting up results in a load of
shatters (as the VM is freshly booted at the time), and then we start
taking IOMMU faults (read/write access denied) against IO-virtual
addresses which have just been shattered.

I am now certain that R and W permissions and translations, are valid at
all times, so am at a loss to explain the IOMMU faults.  The fact that
we can't reproduce this with the same scenario on newer parts does
suggest that it might not be software related.

I know the IO-TLB on SNB is a little weird.  The IOMMU claims support
for superpages but doesn't implement them, and DMA through a 2M
superpage has about 40% lower throughput than through equivalent 4k
mappings.  We never adequately got to the bottom of this so Xen, like
Linux, still uses IO-superpages.

Avoiding sharing the EPT and IOMMU tables does avoid the problem, but
that is to be expected as the shattering activity now has no interaction
with the IOMMU.

Has anyone experimented with the above scenario, and if so, how has
testing gone?

Thanks,

~Andrew