busdma proposal - address coherency, barriers and clarify situations where device/host access happens simultaneously

hi!

mmel@ and i have been poking at a proposal to clean up how busdma
handles coherency, barriers and device/host simultaneous access.

The proposal is here -
https://wiki.freebsd.org/AdrianChadd/BusDmaProposal2026 . We'd
appreciate some feedback. mmel@ has an in progress diff for it on
arm64 and I'll poke him about sharing that soon.

I owe the document some specific examples for specific classes of
drivers on specific platforms and I'll work on that over the next
week.

The "short" of it is this - the current concept/API of busdma is
really geared towards device and host DMA access that takes turns
owning an allocation of memory. For a lot of devices this is true -
either the host owns something, or the device owns something, but
never both. But plenty of devices - especially those whose existence
began on x86 or ppc64 with default coherency in hardware - treat
memory regions as simultaneously accessed by the host AND device. This
popped up (again) when working on the if_rge port - the device is
reading/writing into a contiguous ringbuffer of descriptors at the
same time the host is reading/writing, and the current barrier /
pre+post sync operations don't map well to that. Bounce buffers throw
an even larger wrench in this.

So our proposal is to clarify what bus coherency and memory allocation
coherency means and include two new sync calls specifically designed
as incremental memory ordering / flushing barriers. It notably
clarifies that we will prefer uncached memory behaviour over potential
impossible situations involving cache invalidation / bounce buffers
with cached memory due to the issues surrounding that (which were very
prevalent on MIPS, and are amusingly still prevalent on
ARM/ARM64/RISC-V.)

This does mean that a driver written to use busdma "correctly" in this
proposal should work correctly on both coherent and non-coherent
systems without the driver needing its own barrier/fence/cache
operations and either a partial or non use of busdma. It may not be
the fastest available on a given system, but it will be "correct".

My hope is that once we've landed this, converted drivers and get some
further experience in where the performance issues are, we can
document better best practices and mutate the API in very specific
ways to restore whatever weird side effects may occur.

We'd appreciate feedback!

Thanks,


-adrian