[PATCH v4 00/24] dmaengine: dw-edma: Support dynamic LL appends
Koichiro Den <[email protected]> Wed, 29 Jul 2026 23:30:12 +0900
| Newsgroups | org.kernel.vger.dmaengine,org.kernel.vger.linux-kernel |
|---|---|
| Message-ID | <[email protected]> |
Hi, This is v4 of the dw-edma dynamic-append work originally proposed by Frank Li in his RFT: https://lore.kernel.org/r/[email protected]/ The driver currently treats each linked list as a finite transfer and waits for completion before publishing more work. This series turns the list into a circular producer/consumer ring and lets LL channels accept new requests while running. The motivation and hardware observations are unchanged since v2, linked below. v4 mainly addresses feedback from Frank and Sashiko. It adds no new patches. With multiple DMAengine descriptors in one ring, entry reclamation and descriptor completion need separate boundaries. The series records IRQ-paired LLP samples, uses a conservative running boundary for both eDMA and HDMA, and centralizes when a stopped channel needs another doorbell. The recovery patches add an opt-in safeguard for eDMA with direction reset operations. On one integration, sustained multi-channel traffic could leave a channel stopped with pending entries. Repeated doorbells did not restart it, while toggling ENGINE_EN did. Doorbell timing remains the leading suspect: under heavy multi-channel load, channel-status updates visible to software appear to lag internal start/stop transitions. That said, the exact trigger is still unknown. Any insight from Synopsys would be appreciated. Recovery was not entered in the fio/nvmet_pci_epf tests below. Base ==== This series is based on dmaengine/next. Re-testing with v4 ================== I re-tested v4 on the following three testbeds: - eDMA: RK3588 Rock 5B, DWC PCIe v5.60a R-Car S4, DWC PCIe v5.20a - HDMA: SpacemiT K3, DWC PCIe v6.30a [1] 1. fio The fio setup uses nvmet_pci_epf with a null_blk-backed namespace, runtime=30s, ramp_time=5s. The "Before" values reuse the baseline measurements reported with v2. The "After" values are three consecutive v4 runs. NOTE: The fio tests use nvmet_pci_epf, which does not keep the DMA engine continuously busy. A workload that rarely lets the engine stop would show the benefit of dynamic appends more directly. - eDMA (RK3588) NOTE: The eDMA Before results include the separate regression described in [2]. Summary by group (BW delta %) all n=26 mean=+1129.2 median=+114.6 min= +38.4 max=+5346.2 read n=14 mean=+1996.5 median=+859.9 min= +54.9 max=+5346.2 write n=11 mean=+102.3 median=+107.7 min= +38.4 max=+168.2 qd32 n=16 mean=+1775.0 median=+166.5 min= +67.6 max=+5346.2 q1 n= 9 mean= +75.2 median= +66.4 min= +38.4 max=+131.2 small 4K n= 6 mean=+102.7 median= +88.7 min= +38.4 max=+168.2 large >=128K n=20 mean=+1437.1 median=+120.7 min= +52.2 max=+5346.2 Before mean -> After mean (MiB/s) Case Before After Delta --------------------------- ----------------- ----------------- ------ Rnd read 4KB q1 1j 19.2 (sd 7.5) 36.7 (sd 13.5) +90.6% Rnd read 4KB q32 1j 88.7 (sd 2.1) 148.7 (sd 54.5) +67.6% Rnd read 4KB q32 4j 88.7 (sd 1.3) 165.6 (sd 84.0) +86.7% Rnd read 128KB q1 1j 420.0 (sd 134.6) 719.0 (sd 184.5) +71.2% Rnd read 128KB q32 1j 142.0 (sd 37.7) 2544.3 (sd 132.2) +1691.8% Rnd read 128KB q32 4j 63.7 (sd 10.6) 3020.0 (sd 29.7) +4643.5% Rnd read 512KB q1 1j 572.0 (sd 35.8) 952.0 (sd 44.9) +66.4% Rnd read 512KB q32 1j 66.8 (sd 10.8) 3007.7 (sd 38.2) +4402.5% Rnd read 512KB q32 4j 55.4 (sd 12.0) 3019.0 (sd 39.0) +5346.2% Rnd write 4KB q1 1j 20.7 (sd 7.0) 28.7 (sd 15.9) +38.4% Rnd write 4KB q32 1j 42.5 (sd 0.9) 112.4 (sd 17.8) +164.7% Rnd write 4KB q32 4j 41.7 (sd 1.6) 111.8 (sd 18.8) +168.2% Rnd write 128KB q1 1j 330.0 (sd 90.1) 542.3 (sd 91.2) +64.3% Rnd write 128KB q32 1j 624.0 (sd 111.5) 1309.3 (sd 60.4) +109.8% Rnd write 128KB q32 4j 659.3 (sd 61.8) 1292.7 (sd 59.2) +96.1% Seq read 128KB q1 1j 260.7 (sd 25.8) 602.7 (sd 85.0) +131.2% Seq read 128KB q32 1j 149.7 (sd 30.9) 2527.3 (sd 13.0) +1588.6% Seq read 512KB q1 1j 578.0 (sd 32.2) 895.3 (sd 24.6) +54.9% Seq read 512KB q32 1j 65.0 (sd 10.6) 3017.0 (sd 34.9) +4541.5% Seq read 1MB q32 1j 57.3 (sd 18.8) 3018.7 (sd 30.4) +5168.2% Seq write 128KB q1 1j 272.3 (sd 62.0) 565.7 (sd 111.2) +107.7% Seq write 128KB q32 1j 597.0 (sd 105.7) 1309.3 (sd 62.1) +119.3% Seq write 512KB q1 1j 476.7 (sd 26.7) 725.7 (sd 32.7) +52.2% Seq write 512KB q32 1j 710.7 (sd 44.0) 1297.3 (sd 57.9) +82.6% Seq write 1MB q32 1j 546.0 (sd 100.4) 1213.0 (sd 56.3) +122.2% Rnd rdwr 4K..1MB q8 4j 247.3 (sd 16.3) 943.3 (sd 88.5) +281.4% - HDMA (SpacemiT K3) Summary by group (BW delta %) all n=26 mean= +85.6 median=+100.1 min= -26.9 max=+154.7 read n=14 mean= +76.9 median= +84.2 min= -26.9 max=+144.0 write n=11 mean= +92.0 median=+137.5 min= -24.5 max=+154.7 qd32 n=16 mean=+112.0 median=+106.1 min= +71.8 max=+145.6 q1 n= 9 mean= +33.0 median= +4.7 min= -26.9 max=+154.7 small 4K n= 6 mean= +76.9 median=+101.8 min= -19.5 max=+144.0 large >=128K n=20 mean= +88.2 median=+100.1 min= -26.9 max=+154.7 Before mean -> After mean (MiB/s) Case Before After Delta --------------------------- ----------------- ----------------- ------ Rnd read 4KB q1 1j 52.5 (sd 3.5) 55.3 (sd 4.4) +5.3% Rnd read 4KB q32 1j 108.1 (sd 90.9) 263.7 (sd 7.6) +144.0% Rnd read 4KB q32 4j 117.0 (sd 84.9) 267.0 (sd 8.9) +128.1% Rnd read 128KB q1 1j 835.7 (sd 139.6) 611.0 (sd 171.4) -26.9% Rnd read 128KB q32 1j 802.7 (sd 67.7) 1623.7 (sd 13.1) +102.3% Rnd read 128KB q32 4j 931.3 (sd 262.1) 1666.3 (sd 10.4) +78.9% Rnd read 512KB q1 1j 411.0 (sd 144.0) 895.0 (sd 90.4) +117.8% Rnd read 512KB q32 1j 969.0 (sd 210.7) 1664.3 (sd 10.1) +71.8% Rnd read 512KB q32 4j 889.7 (sd 298.0) 1679.7 (sd 12.5) +88.8% Rnd write 4KB q1 1j 46.0 (sd 15.5) 37.1 (sd 18.7) -19.5% Rnd write 4KB q32 1j 117.7 (sd 4.0) 232.3 (sd 10.1) +97.5% Rnd write 4KB q32 4j 113.0 (sd 5.3) 233.0 (sd 9.5) +106.2% Rnd write 128KB q1 1j 606.7 (sd 149.0) 458.3 (sd 20.2) -24.5% Rnd write 128KB q32 1j 533.6 (sd 512.3) 1267.3 (sd 2.1) +137.5% Rnd write 128KB q32 4j 526.0 (sd 514.9) 1266.7 (sd 2.3) +140.8% Seq read 128KB q1 1j 626.3 (sd 247.5) 655.7 (sd 90.8) +4.7% Seq read 128KB q32 1j 783.3 (sd 51.1) 1613.3 (sd 12.7) +106.0% Seq read 512KB q1 1j 464.0 (sd 191.5) 918.3 (sd 18.0) +97.9% Seq read 512KB q32 1j 918.3 (sd 241.8) 1640.0 (sd 35.5) +78.6% Seq read 1MB q32 1j 918.7 (sd 290.5) 1649.3 (sd 42.7) +79.5% Seq write 128KB q1 1j 611.0 (sd 79.8) 531.7 (sd 128.7) -13.0% Seq write 128KB q32 1j 521.8 (sd 515.3) 1266.0 (sd 2.6) +142.6% Seq write 512KB q1 1j 328.3 (sd 268.7) 836.3 (sd 34.7) +154.7% Seq write 512KB q32 1j 519.6 (sd 516.6) 1266.7 (sd 1.5) +143.8% Seq write 1MB q32 1j 515.1 (sd 517.8) 1265.3 (sd 2.1) +145.6% Rnd rdwr 4K..1MB q8 4j 355.3 (sd 64.7) 840.3 (sd 3.1) +136.5% 2. pci_endpoint_test READ_TEST and WRITE_TEST passed with v4 on all three testbeds. 3. R-Car S4 (PCIe Gen4 x2, DWC PCIe controller v5.20a, eDMA) stress The non-upstream NTB transport/netdev saturated four HW channels in each direction concurrently from multiple QPs. With v4, it still reached EP->RC 18.7 Gbit/s and RC->EP 15.7 Gbit/s with iperf3 -ub0, while keeping the DMA controller almost continuously busy. [1] K3 was tested on an Ubuntu kernel with the prerequisite patches backported (+ some tweaks on top), not directly on the upstream base. The "Before" and "After" kernels differ only by this series. [2] The current base includes commit 35de39e4511f ("dmaengine: dw-edma: Defer channel IRQ handling to workqueue"), which defers channel IRQ handling to an unbound workqueue. Under CPU pressure, some workloads can see delayed completion processing and idle gaps between descriptors. Dynamic appends largely hide those gaps by keeping the engine fed. A BH workqueue would improve the nvmet_pci_epf workload, but it is not a complete replacement because BH workqueues are per-CPU. Retaining cross-channel CPU distribution would require the driver to choose target CPUs and account for housekeeping masks and CPU hotplug. That is too much complexity for a quick fix, so I have not posted a separate change. This partly explains the unusually large eDMA gain in these tests. Best regards, Koichiro --- Changes in v4: - Use spinlock_t for LL event serialization, select the event lock through a per-channel pointer, and use the common accessor in both interrupt providers. (Frank) - Put the remote LL publication read-back back in the high-level doorbell helper, while keeping the provider callback as a raw doorbell write. (Frank) - Arm the stopped-tail recheck after a stopped channel is kicked, reserve its zero sentinel across jiffies wrap, and keep the recheck available without optional direction recovery. (Sashiko) - Keep retrying failed direction resets with cancelable delayed work instead of giving up after a fixed number of attempts. (Sashiko) - Other smaller fixes, patch-ordering changes, and wording updates are listed in the per-patch changelogs. (Frank, Sashiko) Changes in v3: - Main changes: * Serialize IRQ event capture and ABORT handling with channel restart, so deferred events cannot cross into a later hardware run. (Sashiko) * Record each descriptor's physical LL range and validate it before consuming IRQ-paired progress. Keep ring accounting valid across termination, abort, and full resynchronization. (Frank, Sashiko) * Rework stopped-tail and STOP/PAUSE handling: re-sample LLP before restart, add one bounded eDMA recheck, and drain published entries before completing a request. (Frank, Sashiko) * Move LL interrupt placement into the common core and use the same four-entry interval for eDMA and HDMA. (Frank) * Restrict recovery to providers with usable direction-reset operations and serialize it with IRQ/ABORT handling and teardown. Rebuild all exposed LL contexts and flush remote LL writes before re-enabling the engine. (Sashiko) - Many other smaller fixes, refinements, and patch reorganizations came from the v2 review, especially Sashiko's feedback. Each patch has its own changelog below the --- separator, so the details are not repeated here. Changes in v2: - v2 is a substantial rework; per-patch changelogs contain the details: * Rebased onto dmaengine/next. * Moved the remaining v1 fixes and state serialization changes into the independent groundwork series, now merged in the base. * Reworked ring progress around IRQ-paired LLP samples, conservative running boundaries, and stopped-tail reconciliation. * Reworked STOP/PAUSE handling and added full ring resynchronization plus optional direction recovery before enabling dynamic submission. * Updated trace support for the reworked paths and added engine recovery tracing. v3: https://lore.kernel.org/r/[email protected]/ v2: https://lore.kernel.org/r/[email protected]/ v1: https://lore.kernel.org/r/[email protected]/ Frank Li (5): dmaengine: dw-edma: Add dw_edma_core_ll_cur_idx() to get current LL entry index dmaengine: dw-edma: Make DMA link list work as a circular buffer dmaengine: dw-edma: Move callback result helper before LL helpers dmaengine: dw-edma: Dynamically append requests while running dmaengine: dw-edma: Add trace support Koichiro Den (19): dmaengine: dw-edma: Add dw_edma_core_ll_clear() to clear LL control-word dmaengine: dw-edma: Factor out linked-list transfer start dmaengine: dw-edma: Dispatch DONE interrupts by channel request dmaengine: dw-edma: Centralize LL doorbell decisions dmaengine: dw-edma: Prepare LL progress event handling dmaengine: dw-edma: Prepare deferred IRQ reporting for LL events dmaengine: dw-edma: Prepare LL kicks for event serialization dmaengine: dw-edma: Serialize LL event capture with channel kicks dmaengine: dw-edma: Keep channels stopped while ABORT is pending dmaengine: dw-edma: Reclaim issued descriptors from IRQ-paired LL progress dmaengine: dw-edma: Add LL interrupt placement policy dmaengine: dw-edma: Recheck stopped LL channels before restart dmaengine: dw-edma: Use HDMA watermarks as progress events dmaengine: dw-edma: Recover stopped channels from tx_status() dmaengine: dw-edma: Make the LL ring reset a full channel resync dmaengine: dw-edma: Drain LL entries for STOP and PAUSE dmaengine: dw-edma: Add engine reset and enable operations dmaengine: dw-edma: Add engine recovery infrastructure dmaengine: dw-edma: Detect and recover a stalled eDMA engine drivers/dma/dw-edma/Makefile | 2 + drivers/dma/dw-edma/dw-edma-core.c | 1574 ++++++++++++++++++++++--- drivers/dma/dw-edma/dw-edma-core.h | 159 ++- drivers/dma/dw-edma/dw-edma-trace.c | 4 + drivers/dma/dw-edma/dw-edma-trace.h | 169 +++ drivers/dma/dw-edma/dw-edma-v0-core.c | 201 +++- drivers/dma/dw-edma/dw-hdma-v0-core.c | 163 ++- drivers/dma/dw-edma/dw-hdma-v0-regs.h | 1 + include/linux/dma/edma.h | 2 +- 9 files changed, 2010 insertions(+), 265 deletions(-) create mode 100644 drivers/dma/dw-edma/dw-edma-trace.c create mode 100644 drivers/dma/dw-edma/dw-edma-trace.h base-commit: 0613e7934ee233d726af8d8c89f251a1c4df738d -- 2.51.0