Re: [PATCH v6 0/8] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback

On Tue, Feb 10, 2026 at 05:30:10PM +0100, Niklas Cassel wrote:
> On Tue, Feb 10, 2026 at 11:07:16PM +0900, Koichiro Den wrote:
> > On Tue, Feb 10, 2026 at 01:24:29PM +0100, Niklas Cassel wrote:
> > > On Mon, Feb 09, 2026 at 09:53:08PM +0900, Koichiro Den wrote:
> > > > Tested on
> > > > =========
> > > > 
> > > > I tested the embedded (DMA) doorbell fallback path (via pci-epf-test) on
> > > > R-Car Spider boards:
> > > > 
> > > >   $ ./pci_endpoint_test -t DOORBELL_TEST
> > > >   TAP version 13
> > > >   1..1
> > > >   # Starting 1 tests from 1 test cases.
> > > >   #  RUN           pcie_ep_doorbell.DOORBELL_TEST ...
> > > >   #            OK  pcie_ep_doorbell.DOORBELL_TEST
> > > >   ok 1 pcie_ep_doorbell.DOORBELL_TEST
> > > >   # PASSED: 1 / 1 tests passed.
> > > >   # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
> > > > 
> > > > with the following message observed on the EP side:
> > > > 
> > > >   [   80.464653] pci_epf_test pci_epf_test.0: Using embedded (DMA) doorbell fallback
> > > > 
> > > > (Note: for the test to pass on R-Car Spider, one of the following was required:
> > > >  - echo 1048576 > functions/pci_epf_test/func1/pci_epf_test.0/bar2_size
> > > >  - apply https://lore.kernel.org/all/[email protected]/)
> > > 
> > > I applied this series on top of branch pci/controller/dwc
> > > on Rock 5B (pcie-dw-rockchip.c).
> > > 
> > > On EP side:
> > > [   39.218533] pci_epf_test pci_epf_test.0: Can't find MSI domain for EPC
> > > [   39.219125] pci_epf_test pci_epf_test.0: Using embedded (DMA) doorbell fallback
> > > 
> > > On RC side:
> > > #  RUN           pcie_ep_doorbell.DOORBELL_TEST ...
> > > [   40.297892] pci-endpoint-test 0000:01:00.0: Failed to trigger doorbell in endpoint
> > > # pci_endpoint_test.c:279:DOORBELL_TEST:Expected 0 (0) == ret (-22)
> > > # pci_endpoint_test.c:279:DOORBELL_TEST:Test failed for Doorbell
> > > 
> > > # DOORBELL_TEST: Test failed
> > > #          FAIL  pcie_ep_doorbell.DOORBELL_TEST
> > > not ok 23 pcie_ep_doorbell.DOORBELL_TEST
> > > 
> > > Any suggestions?
> > > 
> > > (All BARs in pcie-dw-rockchip.c is marked as BAR_RESIZABLE.)
> > 
> > Thank you for testing.
> > 
> > If the failure was observed in a scenario other than a plain
> > `./pci_endpoint_test -t DOORBELL_TEST`, could you please try again with [1]
> > applied as well?
> > 
> > [1] https://lore.kernel.org/linux-pci/[email protected]/
> 
> I applied that series, but I got the same problem.
> 
> I added debug, and the EP side does use the correct address for the eDMA:
> [   26.279457] msg_addr: 0xa403800a0
> [   26.279898] phys_addr: 0xa40300000 offset: 0x800a0
> 
> 
> If I write to the msg_addr directly on the EP using devmem, I do see the print
> that I added in the IRQ handler:
> # devmem 0xa403800a0 32 0
> [  155.861989] dw_edma_interrupt_emulated:696
> # devmem 0xa403800a0 32 0
> [  158.809160] dw_edma_interrupt_emulated:696
> [  158.809543] pci_epf_test_doorbell_primary:729
> # [  158.809986] pci_epf_test_doorbell_handler:703
> # devmem 0xa403800a0 32 0
> [  161.241326] dw_edma_interrupt_emulated:696
> # devmem 0xa403800a0 32 0
> [  163.466054] dw_edma_interrupt_emulated:696
> # devmem 0xa403800a0 32 0
> [  167.378662] dw_edma_interrupt_emulated:696
> [  167.379045] pci_epf_test_doorbell_primary:729
> # [  167.379512] pci_epf_test_doorbell_handler:703
> # devmem 0xa403800a0 32 0
> [  168.880179] dw_edma_interrupt_emulated:696
> # devmem 0xa403800a0 32 0
> [  170.492176] dw_edma_interrupt_emulated:696
> # devmem 0xa403800a0 32 0
> [  171.729154] dw_edma_interrupt_emulated:696
> # devmem 0xa403800a0 32 0
> [  173.481271] dw_edma_interrupt_emulated:696
> # devmem 0xa403800a0 32 0
> [  174.985787] dw_edma_interrupt_emulated:696
> # devmem 0xa403800a0 32 0
> [  176.517131] dw_edma_interrupt_emulated:696
> [  176.517511] pci_epf_test_doorbell_primary:729
> # [  176.517963] pci_epf_test_doorbell_handler:703
> 
> But not on every write....
> 
> I'm not sure, but could this perhaps be because we are missing this patch:
> https://lore.kernel.org/dmaengine/[email protected]/

Thank you for the detailed debugging.

I don't have a Rock 5B to reproduce locally, but your log indicates that
the emulated interrupt is not always delivered on the same eDMA IRQ line.
On RK3588 (rk3588-extra.dtsi) there are multiple eDMA IRQs (dma0-4), while
pci-epf-test requests only epf->db_msg[0].virq (IRQ 53 in your
/proc/interrupts). For R-Car S4 Spider, chip->nr_irqs == 1 and I wasn't
able to verify whether my earlier concern here:
https://lore.kernel.org/linux-pci/p4ommmpcjegvb4lafzecf67tzmdodtuqexeoifcn5eh7xqyp2y@ss76d3ubbsw7/

  > The proposed dmaengine_{register,unregister}_selfirq() APIs are
  > device-wide (i.e. not per channel), so I'm not sure which "channel" you
  > refer to here.  Also, when chip->nr_irqs > 1 on EP, dw-edma distributes
  > channels across multiple IRQ vectors, and it's unclear (at least to me)
  > which IRQ vector the emulated interrupt ("fake irq") is expected to be
  > delivered on.

actually holds true in practice. Your report makes it clear that the
emulated interrupt can indeed be delivered on different IRQ vectors.

One hypothesis is that: we currently program msg.data = 0 for the
"embedded" doorbell, and we write to DMA_READ_INT_STATUS_OFF. The register
field (RD_DONE_INT_STATUS) is defined per-channel, and the register
supports interrupt emulation by writes, so it might be that writing BIT(n)
selects the channel/irq line, while writing 0 does not consistently map to
a specific one.

Could you try a quick experiment on the Rock 5B EP side?

  devmem 0xa403800a0 32 1
  devmem 0xa403800a0 32 2
  devmem 0xa403800a0 32 4
  devmem 0xa403800a0 32 8

and see if the interrupt consistently lands on a specific one of IRQ53-56
for each value?

If that is the case, we can make msg.data non-zero value instead of always 0.

If that is not the case, then we may need to consider two less ideal
options:
  - switch back to the ~v3 approach, where we run the registered hook
    exactly at the time when the emulated interrupt is deasserted. (ref.
    https://lore.kernel.org/linux-pci/[email protected]/)
  - or, require users to request_irq() for all irq vectors associated with
    all channels. However, this would not be very attractive from a design
    perspective.

> 
> # dmesg | grep eDMA
> [    1.243339] rockchip-dw-pcie a40000000.pcie-ep: eDMA: unroll T, 2 wr, 2 rd
> 
> # cat /proc/interrupts | grep edma
>  53:          8          0          0          0          0          0          0          0    GICv3 303 Level     dw-edma-core:a40000000.pcie-ep, pci-ep-test-doorbell
>  54:          7          0          0          0          0          0          0          0    GICv3 304 Level     dw-edma-core:a40000000.pcie-ep
>  55:         15          0          0          0          0          0          0          0    GICv3 301 Level     dw-edma-core:a40000000.pcie-ep
>  56:          7          0          0          0          0          0          0          0    GICv3 302 Level     dw-edma-core:a40000000.pcie-ep
> 
> 
> 
> Anyway, I was still curious why this did never worked when writing from the
> host side, even when running the test case many, many times.
> AFAICT, the inbound translation looks correct.
> 
> RK3588 (pcie-dw-rockchip.c) exposes the DMA registers in BAR4 by default.
> If I hack pci-epf-test on top of your patch to unconditionally return BAR4 with
> offset 0xa0, it works. So my best guess is that the fixed inbound translation
> in BAR4 (to the eDMA registers) somehow messes with the inbound translation if
> another BAR tries to use an inbound translation to the eDMA registers as well.

Thanks a lot for letting me know that. I see two possible ways forward:

  (a) extend PCI_EPC_AUX_DMA_CTRL_MMIO to optionally describe that the DMA
      MMIO window is already mapped to a fixed BAR and should be reused, so
      EPFs avoid creating a second mapping to the same target. I guess it
      could be treated as a quirk for "rockchip,rk3588-pcie-ep".

  (b) alternatively, clear the default BAR4 mapping on RK3588 at least
      temporarily when using the pci-epf-msi doorbell fallback.

Koichiro

> 
> 
> Kind regards,
> Niklas
>