Re: [PATCH v2] net: airoha: npu: use coherent DMA for mailbox messages

On Aug 06, Daniel Pawlik wrote:
> Commit 6f884eb87a79 ("net: airoha: Fix DMA direction for NPU mailbox
> buffer") switched airoha_npu_send_msg() to DMA_BIDIRECTIONAL so
> non-coherent CPUs invalidate caches before reading NPU GET responses.
> 
> On EN7581 + MT7996 that change regresses probe: the mailbox completes
> successfully, but WLAN_FUNC_GET_WAIT_NPU_VERSION still reads as 0.0 and
> mt76 never binds NPU offload. Healthy boards report 0.1111.
> 
> airoha_npu_send_msg() is also used from PPE foe_commit under
> spin_lock_bh(), so per-message dma_alloc_coherent(GFP_ATOMIC) is a poor
> fit. Allocate one device-managed coherent bounce buffer at probe and
> reuse it under the existing per-core mailbox lock. That also keeps the
> buffer valid if the NPU completes a write after a mailbox timeout.
> 
> Verified on Quantum Fiber / Gemtek W1700K (EN7581 + MT7996).

Hi Daniel,

I tested the issue you reported on a very similar hw (reported below) and it
works fine for me
- EN7581-mmc
- MT7996 SoC

@Christian: Is the issue occurring for you? I guess we should debug it a bit
more.

> 
> Fixes: 6f884eb87a79 ("net: airoha: Fix DMA direction for NPU mailbox buffer")
> Link: https://patchwork.kernel.org/project/linux-mediatek/patch/[email protected]/
> Assisted-by: Cursor:composer-2
> Signed-off-by: Daniel Pawlik <[email protected]>
> ---
> v2:
> - allocate one coherent mailbox buffer at probe and reuse it under the
>   per-core lock instead of per-message dma_alloc_coherent(GFP_ATOMIC)
> 
>  drivers/net/ethernet/airoha/airoha_npu.c  | 31 +++++++++++++++++------
>  include/linux/soc/airoha/airoha_offload.h |  4 +++
>  2 files changed, 27 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/net/ethernet/airoha/airoha_npu.c b/drivers/net/ethernet/airoha/airoha_npu.c
> index b679bed952de..dfa09fed907e 100644
> --- a/drivers/net/ethernet/airoha/airoha_npu.c
> +++ b/drivers/net/ethernet/airoha/airoha_npu.c
> @@ -23,6 +23,8 @@
>  #define NPU_EN7581_FIRMWARE_RV32_MAX_SIZE	0x200000
>  #define NPU_EN7581_FIRMWARE_DATA_MAX_SIZE	0x10000
>  #define NPU_DUMP_SIZE				512
> +/* Enough for struct ppe_mbox_data and small WLAN TLV payloads */
> +#define AIROHA_NPU_MBOX_SIZE			256

Are you sure this is enough for all possible messages?

>  
>  #define REG_NPU_LOCAL_SRAM		0x0
>  
> @@ -165,17 +167,24 @@ static int airoha_npu_send_msg(struct airoha_npu *npu, int func_id,
>  {
>  	u16 core = 0; /* FIXME */
>  	u32 val, offset = core << 4;
> -	dma_addr_t dma_addr;
>  	int ret;
>  
> -	dma_addr = dma_map_single(npu->dev, p, size, DMA_BIDIRECTIONAL);
> -	ret = dma_mapping_error(npu->dev, dma_addr);
> -	if (ret)
> -		return ret;
> +	if (size > AIROHA_NPU_MBOX_SIZE)
> +		return -EINVAL;
>  
> +	/*
> +	 * Mailbox payloads are bidirectional (CPU request, NPU response).
> +	 * On EN7581+MT7996, streaming DMA_BIDIRECTIONAL against the
> +	 * caller kzalloc() buffer can leave WLAN_FUNC_GET_WAIT_NPU_VERSION
> +	 * reading as 0.0 despite MBOX success. Reuse a probe-time coherent
> +	 * bounce buffer under the per-core lock (also used from PPE
> +	 * foe_commit under atomic context).
> +	 */
>  	spin_lock_bh(&npu->cores[core].lock);
>  
> -	regmap_write(npu->regmap, REG_CR_MBQ0_CTRL(0) + offset, dma_addr);
> +	memcpy(npu->mbox_buf, p, size);

Even if it is a just a theoretical issue (we just use core 0 at the moment), we
should have a per-core buffer since we can theoretically have concurrent messages
on different cores.

> +
> +	regmap_write(npu->regmap, REG_CR_MBQ0_CTRL(0) + offset, npu->mbox_dma);
>  	regmap_write(npu->regmap, REG_CR_MBQ0_CTRL(1) + offset, size);
>  	regmap_read(npu->regmap, REG_CR_MBQ0_CTRL(2) + offset, &val);
>  	regmap_write(npu->regmap, REG_CR_MBQ0_CTRL(2) + offset, val + 1);
> @@ -189,9 +198,10 @@ static int airoha_npu_send_msg(struct airoha_npu *npu, int func_id,
>  	if (!ret && FIELD_GET(MBOX_MSG_STATUS, val) != NPU_MBOX_SUCCESS)
>  		ret = -EINVAL;
>  
> -	spin_unlock_bh(&npu->cores[core].lock);
> +	if (!ret)
> +		memcpy(p, npu->mbox_buf, size);

I do not like this double copy here. I guess we could pass a pointer for the
reply buffer to airoha_npu_send_msg() and avoid the copy in
airoha_npu_wlan_msg_get(). What do you think?

>  
> -	dma_unmap_single(npu->dev, dma_addr, size, DMA_BIDIRECTIONAL);
> +	spin_unlock_bh(&npu->cores[core].lock);
>  
>  	return ret;
>  }
> @@ -770,6 +780,11 @@ static int airoha_npu_probe(struct platform_device *pdev)
>  	if (err)
>  		return err;
>  
> +	npu->mbox_buf = dmam_alloc_coherent(dev, AIROHA_NPU_MBOX_SIZE,
> +					    &npu->mbox_dma, GFP_KERNEL);
> +	if (!npu->mbox_buf)
> +		return -ENOMEM;
> +
>  	err = airoha_npu_run_firmware(dev, base, &res);
>  	if (err)
>  		return dev_err_probe(dev, err, "failed to run npu firmware\n");
> diff --git a/include/linux/soc/airoha/airoha_offload.h b/include/linux/soc/airoha/airoha_offload.h
> index 7589fccfeef6..afd481233489 100644
> --- a/include/linux/soc/airoha/airoha_offload.h
> +++ b/include/linux/soc/airoha/airoha_offload.h
> @@ -179,6 +179,10 @@ struct airoha_npu {
>  
>  	struct airoha_foe_stats __iomem *stats;
>  
> +	/* Coherent bounce buffer for mailbox cmd/rsp (airoha_npu_send_msg) */
> +	void *mbox_buf;
> +	dma_addr_t mbox_dma;

you need to move them in airoha_npu_core struct. Please rename them to:

	struct airoha_npu_core {
		...
		void *buf;
		dma_addr_t addr;
		...
	};

Regards,
Lorenzo

> +
>  	struct {
>  		int (*ppe_init)(struct airoha_npu *npu);
>  		int (*ppe_deinit)(struct airoha_npu *npu);
> -- 
> 2.55.0
>

-----BEGIN PGP SIGNATURE-----

iHUEABYKAB0WIQTquNwa3Txd3rGGn7Y6cBh0uS2trAUCanQ/jwAKCRA6cBh0uS2t
rEesAP9YwI3B8KcL0t7IuGBi4EsnYjyf9b2tfEZ1b4GfKQTmuwEAvsXTkH8KLaTN
02bD6Xwo5rgejGryitnWS/humdqftwM=
=Fp5W
-----END PGP SIGNATURE-----