Re: [SPDK] [PATCH] vfio_user: fix the errors in 64K page kernel

Hello Shijie,

Thanks for the patch but we don't use the mailing list for posting patches.=
 Instead, we use Gerrit at https://review.spdk.io. You can read more about =
our development process at https://spdk.io/development.

Thanks,
Konrad

> -----Original Message-----
> From: SPDK <[email protected]> On Behalf Of Huang
> Shijie
> Sent: Friday, June 9, 2023 5:26 AM
> To: [email protected]
> Cc: Huang Shijie <[email protected]>;
> [email protected]
> Subject: [SPDK] [PATCH] vfio_user: fix the errors in 64K page kernel
>=20
> We will meet the followings errors in 64K page kernel:
>     " vfio_user.c:4454:nvmf_vfio_user_listen: *ERROR*: /tmp/nvme-vfio-
> user: error to mmap file /tmp/nvme-vfio-user/bar0: Invalid argument."
>     " vfio_user.c:3088:vfio_user_log: *ERROR*: /tmp/nvme-vfio-user:
> migration registers cannot be memory mapped"
>=20
> Fix them by
>     0.) Pass 0 to mmap's offset parameter.
> 	The doorbell offset is fixed at 0x1000 (kernel NVME driver uses it
> too).
> 	But mmap requires the offset must be a multiple of the page size as
> returned by
> 	sysconf(_SC_PAGE_SIZE).
> 	In 64K page size kernel will meet the failure. So set 0 to mmap's
> offset,
>        	and then change to doorbell offset manually.
>=20
>     1.) convert the hardcode to PAGE_SIZE.
>=20
> Tested this patch with cloud-hypervisor in 64K page size kernel.
>=20
> Signed-off-by: Huang Shijie <[email protected]>
> ---
>  lib/nvmf/vfio_user.c | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)
>=20
> diff --git a/lib/nvmf/vfio_user.c b/lib/nvmf/vfio_user.c index
> 2c05a72bd..22ade6f03 100644
> --- a/lib/nvmf/vfio_user.c
> +++ b/lib/nvmf/vfio_user.c
> @@ -1149,7 +1149,8 @@ nvmf_vfio_user_destroy_endpoint(struct
> nvmf_vfio_user_endpoint *endpoint)
>  	spdk_poller_unregister(&endpoint->accept_poller);
>=20
>  	if (endpoint->bar0_doorbells) {
> -		munmap((void *)endpoint->bar0_doorbells,
> NVMF_VFIO_USER_DOORBELLS_SIZE);
> +		endpoint->bar0_doorbells =3D (uint32_t*)(((unsigned
> long)endpoint->bar0_doorbells) - NVME_DOORBELLS_OFFSET);
> +		munmap((void *)endpoint->bar0_doorbells,
> +NVMF_VFIO_USER_DOORBELLS_SIZE + NVME_DOORBELLS_OFFSET);
>  	}
>=20
>  	if (endpoint->devmem_fd > 0) {
> @@ -4202,7 +4203,7 @@ vfio_user_dev_info_fill(struct
> nvmf_vfio_user_transport *vu_transport,
>=20
>  	vfu_setup_device_quiesce_cb(vfu_ctx, vfio_user_dev_quiesce_cb);
>=20
> -	migr_sparse_mmap.iov_base =3D (void *)4096;
> +	migr_sparse_mmap.iov_base =3D (void *)PAGE_SIZE;
>  	migr_sparse_mmap.iov_len =3D vfio_user_migr_data_len();
>  	ret =3D vfu_setup_region(vfu_ctx, VFU_PCI_DEV_MIGR_REGION_IDX,
>  			       vfu_get_migr_register_area_size() +
> vfio_user_migr_data_len(), @@ -4448,14 +4449,22 @@
> nvmf_vfio_user_listen(struct spdk_nvmf_transport *transport,
>  		goto out;
>  	}
>=20
> -	endpoint->bar0_doorbells =3D mmap(NULL, NVMF_VFIO_USER_DOORBELLS_SIZE,
> -					PROT_READ | PROT_WRITE, MAP_SHARED,
> endpoint->devmem_fd, NVME_DOORBELLS_OFFSET);
> +	/*
> +	 * The doorbell offset is fixed at 0x1000.
> +	 * But mmap requires the offset must be a multiple of the page size
> as returned by
> +	 * sysconf(_SC_PAGE_SIZE).
> +	 * In order to avoid the mmap failure in non-4K page size kernel,
> +	 * set 0 to mmap's offset, and then change to doorbell offset
> manually.
> +	 */
> +	endpoint->bar0_doorbells =3D mmap(NULL, NVMF_VFIO_USER_DOORBELLS_SIZE
> + NVME_DOORBELLS_OFFSET,
> +				PROT_READ | PROT_WRITE, MAP_SHARED, endpoint-
> >devmem_fd, 0);
>  	if (endpoint->bar0_doorbells =3D=3D MAP_FAILED) {
>  		SPDK_ERRLOG("%s: error to mmap file %s: %s.\n",
> endpoint_id(endpoint), path, spdk_strerror(errno));
>  		endpoint->bar0_doorbells =3D NULL;
>  		ret =3D -1;
>  		goto out;
>  	}
> +	endpoint->bar0_doorbells =3D (uint32_t *)(((unsigned
> +long)endpoint->bar0_doorbells) + NVME_DOORBELLS_OFFSET);
>=20
>  	ret =3D snprintf(path, PATH_MAX, "%s/migr", endpoint_id(endpoint));
>  	if (ret < 0 || ret >=3D PATH_MAX) {
> --
> 2.39.2
>=20
> _______________________________________________
> SPDK mailing list
> [email protected]
> https://lists.linuxfoundation.org/mailman/listinfo/spdk