Re: [RFC 26/26] plugins/amdgpu: Hack to work around GPU virtual address conflicts

On 02/03/2026 19:58, Francis, David wrote:
> Patches 22, 23, and 24 are
> This patch is
> Reviewed-By: David Francis <[email protected]>
> 
> Patches 21, 25, and 26 somewhat depend on what the future plan is for this feature.
> Full support of no-kfd amdgpu with CRIU is going to involve changes to the kernel
> interfaces that will replace the code added in these patches. If these are allowing real
> workloads to complete that would otherwise fail, go ahead. But if they are meant to
> be first drafts, I might want to wait until the work can be done properly.

Yep, the tail of the series opens up many design questions. I am not 
suggesting to merge anything after 22, 23 the latest.

Some of the design questions, and you do not have to discuss them, I am 
only mentioning what needs solving to make this work:

Patch 21:
Probably acceptable on it's own, even without new kernel UAPIs, *if* we 
say "Lets allow users to force restore on a single GPU system". Because 
why not? I checkpoint on my system, I can restore on my system, and if 
things go bad so what. Probably gated behind a command line flag like 
--plugin-opt=amdgpu:force-single-gpu-restore.

Opens:

1. Does this sound acceptable?
2. Is there a way to pass plugin specific command line options?

Patches 23-25:

Patch 23 is not very useful on it's own. It's premise is that the 24 
(the topic of how to reliably handle VMAs) can be converted from a hack 
to something better. But I am not sure if that is the case. 25 is 
similarly a hack which may be avoided by rewriting the VMA handling 
completely.

Opens:

1. Is there even a way to allocate shared memory at the point of 
CR_PLUGIN_HOOK__RESTORE_EXT_FILE? It cannot be done via shmalloc and 
manually via mmap crashes the process.

2. Even if 1. is possible, could it handle all scenarios, or perhaps we 
need to re-think VMA handling more extensively.

For examples keying on offsets is not good enough if process has 
multiple GPU devices open. Some sort of the device id + offset would be 
required instead.

Or should we save a more complete view of the VMAs during checkpoint 
time rather than building the metadata during restore.

I will focus on the above questions next.

Patch 26:

Also a hack. I need to check if the order of restore operations can be 
shuffled so that it wouldn't be needed. For example can we afford to not 
map the BO in GPU VA space until after the content has been restored. 
That would avoid sdma_copy_bo having GPU VA conflicts. All objects would 
only be mapped as the last step.

If the above is not possible we will need a way to initialize libdrm's 
VA manager with a list of reserved ranges.

In closing, once the patches up to 21 or so are reviewed I can re-send 
that portion with a minor update or two. But the rest were just hacks to 
demonstrate what does not work and I will continue working on those 
until I have something more robust.

Regards,

Tvrtko

> ________________________________________
> From: Tvrtko Ursulin <[email protected]>
> Sent: Friday, February 20, 2026 7:05 AM
> To: [email protected]
> Cc: Francis, David; Tvrtko Ursulin
> Subject: [RFC 26/26] plugins/amdgpu: Hack to work around GPU virtual address conflicts
> 
> During the restore process buffer objects are created and mapped into the
> GPU virtual address space to their final addresses, and then their content
> is attempted to be restored.
> 
> This step is done by instantiating a libdrm instance and the associated
> userspace virtual address range manager. This manager is unaware of the
> previous address range allocations on the same file, which in practice
> means restore will simply fail.
> 
> To work around it we add some retries to try different addresses a few
> times, but for a proper solution we will need to either add a mechanism
> for reserving ranges in libdrm va manager, or rework the order of the
> restore operations so that the buffer objects are only mapped into their
> final locations once their content has been restored.
> 
> Signed-off-by: Tvrtko Ursulin <[email protected]>
> ---
>   plugins/amdgpu/amdgpu_plugin.c | 10 ++++++++++
>   1 file changed, 10 insertions(+)
> 
> diff --git a/plugins/amdgpu/amdgpu_plugin.c b/plugins/amdgpu/amdgpu_plugin.c
> index 239eeebc5111..b2f192a80f65 100644
> --- a/plugins/amdgpu/amdgpu_plugin.c
> +++ b/plugins/amdgpu/amdgpu_plugin.c
> @@ -573,6 +573,7 @@ int sdma_copy_bo(int shared_fd, uint64_t size, int storage_fd,
>          amdgpu_context_handle h_ctx;
>          uint32_t *ib = NULL;
>          int j, err, err2, packets_per_buffer;
> +       unsigned int retries = 0;
> 
>          buffer_bo_size = min(size, buffer_size);
>          packets_per_buffer = ((buffer_bo_size - 1) / max_copy_size) + 1;
> @@ -603,6 +604,8 @@ int sdma_copy_bo(int shared_fd, uint64_t size, int storage_fd,
>                  return -EINVAL;
>          }
> 
> +retry_src_va:
> +       retries++;
>          err = amdgpu_va_range_alloc(h_dev, amdgpu_gpu_va_range_general, src_bo_size, 0x1000, 0, &gpu_addr_src,
>                                      &h_va_src, 0);
>          if (err) {
> @@ -611,6 +614,8 @@ int sdma_copy_bo(int shared_fd, uint64_t size, int storage_fd,
>          }
>          err = amdgpu_bo_va_op(h_bo_src, 0, src_bo_size, gpu_addr_src, 0, AMDGPU_VA_OP_MAP);
>          if (err) {
> +               if (retries < 1000)
> +                       goto retry_src_va;
>                  pr_perror("failed to GPU map the src BO");
>                  goto err_src_bo_map;
>          }
> @@ -639,6 +644,9 @@ int sdma_copy_bo(int shared_fd, uint64_t size, int storage_fd,
>                  goto err_dst_bo_prep;
>          }
> 
> +       retries = 0;
> +retry_dst_va:
> +       retries++;
>          err = amdgpu_va_range_alloc(h_dev, amdgpu_gpu_va_range_general, dst_bo_size, 0x1000, 0, &gpu_addr_dst,
>                                      &h_va_dst, 0);
>          if (err) {
> @@ -647,6 +655,8 @@ int sdma_copy_bo(int shared_fd, uint64_t size, int storage_fd,
>          }
>          err = amdgpu_bo_va_op(h_bo_dst, 0, dst_bo_size, gpu_addr_dst, 0, AMDGPU_VA_OP_MAP);
>          if (err) {
> +               if (retries < 1000)
> +                       goto retry_dst_va;
>                  pr_perror("failed to GPU map the dest BO");
>                  goto err_dst_bo_map;
>          }
> --
> 2.52.0
>