[RFC 00/26] amdgpu plugin render node only discussion

Hi all,

Long story short, I am trying to make some inroads towards having a working
checkpoint and restore for render node only applications (ie. no KFD).

Plan is to write test scenarios in an increasing order of complexity and make
them work. Those tests can be found here:

https://cgit.freedesktop.org/~tursulin/intel-gpu-tools/log/?h=amd-criu

For those unfamiliar, once built the list of subtest can be obtained like this:

$ amd_criu --l
open
one-bo
map-one
mmap-one
map-and-mmap-one
map-mmap-fork

Each test will pause after the setup phase waiting for a return key. This is the
point when they can be checkpointed and restored. The tests are in the order of
how I was fixing issues one by one, give or take some details.

Without this series nothing works, while with it, they all pass (last one is the
superset of all previous ones, or the most complex test):

$ amd_criu --r map-mmap-fork
...
$ sudo /usr/local/sbin/criu dump -t `pgrep amd_criu | head -1` -L /usr/local/lib/criu/ -vvv -o criu.log -j --link-remap --tcp-established --file-locks --ext-unix-sk
$ sudo /usr/local/sbin/criu restore  -L /usr/local/lib/criu/ -vvv -o restore.log --shell-job --link-remap --tcp-established --file-locks --ext-unix-sk

Subtest map-mmap-fork: SUCCESS (6.310s)

However trivial the tests may currently be, the combination of them and this
series is supposed to be a starting point for a design discussion since it
appears there are many issues to address.

Having said that, this series actually starts with a fair number of cleanup
patches which can be reviewed and merged independently. Or I can easily send
them as a standalone pull request if that would work better.

Only the last six patches, plus "plugins/amdgpu: Add plugin to inventory even if
process has no vmas", are the ones with fixes and hacks which make the tests
pass.

In no particular order the main issues I currently see are:

 * GPU discovery with no KFD open or present
 * Rendernode VMA handling
 * VMA handling with forked processes
 * GPU VA conflicts during restore

Some of these may require changes to CRIU plugin parameters and input from
CRIU experts.

More details on each of these topics can be found in the respective patch commit
messages. I will also gladly expand on any.

For now I wanted to test the waters first by sending this early and see who is
interested and to what extent. This is far from a working render node support
but it illustrates that such basic scenarios currently do not work, that I think
those basics need to be fixed first before proceeding with anything more
complex.

Also, nothing in this series is supposed to negatively influence the KFD paths,
but with a disclaimer that I did not test that side at all. Suggestions on a
suitable KFD test program would be welcomed.

Tvrtko

Tvrtko Ursulin (26):
  plugins/amgdpu: Fix one error message
  plugins/amdgpu: Remove unused current_pid global variable
  plugins/amdgpu: Remove unused new_minor from struct vma_metadata
  plugins/amdgpu: Fix drm pages size header
  plugins/amdgpu: Fix logging of failures to open files during restore
    init
  plugins/amdgpu: Propagate failure to save buffer object content
  plugins/amdgpu: Close the directory when image probing fails
  plugins/amdgpu: Close dma-buf image file if the read fails
  plugins/amdgpu: Check images were opened successfully
  plugins/amdgpu: Flatten amdgpu_restore_init a bit
  plugins/amdgpu: Fix error handling in amdgpu_plugin_drm_restore_file()
  plugins/amdgpu: Add error handling for seek operations
  plugins/amdgpu: Consolidate vm_info collection
  plugins/amdgpu: Remove plugin_log_msg()
  plugins/amdgpu: Add plugin to inventory even if process has no vmas
  plugins/amdgpu: Move drm file dump and restore into helpers
  plugins/amdgpu: Use the load_img helper in drm file restore
  plugins/amdgpu: Convert away from libc buffered file IO
  plugins/amdgpu: Use save_vma_updates for all call sites
  plugins/amdgpu: amdgpu_plugin_drm_restore_file() does not need to use
    libdrm
  plugins/amdgpu: Enable restore with only the drm node open
  plugins/amdgpu: Handle DRM render node mmaps
  plugins/amdgpu: Consolidate shared memory handling
  plugins/amdgpu: Store VMA data in shared memory
  plugins/amdgpu: Hack to save a fd number which works for vma restore
  plugins/amdgpu: Hack to work around GPU virtual address conflicts

 plugins/amdgpu/amdgpu_plugin.c          | 856 ++++++++++++++----------
 plugins/amdgpu/amdgpu_plugin_dmabuf.c   |  17 +-
 plugins/amdgpu/amdgpu_plugin_drm.c      | 125 ++--
 plugins/amdgpu/amdgpu_plugin_drm.h      |   5 +-
 plugins/amdgpu/amdgpu_plugin_topology.c |   2 +
 plugins/amdgpu/amdgpu_plugin_topology.h |   2 +
 plugins/amdgpu/amdgpu_plugin_util.c     | 106 +--
 plugins/amdgpu/amdgpu_plugin_util.h     |  17 +-
 8 files changed, 636 insertions(+), 494 deletions(-)

-- 
2.52.0