Re: [RFC PATCH v2 1/1] drm/xe: keep VM-bound WC BOs resident during reclaim

On Thu, Jul 30, 2026 at 07:44:02PM -0700, Matthew Brost wrote:
> So I posted another rev priority changes to account WB vs WB/UC:
>
> [7] https://patchwork.freedesktop.org/series/170454/
> gitlab branch: https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svn-perf-6-15-2025/-/commits/priority_fixes.v4
>
> But after reading this a bit more carefully, I suspect the real issue is
> that fragmentation is driving shrinker activity. Your changes more or
> less prevent Xe from shrinking its working set, since nearly every
> buffer is WC. In my opinion, that behavior is reasonable for
> fragmentation-driven reclaim, but it is much less appropriate in genuine
> low-memory situations.
>
> This is also a known issue for us. We most recently attempted to address
> it here [8], but the fix was blocked by the shrinker maintainer - the
> same has happened to other proposed solutions as well. We have several
> downstream customers carrying non-upstream fixes to work around this
> problem too...
>
> [8] https://patchwork.freedesktop.org/series/168651/
>
> I guess if it isn't too much trouble could you test:
>
> 1. Priority changes [7]
> 2. Avoid shrinking working sets on fragmentation [8]
> 3. Both [7], [8]
>
> Matt

Hi Matt,

Thanks again for the priority series. I have now done a longer test of
[7], including a system-wide Xe/TTM trace that records the exact BO
backup and restore traffic.

Test setup
----------

The test machine has a Panther Lake iGPU and 8 GiB of system memory. It
is running Linux 6.18.15 with the applicable parts of the current v6
priority series backported.

Since 6.18 does not have the newer Xe purgeable madvise state machine, I
backported patches 1-4 and 7-8. Patches 5-6 depend on the newer
xe_bo_set_purgeable_state() infrastructure. The active VM-bound WC path
in this reproducer does not use those purgeable states.

The workload is continuous 4K60 HDR playback. Decoded video frames are
passed through a libplacebo/OpenGL rendering path, with every HDR frame
going through GPU HDR rendering/tone mapping before display. glFlush()
submits the rendering work for each frame. The working set is mostly
DMA-BUF-backed video/render surfaces and other VM-bound WC BOs.

Results with [7]
----------------

The priority changes alone did not fix the stalls. They appear to delay
the first visible failure after playback starts, but once reclaim and
fragmentation activity builds up, the same failure mechanism and
hundreds-of-milliseconds Flush stalls return.

In a 599.545-second trace, the player recorded:

  Flush samples:                 16,462
  Flush > 16.667 ms:               157
  Flush > 100 ms:                   67
  Maximum Flush:               494.434 ms

Every one of the 157 Flush calls over the frame interval contained both
ttm_tt_restore() and set_pages_array_wc(). The maximum Flush contained
73 BO restores and 65 WC conversions.

For the memory trace I used temporary entry/return probes to record the
ttm_tt pointer, num_pages, and the actual page count returned by
ttm_tt_backup(). This allowed a later restore to be matched to the exact
same ttm_tt rather than only comparing aggregate counters.

The ten-minute working-set traffic was:

  successful ttm_tt_backup:          6,743  (11.247 objects/s)
  pages actually backed up:      6,717,364  (26,239.703 MiB total)
  ttm_tt_restore:                    6,693  (11.163 objects/s)
  restore matched to same ttm_tt:    6,673 / 6,693
  matched restore volume:       25,993.043 MiB

This is at least 52,232.746 MiB of backup-plus-restore copy traffic in
ten minutes, or 87.121 MiB/s on average. It is cumulative migration
traffic, not resident memory growth.

Of 639 unique ttm_tt objects that were backed up, 491 were backed up at
least twice. One object went through 125 backup cycles during the
capture. The delay from completion of a backup to restoration of the
same ttm_tt was:

  median:       47.932 ms
  p95:         189.083 ms
  p99:         360.138 ms
  within 100 ms: 5,190 / 6,673 (77.8%)
  within 1 s:    6,653 / 6,673 (99.7%)

This looks like working-set thrashing rather than reclaim of cold BOs.
The caller distribution makes the cycle particularly clear: kswapd0
performed 6,460 of the 6,743 backups (95.8%), while the restores were
mostly performed by the player's rendering and decode threads.

The activity was bursty. Backup occurred in only 42 of the 600 trace
seconds, grouped into 14 reclaim/restore storms, with up to 397 BO
restores in one second. Backup and restore page volume had a same-second
correlation of 0.997. Per-second maximum Flush had correlations of 0.904
with backup pages and 0.917 with restore-request pages.

The trigger also looks fragmentation driven:

  kswapd wake order-10:             577 / 590 (97.8%)
  wakeup_kswapd order-10:           560 / 576 (97.2%)
  direct reclaim order-10:          512 / 518 (98.8%)
  MemAvailable during the trace:    2.89 - 3.15 GiB
  compact_stall/success/fail:       1,073 / 132 / 941

Thus the system still had about 3 GiB available while nearly all of the
reclaim wakeups were for order-10 allocations. This is consistent with
high-order fragmentation repeatedly invoking shrinkers, rather than
sustained order-0 memory shortage.

There is also normal small-object allocation activity, but it is not the
main problem. ttm_pool_alloc/free ran at about 41.6/41.5 calls per second,
but 97.6% of allocations were only one or two pages and are normally
reused through the TTM pool. In contrast, the same active working set was
copied back and forth for more than 52 GiB during the test.

For comparison, with my earlier workaround that skipped non-purge
shrinking of VM-bound WC BOs, set_pages_array_wc() ran at 0.179/s,
versus 9.859/s in this test. The current trace restored backed-up content
at about 11,099 pages/s, while the earlier capture recorded
ttm_backup_copy_page() at 26.74/s. The earlier 21-minute capture also had
no Flush, XE_EXEC, VM_BIND, or DMA-BUF ioctl over 16.7 ms.

The current trace is system-wide whereas that older function graph was
player-TID filtered, so these are not fully identical A/B collection
scopes. However, the difference in WC restore activity and the
user-visible result is large and consistent.

Interpretation of [7]
---------------------

The priority changes seem to be doing what they are intended to do: they
change which object is considered first. However, most of the active
working set in this workload is VM-bound WC. Once the lower-priority and
WB candidates are absent or insufficient, the shrinker still reaches the
WC half of the bands and then the higher-priority WC BOs.

Therefore [7] has no clear steady-state benefit for this reproducer. It
mostly postpones the point at which the active WC working set becomes the
remaining reclaim target. Once that point is reached, the backup/restore
and CPA cost is essentially the original problem.

Possible Xe-local alternative to [8]
------------------------------------

I agree that [8] may help this workload, and the order-10 trace data is
consistent with the condition that [8] is trying to identify. I have not
yet completed the [8]-only and [7]+[8] tests.

I also understand the MM feedback that this is a wider compaction versus
shrinker-working-set problem, not something unique to Xe. Nevertheless,
for Xe there may be a narrower implementation that does not require a new
hint to be propagated through MM.

My idea is to make the Xe shrinker distinguish base-page shortage from
fragmentation before entering its destructive, non-purge backup pass:

  1. Always keep the purgeable-object pass available.

  2. Check order-0 free pages against the relevant node/zone watermarks,
     using sc->nid and the zones allowed by sc->gfp_mask. This should be a
     watermark check, not a global MemAvailable threshold.

  3. If the relevant order-0 supply is healthy, for example above the
     high watermark, treat the shrink request as fragmentation-only for
     Xe and skip backing up the active VM-bound working set. Reclaiming
     arbitrary BO-sized objects is unlikely to produce the contiguous
     order-10 extent and is very likely to destroy the GPU working set.

  4. If the order-0 watermark is below low, treat it as genuine memory
     pressure and retain the normal Xe shrinker behavior, including WC
     reclaim if cheaper candidates are insufficient. Between low and high
     the existing priority policy could remain the conservative fallback.

In pseudocode, the special case would be placed between the purgeable pass
and the non-purge backup pass in xe_shrinker_scan():

  run_purgeable_pass();

  if (order0_watermarks_healthy(sc))
          skip_non_purge_working_set_backup;

  run_normal_priority_based_backup_pass();

This differs from my original workaround because it would not make
VM-bound WC BOs unconditionally unreclaimable. They remain reclaimable
under real base-page pressure, so the system can still recover memory in
a genuine low-memory situation. It only avoids the expensive backup and
immediate restore cycle while base-page watermarks say that the problem is
fragmentation rather than capacity.

This Xe-local policy would not solve the broader problem for inode, dentry,
or other shrinker-managed working sets, so I do not see it as a replacement
for a general MM solution. It may, however, be a small and testable Xe-side
safeguard while the general compaction/shrinker policy is being worked out.

Would this be a reasonable Xe-specific experiment, or do you think Xe
should rely exclusively on an MM-provided indication such as [8]?

I have retained the raw trace, the per-second timeline, and the probe
script and can provide them if useful.

Thanks,
Neil