Re: [BUG] drm/vmwgfx: vmw_cmdbuf_alloc leaks ~103 MiB/day of unreclaimable slab via the fbdev damage worker

On Wed, Aug 5, 2026 at 12:59 PM Дима Дьячков <[email protected]> wrote:
>
> Hi,
>
> I have a reproducible, quantified kernel memory leak in vmwgfx on a headless
> VMware guest. It is attributed to a specific call site with slab_debug, and I
> have a clean causal test that stops it. I could not find this reported anywhere
> - I searched the dri-devel archives, the mainline and 6.12.y git logs, and the
> distro trackers.
>
> Summary
> =======
>
> On a headless VMware guest with the text console bound to the vmwgfx
> framebuffer, the fbdev-emulation damage worker issues a full atomic commit for
> every console damage event, and vmwgfx leaks a command buffer on each one. The
> result is unreclaimable kernel slab growing linearly at ~103 MiB/day, never
> reclaimed under memory pressure, released only by reboot.
>
> Environment
> ===========
>
> Kernel: 6.12.100+deb13-amd64 (Debian 13 stock, tracks 6.12.y)
> Guest: VMware Virtual Platform, BIOS 6.00; vmwgfx drives the virtual display
> Console: headless, administered over SSH; a single agetty on the console
> fb0: vmwgfxdrmfb
> vtcon0: "(M) frame buffer device", bind=1
> Config: CONFIG_HZ=250, CONFIG_SLUB_DEBUG=y, CONFIG_STACKDEPOT=y
> (CONFIG_DEBUG_KMEMLEAK is not set in Debian's kernel)
>
> i915 is also loaded on this host (a passed-through GPU used for video
> transcoding) but is unrelated - the console framebuffer is vmwgfx.
>
> Symptom
> =======
>
> SUnreclaim grows linearly at ~103 MiB/day and is never reclaimed under pressure.
> The growth is confined to the plain kmalloc-256 cache - notably *not*
> kmalloc-cg-256, so it is plain GFP_KERNEL from an in-kernel allocator rather
> than anything cgroup-accounted.
>
> Attribution
> ===========
>
> Booted with slab_debug=U,kmalloc-256, then read
> /sys/kernel/debug/slab/kmalloc-256/alloc_traces, which ranks by live object
> count:
>
> 41025 vmw_cmdbuf_alloc+0x40/0x200 [vmwgfx] waste=328200/8 age=254700/1301825/2351179 pid=9-208926 cpus=0-5
> vmw_cmdbuf_alloc+0x40/0x200 [vmwgfx]
> vmw_cmdbuf_reserve+0x142/0x170 [vmwgfx]
> vmw_cmd_ctx_reserve+0x24/0x60 [vmwgfx]
> vmw_stdu_bind_st+0x6b/0x100 [vmwgfx]
> vmw_stdu_primary_plane_atomic_update+0xa8/0x1d0 [vmwgfx]
> drm_atomic_helper_commit_planes+0x112/0x320 [drm_kms_helper]
> drm_atomic_helper_commit_tail+0x48/0x90 [drm_kms_helper]
> vmw_atomic_commit_tail+0x1b/0x80 [vmwgfx]
> commit_tail+0x91/0x130 [drm_kms_helper]
> drm_atomic_helper_commit+0x11a/0x140 [drm_kms_helper]
> drm_atomic_commit+0xa9/0xe0 [drm]
> drm_atomic_helper_dirtyfb+0x1c6/0x2a0 [drm_kms_helper]
> drm_fbdev_ttm_helper_fb_dirty+0x216/0x310 [drm_ttm_helper]
> drm_fb_helper_damage_work+0x94/0x170 [drm_kms_helper]
> process_one_work+0x174/0x330
> worker_thread+0x191/0x2f0
>
> The next-largest entry in that cache held 1056 objects - this one is ~30x
> everything else combined.
>
> Quantitative confirmation
> =========================
>
> With CONFIG_HZ=250, from the age=min/avg/max field above:
>
> - max age 2,351,179 jiffies = 9405 s = the full uptime at time of reading
> - avg age is almost exactly half of max, which is the signature of uniform
> allocation with zero frees
> - 41,025 objects over the 8386 s during which allocation was active
> = 4.89 objects/sec = 4.30 MiB/h = ~103 MiB/day
>
> That matches the independently measured SUnreclaim growth rate to within noise.
>
> Causal test
> ===========
>
> Unbinding the text console from the framebuffer stops it instantly:
>
> # echo 0 > /sys/class/vtconsole/vtcon0/bind
>
> After that, the minimum object age jumped from 26 jiffies to 254,700 and kept
> climbing - i.e. no new allocations at all. SUnreclaim also dropped ~93 MiB
> immediately as the fbdev shadow buffer was released, and has been flat-to-
> falling since.
>
> Two things I checked so you don't have to
> =========================================
>
> This is not a recent regression. drivers/gpu/drm/vmwgfx/vmwgfx_cmdbuf.c has had
> no functional change between v6.12 and current mainline - only the command
> buffer ID addition (922f9de) and the treewide kmalloc -> kmalloc_obj
> conversions. vmwgfx_stdu.c is similarly untouched in the relevant paths, and
> nothing in the 6.12.y stable series touches cmdbuf, damage or dirtyfb handling.
> So this looks long-standing rather than newly introduced.
>
> It is also not caused by the fbdev-ttm conversion, which is the obvious suspect
> given the stack. drm_fbdev_ttm was a pure rename of drm_fbdev_generic
> (aae4682e5d66, v6.10), and vmwgfx has used the generic DRM fbdev emulation since
> v6.2 (df42523c12f8, which deleted vmwgfx_fb.c). What did change at v6.2 is that
> console damage now goes through a full atomic commit, which the old
> vmwgfx_fb.c did not do - but the leak itself is on the vmwgfx side.
>
> Possibly related
> ================
>
> There is an open, unresolved report of a different vmwgfx leak in the same
> atomic-commit path - Yuma Kakei, 12 Mar 2026, "[BUG] vmwgfx: Severe Slab memory
> leak (radix_tree_node) with Wayland Atomic KMS" (kernels 6.17 / 7.0-rc3, KWin
> Wayland with Atomic KMS, stops with KWIN_DRM_NO_AMS=1). Ian Forbes proposed a
> patch; the reporter confirmed on 15 Mar 2026 that it did not fix it.
>
> I do *not* claim these are the same bug - different slab caches, different
> trigger, different kernels. But both are vmwgfx leaking on every atomic commit
> and stopping dead when the atomic path stops, so they may share a root cause.
>
> Workarounds
> ===========
>
> - echo 0 > /sys/class/vtconsole/vtcon0/bind
> runtime, reversible, stops it immediately
>
> - drm_kms_helper.fbdev_emulation=0 on the kernel command line
> cleaner for a headless host; disables fbdev emulation outright so no
> damage worker exists. Must be set at boot.
>
> - modprobe.blacklist=vmwgfx
> heavy-handed; loses all console output
>
> All three cost the virtual console, which is fine headless but is not a general
> fix.
>
> Happy to test patches, run instrumented kernels, or provide further data - the
> system reproduces this continuously and I can rebind the console at will to
> toggle it on and off.
>
> Thanks, Dmitry

Could you send us the full system log? (ideally journalctl -b, if not
then, dmesg, or /var/log/kern.lorg). I suspect you have some command
buffer errors in the kernel log that are causing this. Based on your
analysis disabling the cursor_blink is probably a better workaround,
i.e. "echo 0 > /sys/class/graphics/fbcon/cursor_blink" . Besides the
system logs i'd be also intersted in pool info, i.e. cat
/sys/bus/pci/devices/<vmwgfx card id>/pools .

z