[PR] avcodec/h264_slice: do not allocate motion_val/ref_index tables under a hwaccel (PR #24322)

PR #24322 opened by Bradyok
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24322
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24322.patch

alloc_picture() takes two motion_val and two ref_index entries from the
refstruct pools for every H264Picture regardless of the decode path. Pool
entries are zeroed on allocation, so each picture carries about 1 MiB of
resident memory at 1080p (2 x 523344 B motion_val + 2 x 32912 B ref_index)
that nothing reads when a hwaccel does the decoding:

- h264_mvpred.h, h264_direct.c and fill_filter_caches() are reached only
  from decode_slice() via ff_h264_execute_decode_slices(), which returns
  before doing any work when avctx->hwaccel is set;
- error resilience runs from ff_er_frame_end(), whose er_supported()
  returns 0 under a hwaccel, and ff_er_frame_end() allocates its own
  tables when cur_pic.motion_val[0] is NULL anyway;
- ff_print_debug_info2() is guarded on motion_val[0] being non-NULL;
- ff_h264_set_erpic() and the H264Picture copy/unref helpers only copy
  the pointers, through av_refstruct_replace()/av_refstruct_unref(),
  which accept NULL.

avctx->hwaccel is settled by get_pixel_format() in h264_init_ps() before
h264_frame_start() allocates the picture, and a pixel format change
reinitializes through ff_h264_flush_change() -> ff_h264_remove_all_refs(),
so a hwaccel picture without tables can never become a reference for a
software-decoded one. The condition is the same avctx->hwaccel check that
alloc_picture() and h264_frame_start() already use.

Decoding 32 concurrent 1080p H.264 streams through VideoToolbox, this
saves about 160 MB of RSS.

Signed-off-by: Braden O'Keefe <[email protected]>

Tested: `make fate-h264` passes (222 tests) on macOS arm64 with the patch applied to a clean n9.0 tree; the change is also in daily use on a VideoToolbox-based multi-stream player (that is where the 160 MB figure was measured). The patch applies unchanged to master.



>From 2322a65960a54f1b03f4941b5b5e45f76011ef4d Mon Sep 17 00:00:00 2001
From: Braden O'Keefe <[email protected]>
Date: Sat, 29 Aug 2026 18:42:50 -0400
Subject: [PATCH] avcodec/h264_slice: do not allocate motion_val/ref_index
 tables under a hwaccel

alloc_picture() takes two motion_val and two ref_index entries from the
refstruct pools for every H264Picture regardless of the decode path. Pool
entries are zeroed on allocation, so each picture carries about 1 MiB of
resident memory at 1080p (2 x 523344 B motion_val + 2 x 32912 B ref_index)
that nothing reads when a hwaccel does the decoding:

- h264_mvpred.h, h264_direct.c and fill_filter_caches() are reached only
  from decode_slice() via ff_h264_execute_decode_slices(), which returns
  before doing any work when avctx->hwaccel is set;
- error resilience runs from ff_er_frame_end(), whose er_supported()
  returns 0 under a hwaccel, and ff_er_frame_end() allocates its own
  tables when cur_pic.motion_val[0] is NULL anyway;
- ff_print_debug_info2() is guarded on motion_val[0] being non-NULL;
- ff_h264_set_erpic() and the H264Picture copy/unref helpers only copy
  the pointers, through av_refstruct_replace()/av_refstruct_unref(),
  which accept NULL.

avctx->hwaccel is settled by get_pixel_format() in h264_init_ps() before
h264_frame_start() allocates the picture, and a pixel format change
reinitializes through ff_h264_flush_change() -> ff_h264_remove_all_refs(),
so a hwaccel picture without tables can never become a reference for a
software-decoded one. The condition is the same avctx->hwaccel check that
alloc_picture() and h264_frame_start() already use.

Decoding 32 concurrent 1080p H.264 streams through VideoToolbox, this
saves about 160 MB of RSS.

Signed-off-by: Braden O'Keefe <[email protected]>
---
 libavcodec/h264_slice.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/libavcodec/h264_slice.c b/libavcodec/h264_slice.c
index a13047ffe8..5713d76948 100644
--- a/libavcodec/h264_slice.c
+++ b/libavcodec/h264_slice.c
@@ -250,13 +250,25 @@ static int alloc_picture(H264Context *h, H264Picture *pic)
     pic->mb_type      = pic->mb_type_base + 2 * h->mb_stride + 1;
     pic->qscale_table = pic->qscale_table_base + 2 * h->mb_stride + 1;
 
-    for (i = 0; i < 2; i++) {
-        pic->motion_val_base[i] = av_refstruct_pool_get(h->motion_val_pool);
-        pic->ref_index[i]       = av_refstruct_pool_get(h->ref_index_pool);
-        if (!pic->motion_val_base[i] || !pic->ref_index[i])
-            goto fail;
+    /* The motion-vector and reference-index tables are read only by the
+     * software macroblock path (h264_mvpred.h, h264_direct.c,
+     * fill_filter_caches()), which is never entered under a hwaccel
+     * (ff_h264_execute_decode_slices() returns before doing anything), and
+     * by error resilience, which er_supported() disables under a hwaccel and
+     * which allocates its own tables when these are NULL. Every other user
+     * (ff_print_debug_info2(), ff_h264_set_erpic(), the picture copies) is
+     * NULL-safe. The pool entries are zeroed on allocation, so skipping them
+     * for hwaccel pictures saves about 1 MiB of resident memory per 1080p
+     * picture. */
+    if (!h->avctx->hwaccel) {
+        for (i = 0; i < 2; i++) {
+            pic->motion_val_base[i] = av_refstruct_pool_get(h->motion_val_pool);
+            pic->ref_index[i]       = av_refstruct_pool_get(h->ref_index_pool);
+            if (!pic->motion_val_base[i] || !pic->ref_index[i])
+                goto fail;
 
-        pic->motion_val[i] = pic->motion_val_base[i] + 4;
+            pic->motion_val[i] = pic->motion_val_base[i] + 4;
+        }
     }
 
     pic->pps = av_refstruct_ref_c(h->ps.pps);
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]