[PR] avformat/rtpenc_av1: bound OBU size in the keyframe search loop (PR #24096)

PR #24096 opened by michaelni
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24096
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24096.patch

The is_keyframe sequence-header search loop advanced buf_ptr/rem_size by
num_lebs + obu_size without bounding obu_size against the remaining data
(unlike the main packetization loop). A crafted obu_size (~0x80000010)
wraps the signed rem_size back positive, so the next iteration
dereferences a pointer past the packet. Consume the LEB bytes first, then
reject an OBU larger than the remaining size. Out-of-bounds read reachable
from a crafted AV1 packet muxed to RTP.

Fixes: out of array read




>From fec2dc2a1df0cfa85bc85d1b4148e034d7a5f86f Mon Sep 17 00:00:00 2001
From: Joshua Rogers <[email protected]>
Date: Tue, 4 Aug 2026 12:11:55 +0000
Subject: [PATCH] avformat/rtpenc_av1: bound OBU size in the keyframe search
 loop

The is_keyframe sequence-header search loop advanced buf_ptr/rem_size by
num_lebs + obu_size without bounding obu_size against the remaining data
(unlike the main packetization loop). A crafted obu_size (~0x80000010)
wraps the signed rem_size back positive, so the next iteration
dereferences a pointer past the packet. Consume the LEB bytes first, then
reject an OBU larger than the remaining size. Out-of-bounds read reachable
from a crafted AV1 packet muxed to RTP.

Fixes: out of array read
---
 libavformat/rtpenc_av1.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/libavformat/rtpenc_av1.c b/libavformat/rtpenc_av1.c
index fbf9212216..ae66fdb86f 100644
--- a/libavformat/rtpenc_av1.c
+++ b/libavformat/rtpenc_av1.c
@@ -116,8 +116,15 @@ void ff_rtp_send_av1(AVFormatContext *ctx, const uint8_t *frame_buf, int frame_s
             if (!num_lebs) {
                 break;
             }
-            buf_ptr += num_lebs + obu_size;
-            rem_size -= num_lebs + obu_size;
+            buf_ptr += num_lebs;
+            rem_size -= num_lebs;
+            // bound OBU payload against remaining data to avoid pointer/size
+            // wraparound (mirrors the check in the packetization loop below)
+            if (obu_size > (uint32_t) rem_size) {
+                break;
+            }
+            buf_ptr += obu_size;
+            rem_size -= obu_size;
         }
 #else // RTPENC_AV1_SEARCH_SEQ_HEADER
         av_log(ctx, AV_LOG_DEBUG, "Marking FIRST packet\n");
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]