[PR] avformat/rtpenc_av1: do not narrow the OBU size to (long) (PR #24090)

PR #24090 opened by michaelni
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24090
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24090.patch

(long)obu_size sign-flips values in 0x80000000..0xfffffffd on ILP32/LLP64
targets (32-bit long, e.g. 64-bit Windows), bypassing the size check.
Compare as uint32_t; frame_size is non-negative at this point.

No PoC as the used setup was 64bit specific


>From d6b3e68aed9879af22ea548c5e57b3ad1562314e Mon Sep 17 00:00:00 2001
From: Michael Niedermayer <[email protected]>
Date: Tue, 11 Aug 2026 21:28:17 +0200
Subject: [PATCH 1/2] avformat/rtpenc_av1: Check num_lebs

Signed-off-by: Michael Niedermayer <[email protected]>
---
 libavformat/rtpenc_av1.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavformat/rtpenc_av1.c b/libavformat/rtpenc_av1.c
index fbf9212216..93b78ab05f 100644
--- a/libavformat/rtpenc_av1.c
+++ b/libavformat/rtpenc_av1.c
@@ -160,7 +160,7 @@ void ff_rtp_send_av1(AVFormatContext *ctx, const uint8_t *frame_buf, int frame_s
             obu_hdr &= ~AV1F_OBU_HAS_SIZE_FIELD; // remove size field
             // read out explicit OBU size
             num_lebs = parse_leb(ctx, obu_ptr, frame_size, &obu_size);
-            if (!num_lebs) {
+            if (num_lebs <= 0 || num_lebs > INT_MAX - frame_size) {
                 return;
             }
             obu_ptr += num_lebs;
-- 
2.52.0


>From 7eacd80643438c0cc8cab30081cd533289c5ff70 Mon Sep 17 00:00:00 2001
From: Joshua Rogers <[email protected]>
Date: Tue, 4 Aug 2026 12:11:55 +0000
Subject: [PATCH 2/2] avformat/rtpenc_av1: do not narrow the OBU size to (long)

(long)obu_size sign-flips values in 0x80000000..0xfffffffd on ILP32/LLP64
targets (32-bit long, e.g. 64-bit Windows), bypassing the size check.
Compare as uint32_t; frame_size is non-negative at this point.

No PoC as the used setup was 64bit specific
---
 libavformat/rtpenc_av1.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavformat/rtpenc_av1.c b/libavformat/rtpenc_av1.c
index 93b78ab05f..477aa9edb5 100644
--- a/libavformat/rtpenc_av1.c
+++ b/libavformat/rtpenc_av1.c
@@ -170,7 +170,7 @@ void ff_rtp_send_av1(AVFormatContext *ctx, const uint8_t *frame_buf, int frame_s
             return;
         }
 
-        if ((long) obu_size > frame_size) {
+        if (obu_size > (uint32_t) frame_size) {
             av_log(ctx, AV_LOG_ERROR, "AV1 OBU size %d larger than remaining frame size %d\n", obu_size, frame_size);
             return;
         }
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]