[PR] avfilter/dnn: reject channel-mismatched model output in dnn_processing (PR #24048)

PR #24048 opened by Jake (jakefineman)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24048
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24048.patch

The `AV_PIX_FMT_RGB24`/`AV_PIX_FMT_BGR24` output case in `ff_proc_from_dnn_to_frame()` (`libavfilter/dnn/dnn_io_proc.c`) hardcodes the `sws_getContext`/`sws_scale` width arguments and strides to `frame->width * 3`, unconditional on the model's real output channel count. For any model whose output has fewer than 3 channels — e.g. a single-channel alpha-matte output from an RGB-in background-removal model such as MODNet — that mismatch is memory-unsafe on both layouts:

- The first `sws_scale` runs for **both** `DL_NCHW` and `DL_NHWC` output, before the planar-to-packed branch, and reads `frame->width * 3 * src_datatype_size * frame->height` bytes from `output->data`; a 1-channel tensor holds a third of that: a ~3x source-side over-read.
- On `DL_NCHW`, `middle_data` is correctly sized for the real channel count (`plane_size * output->dims[1]`), so the same hardcoded width also makes `sws_scale` write `frame->width * 3 * frame->height` bytes into a buffer sized for only `frame->width * frame->height * output->dims[1]` bytes: a heap write 3x past the allocation for `dims[1] == 1`. glibc's allocator catches this on a subsequent `free()`/malloc consistency check, producing `SIGABRT`, not a clean FFmpeg error return.

## Fix

Reject channel-mismatched model output at the top of the RGB24/BGR24 case, layout-agnostically: read the channel dim via `dnn_get_channel_idx_by_layout(output->layout)` (the same helper this file already uses on the input side, `dnn_io_proc.c:370`), and if `output->dims[channel_idx] != 3`, log a clear error and return `AVERROR(ENOSYS)` before any of the mismatched-stride arithmetic runs.

One check covers all three hazards: the source-side over-read (both layouts), the `DL_NCHW` destination over-write, and the `DL_NCHW` planar-to-packed GBRP repack a few lines further down, which likewise unconditionally assumes 3 planes and is never reached once channel count is checked.

## Repro

Build: FFmpeg `n9.0` (pinned commit `d32b387`), debian trixie, ONNX Runtime 1.28.0, `--enable-libonnxruntime`, native linux/arm64. Model: `onnx-community/modnet-webnn` `onnx/model.onnx`, sha256 `07c308cf0fc7e6e8b2065a12ed7fc07e1de8febb7dc7839d7b7f15dd66584df9` (output `[1,1,H,W]`, i.e. `DL_NCHW` with `dims[1] == 1`).

**Before** — identical build with this patch omitted:

```
$ ffmpeg -f lavfi -i testsrc2=size=320x320:rate=1:duration=1 \
    -vf 'format=rgb24,dnn_processing=dnn_backend=onnx:model=modnet.onnx:input=input:output=output' \
    -frames:v 1 -f null -
[dnn_base] Using CPU execution provider
free(): invalid pointer
$ echo $?
134
```

**After** — same command, same model, patched build:

```
[dnn_base] dnn_processing to a rgb24 frame requires a 3-channel model output, got 1 channels;
channel-reducing/expanding models (e.g. single-channel matte output) are not supported by this filter
$ echo $?
0
```

Scope note on the receipt: the runtime repro above exercises the `DL_NCHW` destination over-write, which is the path that aborts. The `DL_NHWC` source-side over-read is identified from source rather than separately reproduced — I did not have a channel-reducing NHWC model to hand — but it runs through the same hardcoded-stride `sws_scale` call above the layout branch, which is why the guard is keyed on the layout-resolved channel index rather than on `dims[1]`.

## Deliberate scope

This patch rejects channel-reducing model output cleanly; it does **not** add support for actually rendering a 1-channel (or 2-channel) model output through this filter. Real support for that would need a new pixel-format/plane-count target on the output side (e.g. gray/ya8 for a matte, or a caller-supplied channel mapping) — a bigger design decision than a minimal heap-safety fix should make unilaterally, and out of scope here.

I'm sending the minimal safety fix first because the crash is the more urgent problem: a SIGABRT driven by model-controlled output shape is a hardening bug independent of whether anyone wants channel-reducing rendering. Happy to follow up with a rendering-support patch, or take direction on it, if a maintainer prefers that be done in the same series instead of split.

One further caveat, so the receipt does not overclaim: `vf_dnn_processing.c`'s own caller does not hard-abort the filter pipeline on this specific `AVERROR(ENOSYS)` return (pre-existing behaviour of that file, untouched here) — for a real (non-`-f null`) pipeline the practical effect is an unfiltered/passthrough frame rather than a full pipeline failure.

## Checks

Applies clean with `git am` on both `n9.0` and current `master` (`master`'s `dnn_io_proc.c` still hardcodes `frame->width * 3` unconditionally as of this writing — not yet fixed upstream). Builds with `--enable-libonnxruntime`. `tools/patcheck` reports only the expected "missing changelog entry" advisory for a one-function bugfix.

No existing trac ticket or patchwork series found for this bug in a bounded search (`dnn_processing sws_scale`, `dnn_io_proc heap`, `background removal matte crash dnn`).



From b3f3be7585bd7ac299c044316843acfd047c8626 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Fri, 7 Aug 2026 17:57:03 -0400
Subject: [PATCH] avfilter/dnn: reject channel-mismatched model output in
 dnn_processing

ff_proc_from_dnn_to_frame()'s AV_PIX_FMT_RGB24/BGR24 case assumes the
model's output has exactly 3 channels, but never checks it.

The first sws_scale() in that case reads from output->data with a
hardcoded source stride of frame->width * 3 * src_datatype_size over
frame->height rows, i.e. frame->width * 3 * frame->height *
src_datatype_size bytes. For a model whose output has fewer channels
than that -- e.g. the single-channel alpha matte an RGB-in
background-removal model produces -- the real tensor only holds
frame->width * frame->height * channels * src_datatype_size bytes, so
the call reads roughly 3x past the end of it. This affects both
DL_NCHW and DL_NHWC output, since it happens before the layout-specific
planar-to-packed branch.

For DL_NCHW there is a second, independent overflow on the write side:
middle_data is allocated plane_size * output->dims[1] bytes, but
linesize[0] is set to frame->width * 3 unconditionally, so the same
sws_scale() also writes past that allocation. This reproduces as a
SIGABRT (glibc "free(): invalid pointer", or a malloc.c assertion,
depending on allocator state) inside dnn_processing, after inference
has already completed successfully in the backend.

The input side is guarded -- check_modelinput_inlink() in
vf_dnn_processing.c -- but the output side is not. dnn_processing only
supports channel-preserving models for these pixel formats by design,
so reject a mismatched channel count with a clear error instead of
reading and writing out of bounds. The channel dimension is resolved
via dnn_get_channel_idx_by_layout() so the check covers both layouts.

This does not add support for rendering a 1- or 2-channel model output
through this filter; that would need a different pixel-format target
and is left as future work.

Signed-off-by: Jake Fineman <[email protected]>
---
 libavfilter/dnn/dnn_io_proc.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index 826110dab0..55981c64da 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -50,6 +50,7 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx)
     int plane_size = frame->width * frame->height * sizeof(uint8_t);
     enum AVPixelFormat src_fmt = AV_PIX_FMT_NONE;
     int src_datatype_size = get_datatype_size(output->dt);
+    int channel_idx;
 
     int bytewidth = av_image_get_linesize(frame->format, frame->width, 0);
     if (bytewidth < 0) {
@@ -83,6 +84,17 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx)
     switch (frame->format) {
     case AV_PIX_FMT_RGB24:
     case AV_PIX_FMT_BGR24:
+        channel_idx = dnn_get_channel_idx_by_layout(output->layout);
+        if (output->dims[channel_idx] != 3) {
+            av_log(log_ctx, AV_LOG_ERROR,
+                   "dnn_processing to a %s frame requires a 3-channel model "
+                   "output, got %d channels; channel-reducing/expanding "
+                   "models (e.g. single-channel matte output) are not "
+                   "supported by this filter\n",
+                   av_get_pix_fmt_name(frame->format), output->dims[channel_idx]);
+            ret = AVERROR(ENOSYS);
+            goto err;
+        }
         sws_ctx = sws_getContext(frame->width * 3,
                                  frame->height,
                                  src_fmt,
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]