[PR] avfilter/dnn_backend_onnx: implement DFT_ANALYTICS_CLASSIFY for dnn_classify (PR #24049)

PR #24049 opened by Jake (jakefineman)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24049
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24049.patch

The ONNX backend (`dnn_backend_onnx.c`) switches on `model.func_type` in both `fill_model_input_onnx()` and `infer_completion_callback()`, but only handles `DFT_PROCESS_FRAME` (plus `DFT_ANALYTICS_DETECT` on the input side). `DFT_ANALYTICS_CLASSIFY` falls through to the default `avpriv_report_missing_feature()` branch unconditionally, so `dnn_classify` cannot use the ONNX backend at all:

```
model function type 3 is not implemented
```

## 1/2 — implement `DFT_ANALYTICS_CLASSIFY`

Adds the classify case to both switches, mirroring the pattern the OpenVINO backend already uses (`dnn_backend_openvino.c`): the input side calls `ff_frame_to_dnn_classify()` with the bbox index carried on the task item; the output side invokes the filter's `classify_post_proc` callback (set by `vf_dnn_classify.c`) with the same bbox index.

The completion callback ignores `classify_post_proc`'s return value. That is deliberate — it matches the OpenVINO backend's own precedent exactly, which calls `classify_post_proc(...)` as a bare statement at `dnn_backend_openvino.c:473` (and `detect_post_proc` likewise at :463). Changing that convention is out of scope here; if maintainers want the return propagated it should be done for both backends in one change.

## 2/2 — register the `onnx` named constant for `dnn_classify`

`dnn_classify` registered only `tensorflow`/`openvino` named `AVOption` constants for `dnn_backend`, so after 1/2 the ONNX backend was reachable only by numeric value (`dnn_backend=8`). This registers the `onnx` constant behind `CONFIG_LIBONNXRUNTIME` — same macro and same position in the per-backend `#if` chain as `vf_dnn_processing.c` — and adds `DNN_ONNX` to `dnn_classify`'s `AVFILTER_DNN_DEFINE_CLASS` backend mask so `-h filter=dnn_classify` lists the ONNX sub-options.

**`dnn_detect` is deliberately not included.** An earlier revision of this series also registered the constant for `dnn_detect`, added `DNN_ONNX` to its backend mask, and allowlisted `DNN_ONNX` in its `check_output_nb()`. That was wrong and has been dropped: `infer_completion_callback()` has no `DFT_ANALYTICS_DETECT` case, so the ONNX backend cannot produce bounding boxes. Admitting `dnn_detect` past its init-time rejection would only move the failure later, from a clear "does not support current backend" at filter init to "model function type 2 is not implemented" at inference time. Detect support belongs in a separate change that implements the output-side dispatch first. This series leaves `vf_dnn_detect.c` byte-identical to master.

## Repro

Build: FFmpeg `n9.0` (pinned commit `d32b387`), debian trixie, ONNX Runtime 1.28.0, `--enable-libonnxruntime`, native linux/arm64.

Before 1/2 — bare code-path check, no model needed:

```
$ ffmpeg -f lavfi -i testsrc2=size=64x64:rate=1:duration=1 \
    -vf 'dnn_classify=dnn_backend=8:model=/tmp/x.onnx' -frames:v 1 -f null -
[dnn_backend_onnx] model function type 3 is not implemented
```

After both patches — the named constant parses and the filter reaches ONNX session creation, erroring only on the missing model file:

```
$ ffmpeg -f lavfi -i testsrc2=size=64x64:rate=1:duration=1 \
    -vf 'dnn_classify=dnn_backend=onnx:model=/tmp/nonexistent.onnx' -frames:v 1 -f null -
[dnn_base] Using CPU execution provider
[dnn_base] Failed to create ONNX session: Load model from /tmp/nonexistent.onnx failed. File doesn't exist
[Parsed_dnn_classify_0] could not load DNN model
```

`-h filter=dnn_classify` now lists it, and `dnn_detect` is unchanged:

```
$ ffmpeg -h filter=dnn_classify | grep onnx
   onnx    8    ..FV....... onnx backend flag

$ ffmpeg -h filter=dnn_detect | grep onnx        # no output — unchanged by this series
```

`dnn_detect` with the ONNX backend selected numerically still fails closed at init, as it does on master:

```
[dnn_detect] Dnn detect filter does not support current backend
```

With a real model (MobileNetV2, `onnx/models` zoo, Apache-2.0, sha256 `c1c513582d56afceff8516c73804e484c81c6a830712ab6d682253f4a3cd042f`) an ONNX Runtime session loads and the classify path is entered. To be precise about what that does and does not prove: session creation is reached, but a standalone `dnn_classify` run then stops at the pre-existing `av_assert0(sd)` in `ff_frame_to_dnn_classify()` (`dnn_io_proc.c:355`), because `dnn_classify` expects `AV_FRAME_DATA_DETECTION_BBOXES` side data — it is architecturally designed to run chained after `dnn_detect`, and a bare `testsrc2` frame has none. So this is a code-path-entry receipt, not an end-to-end inference receipt.

That assertion is pre-existing, backend-agnostic code that this series does not touch: `ff_frame_to_dnn_classify()` is called identically from `dnn_backend_openvino.c:315`, so a standalone `dnn_classify` behaves the same way on the OpenVINO backend today. This series makes ONNX match the established behaviour rather than introducing a new failure mode. Whether that bare-frame case deserves a clean error instead of an assert is a real question, but it applies to every backend and belongs in its own patch — happy to send one if maintainers agree.

## Checks

Both patches apply clean with `git am` on `n9.0` and on current `master`, build with `--enable-libonnxruntime`, and pass `tools/patcheck` with only the expected "missing changelog entry" advisory.

No existing trac ticket or patchwork series was found for this gap (searched `dnn_classify`, `dnn_backend_onnx classify`, and the in-flight 20-part `dnn_classify`-as-multimedia-filter series on patchwork — that series targets the Torch backend's zero-shot-classification path and does not overlap this fix).



From 33fb19c89f88fb158b876337a1ed6090d5147f35 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Fri, 7 Aug 2026 17:55:34 -0400
Subject: [PATCH 1/2] avfilter/dnn_backend_onnx: implement
 DFT_ANALYTICS_CLASSIFY

The ONNX backend switch()es on model.func_type in both the input-prep
path (fill_model_input_onnx) and the output-extraction path
(infer_completion_callback), but only DFT_PROCESS_FRAME and (on the
input side) DFT_ANALYTICS_DETECT were handled; DFT_ANALYTICS_CLASSIFY
fell through to the default avpriv_report_missing_feature() branch
unconditionally, regardless of model shape ("model function type 3 is
not implemented").

Add the classify case to both switches, mirroring the pattern already
used by the OpenVINO backend (dnn_backend_openvino.c): the input path
calls ff_frame_to_dnn_classify() with the per-bounding-box index
carried on the LastLevelTaskItem, and the output path invokes the
filters classify_post_proc callback (set by vf_dnn_classify.c) with
that same bbox index.

Signed-off-by: Jake Fineman <[email protected]>
---
 libavfilter/dnn/dnn_backend_onnx.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/libavfilter/dnn/dnn_backend_onnx.c b/libavfilter/dnn/dnn_backend_onnx.c
index 6c75d6eb24..1bdd4ea1da 100644
--- a/libavfilter/dnn/dnn_backend_onnx.c
+++ b/libavfilter/dnn/dnn_backend_onnx.c
@@ -383,6 +383,9 @@ static int fill_model_input_onnx(ONNXModel *onnx_model, ONNXRequestItem *request
     case DFT_ANALYTICS_DETECT:
         ff_frame_to_dnn_detect(task->in_frame, &input, ctx);
         break;
+    case DFT_ANALYTICS_CLASSIFY:
+        ff_frame_to_dnn_classify(task->in_frame, &input, lltask->bbox_index, ctx);
+        break;
     default:
         avpriv_report_missing_feature(ctx, "model function type %d", onnx_model->model.func_type);
         ret = AVERROR(ENOSYS);
@@ -599,6 +602,17 @@ static void infer_completion_callback(void *args)
             task->out_frame->height = outputs.dims[dnn_get_height_idx_by_layout(outputs.layout)];
         }
         break;
+    case DFT_ANALYTICS_CLASSIFY:
+        if (!onnx_model->model.classify_post_proc) {
+            av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post proc\n");
+            av_free(dims);
+            g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+            goto err;
+        }
+        onnx_model->model.classify_post_proc(task->in_frame, &outputs,
+                                              lltask->bbox_index,
+                                              onnx_model->model.filter_ctx);
+        break;
     default:
         avpriv_report_missing_feature(ctx, "model function type %d", onnx_model->model.func_type);
         av_free(dims);
-- 
2.52.0


From 3c615716f60a33aa2a1cf4d750deb42f1164224a Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Fri, 7 Aug 2026 17:55:34 -0400
Subject: [PATCH 2/2] avfilter/vf_dnn_classify: register named onnx backend
 constant

dnn_classify only registered "tensorflow"/"openvino" named AVOption
constants for dnn_backend, even though the ONNX execution provider is
reachable by numeric value (dnn_backend=8, DNN_ONNX). This left "onnx"
undefined as a symbolic option value ("Undefined constant or missing
(in onnx") and excluded DNN_ONNX from the backend mask that gates the
filter's child-class option iteration, so -h filter=dnn_classify never
listed the ONNX-backend sub-options.

Register the "onnx" named constant behind CONFIG_LIBONNXRUNTIME,
matching the existing pattern in vf_dnn_processing.c, and add DNN_ONNX
to dnn_classify's AVFILTER_DNN_DEFINE_CLASS backend mask.

dnn_detect is deliberately left alone. The ONNX backend's
infer_completion_callback() has no DFT_ANALYTICS_DETECT case, so it
cannot produce bounding boxes: advertising the backend there would
admit dnn_detect past its init-time check_output_nb() rejection only
to fail later at inference with "model function type 2 is not
implemented", replacing a clear early error with a late one. Detect
support belongs in a separate change that implements the output-side
dispatch first.

Signed-off-by: Jake Fineman <[email protected]>
---
 libavfilter/vf_dnn_classify.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index f92c41ab76..8faf6685ee 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -48,6 +48,9 @@ static const AVOption dnn_classify_options[] = {
     { "dnn_backend", "DNN backend",                OFFSET(backend_type),     AV_OPT_TYPE_INT,       { .i64 = DNN_OV },    INT_MIN, INT_MAX, FLAGS, .unit = "backend" },
 #if (CONFIG_LIBOPENVINO == 1)
     { "openvino",    "openvino backend flag",      0,                        AV_OPT_TYPE_CONST,     { .i64 = DNN_OV },    0, 0, FLAGS, .unit = "backend" },
+#endif
+#if (CONFIG_LIBONNXRUNTIME == 1)
+    { "onnx",        "onnx backend flag",          0,                        AV_OPT_TYPE_CONST,     { .i64 = DNN_ONNX },  0, 0, FLAGS, .unit = "backend" },
 #endif
     { "confidence",  "threshold of confidence",    OFFSET2(confidence),      AV_OPT_TYPE_FLOAT,     { .dbl = 0.5 },  0, 1, FLAGS},
     { "labels",      "path to labels file",        OFFSET2(labels_filename), AV_OPT_TYPE_STRING,    { .str = NULL }, 0, 0, FLAGS },
@@ -55,7 +58,7 @@ static const AVOption dnn_classify_options[] = {
     { NULL }
 };
 
-AVFILTER_DNN_DEFINE_CLASS(dnn_classify, DNN_OV);
+AVFILTER_DNN_DEFINE_CLASS(dnn_classify, DNN_OV | DNN_ONNX);
 
 static int dnn_classify_post_proc(AVFrame *frame, DNNData *output, uint32_t bbox_index, AVFilterContext *filter_ctx)
 {
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]