[PR] avfilter/vf_mestimate{,_d3d12}: flush final frame at EOF (PR #24314)
Edison Ling via ffmpeg-devel <[email protected]>
| Newsgroups | gmane.comp.video.ffmpeg.devel |
|---|---|
| Message-ID | <[email protected]> |
PR #24314 opened by Edison Ling (edisonling)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24314
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24314.patch
Both `mestimate` and `mestimate_d3d12` keep a three-frame
`previous/current/next` window so they can estimate motion against both past
and future frames. This delays output by one frame. When input reaches EOF,
the frame in the `next` slot is never moved to `current`; it is instead
discarded during filter teardown, so every non-empty input produces one fewer
output frame. A single-frame input produces no output and fails conversion.
Add an output `request_frame` callback to both filters. When the upstream link
returns `AVERROR_EOF`, the callback drains the frame still held in `next`.
There is no future reference for that final frame, so the flush path emits
backward motion vectors only and sizes the motion-vector side data for one
direction. The `next` guard makes the drain idempotent.
The software filter's existing cropdetect FATE tests now receive the final
frame, so their references gain one trailing metadata record.
### Reproducing the original issue
The PR is a single commit, so its parent (`HEAD^`) is the pre-fix tree. A
separate worktree can be used without changing the checked-out PR branch:
```sh
git worktree add ../ffmpeg-mestimate-before HEAD^
cd ../ffmpeg-mestimate-before
./configure --enable-gpl
make -j"$(nproc)"
```
Generate exact-length software inputs and compare each input count with the
count after `mestimate`:
```sh
mkdir -p /tmp/mestimate-eof
for n in 1 2 3 12 60 180; do
./ffmpeg -v error -f lavfi \
-i testsrc2=size=176x144:rate=25 \
-frames:v "$n" -pix_fmt yuv420p -y \
"/tmp/mestimate-eof/in-${n}.y4m"
input_count=$(./ffprobe -v error -select_streams v:0 \
-count_frames -show_entries stream=nb_read_frames \
-of csv=p=0 "/tmp/mestimate-eof/in-${n}.y4m")
if ./ffmpeg -v error -i "/tmp/mestimate-eof/in-${n}.y4m" \
-vf mestimate -y "/tmp/mestimate-eof/out-${n}.y4m"; then
output_count=$(./ffprobe -v error -select_streams v:0 \
-count_frames -show_entries stream=nb_read_frames \
-of csv=p=0 "/tmp/mestimate-eof/out-${n}.y4m")
else
output_count=0
fi
printf 'frames=%s input=%s output=%s\n' \
"$n" "$input_count" "$output_count"
done
```
On `HEAD^`, the one-frame case fails with an empty output. Every longer case
reports `output=input-1`.
Run the same commands from the built PR branch. Every tested length should
report `output=input`.
### FATE
Configure with `--enable-gpl`, build, and run:
```sh
make fate-filter-metadata-cropdetect1 \
fate-filter-metadata-cropdetect2 \
SAMPLES=/path/to/fate-suite
```
Both tests pass with this commit. Before updating the references, each test
failed only because one final metadata record was appended; no existing
record changed.
### D3D12 validation
This test requires Windows and a D3D12 adapter supporting video motion
estimation. Use an H.264 input that begins at a keyframe. First create exact
input lengths:
```sh
for n in 1 2 3 12 60 180; do
./ffmpeg.exe -v error -y -i input.mp4 \
-frames:v "$n" -c:v copy \
-bsf:v h264_mp4toannexb -an "in-${n}.h264"
done
```
For each input length, encode once without the filter and once with
`mestimate_d3d12`, using both D3D12 encoders:
```sh
count_frames()
{
./ffprobe.exe -v error -select_streams v:0 -count_frames \
-show_entries stream=nb_read_frames -of csv=p=0 "$1"
}
for encoder in h264_d3d12va hevc_d3d12va; do
for n in 1 2 3 12 60 180; do
./ffmpeg.exe -v error -y \
-hwaccel d3d12va -hwaccel_output_format d3d12 \
-i "in-${n}.h264" \
-c:v "$encoder" -b:v 5M \
"baseline-${encoder}-${n}.mp4"
./ffmpeg.exe -v error -y \
-hwaccel d3d12va -hwaccel_output_format d3d12 \
-i "in-${n}.h264" -vf mestimate_d3d12 \
-c:v "$encoder" -b:v 5M \
"filtered-${encoder}-${n}.mp4"
printf '%s frames=%s baseline=%s filtered=%s\n' \
"$encoder" "$n" \
"$(count_frames "baseline-${encoder}-${n}.mp4")" \
"$(count_frames "filtered-${encoder}-${n}.mp4")"
done
done
```
For both encoders, baseline and filtered counts match at all six lengths. The
one-frame case produces one frame instead of failing.
The flush path was also exercised with a 600-frame input:
```sh
./ffmpeg.exe -v error -y -stream_loop -1 -i input.mp4 \
-frames:v 600 -c:v copy \
-bsf:v h264_mp4toannexb -an in-600.h264
for encoder in h264_d3d12va hevc_d3d12va; do
output="filtered-${encoder}-600.mp4"
./ffmpeg.exe -v error -y \
-hwaccel d3d12va -hwaccel_output_format d3d12 \
-i in-600.h264 -vf mestimate_d3d12 \
-c:v "$encoder" -b:v 5M "$output"
test "$(count_frames "$output")" = 600
./ffmpeg.exe -v error -i "$output" -f null -
done
```
Both filtered outputs contained exactly 600 frames and decoded without errors.
>From 61e2863abad4a63d31fcfa85d6b80d8cea40dd25 Mon Sep 17 00:00:00 2001
From: Edison Ling <[email protected]>
Date: Fri, 28 Aug 2026 18:59:13 -0400
Subject: [PATCH] avfilter/vf_mestimate{,_d3d12}: flush final frame at EOF
Both mestimate filters buffer one frame so they can estimate motion
against a future frame. At EOF, that frame remains in the next slot
and is discarded during uninitialization, making the output one frame
shorter than the input.
Add output request callbacks that drain the buffered frame when input
reaches EOF. Since no future reference exists while draining, emit only
backward motion vectors and size the side data accordingly.
Update the cropdetect FATE references for the newly emitted frame.
---
libavfilter/vf_mestimate.c | 36 +++++++++++++++++++---
libavfilter/vf_mestimate_d3d12.c | 33 +++++++++++++++-----
tests/ref/fate/filter-metadata-cropdetect1 | 1 +
tests/ref/fate/filter-metadata-cropdetect2 | 1 +
4 files changed, 59 insertions(+), 12 deletions(-)
diff --git a/libavfilter/vf_mestimate.c b/libavfilter/vf_mestimate.c
index 413838e956..759883df45 100644
--- a/libavfilter/vf_mestimate.c
+++ b/libavfilter/vf_mestimate.c
@@ -26,7 +26,6 @@
#include "libavutil/motion_vector.h"
#include "avfilter.h"
#include "filters.h"
-#include "video.h"
typedef struct MEContext {
const AVClass *class;
@@ -146,7 +145,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame)
int32_t mv_count = 0;
int ret;
- if (frame->pts == AV_NOPTS_VALUE) {
+ if (frame && frame->pts == AV_NOPTS_VALUE) {
ret = ff_filter_frame(ctx->outputs[0], frame);
return ret;
}
@@ -160,6 +159,8 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame)
s->mv_table[1] = memcpy(s->mv_table[1], s->mv_table[0], sizeof(*s->mv_table[0]) * s->b_count);
if (!s->cur) {
+ if (!frame)
+ return 0;
s->cur = av_frame_clone(frame);
if (!s->cur)
return AVERROR(ENOMEM);
@@ -172,7 +173,11 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame)
if (!out)
return AVERROR(ENOMEM);
- sd = av_frame_new_side_data(out, AV_FRAME_DATA_MOTION_VECTORS, 2 * s->b_count * sizeof(AVMotionVector));
+ /* The last frame has no forward reference. */
+ const int nb_dirs = s->next ? 2 : 1;
+
+ sd = av_frame_new_side_data(out, AV_FRAME_DATA_MOTION_VECTORS,
+ nb_dirs * s->b_count * sizeof(AVMotionVector));
if (!sd) {
av_frame_free(&out);
return AVERROR(ENOMEM);
@@ -181,7 +186,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame)
me_ctx->data_cur = s->cur->data[0];
me_ctx->linesize = s->cur->linesize[0];
- for (dir = 0; dir < 2; dir++) {
+ for (dir = 0; dir < nb_dirs; dir++) {
me_ctx->data_ref = (dir ? s->next : s->prev)->data[0];
if (s->method == AV_ME_METHOD_DS)
@@ -326,6 +331,19 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame)
return ff_filter_frame(ctx->outputs[0], out);
}
+static int request_frame(AVFilterLink *outlink)
+{
+ AVFilterContext *ctx = outlink->src;
+ MEContext *s = ctx->priv;
+ int ret;
+
+ ret = ff_request_frame(ctx->inputs[0]);
+ if (ret == AVERROR_EOF && s->next)
+ ret = filter_frame(ctx->inputs[0], NULL);
+
+ return ret;
+}
+
static av_cold void uninit(AVFilterContext *ctx)
{
MEContext *s = ctx->priv;
@@ -348,6 +366,14 @@ static const AVFilterPad mestimate_inputs[] = {
},
};
+static const AVFilterPad mestimate_outputs[] = {
+ {
+ .name = "default",
+ .type = AVMEDIA_TYPE_VIDEO,
+ .request_frame = request_frame,
+ },
+};
+
const FFFilter ff_vf_mestimate = {
.p.name = "mestimate",
.p.description = NULL_IF_CONFIG_SMALL("Generate motion vectors."),
@@ -356,6 +382,6 @@ const FFFilter ff_vf_mestimate = {
.priv_size = sizeof(MEContext),
.uninit = uninit,
FILTER_INPUTS(mestimate_inputs),
- FILTER_OUTPUTS(ff_video_default_filterpad),
+ FILTER_OUTPUTS(mestimate_outputs),
FILTER_PIXFMTS_ARRAY(pix_fmts),
};
diff --git a/libavfilter/vf_mestimate_d3d12.c b/libavfilter/vf_mestimate_d3d12.c
index 710ba6ff33..6d45a93e39 100644
--- a/libavfilter/vf_mestimate_d3d12.c
+++ b/libavfilter/vf_mestimate_d3d12.c
@@ -561,6 +561,8 @@ static int mestimate_d3d12_filter_frame(AVFilterLink *inlink, AVFrame *frame)
int mb_width, mb_height, mb_count;
if (!s->initialized) {
+ if (!frame)
+ return 0;
err = mestimate_d3d12_config_props(ctx->outputs[0]);
if (err < 0) {
av_frame_free(&frame);
@@ -575,6 +577,8 @@ static int mestimate_d3d12_filter_frame(AVFilterLink *inlink, AVFrame *frame)
s->next_frame = frame;
if (!s->cur_frame) {
+ if (!frame)
+ return 0;
s->cur_frame = av_frame_clone(frame);
if (!s->cur_frame)
return AVERROR(ENOMEM);
@@ -588,13 +592,14 @@ static int mestimate_d3d12_filter_frame(AVFilterLink *inlink, AVFrame *frame)
if (!out)
return AVERROR(ENOMEM);
- mb_width = (frame->width + s->block_size - 1) / s->block_size;
- mb_height = (frame->height + s->block_size - 1) / s->block_size;
+ mb_width = (s->cur_frame->width + s->block_size - 1) / s->block_size;
+ mb_height = (s->cur_frame->height + s->block_size - 1) / s->block_size;
mb_count = mb_width * mb_height;
+ /* The last frame has no forward reference. */
+ const int nb_dirs = s->next_frame ? 2 : 1;
- // Allocate side data for motion vectors (2 directions)
sd = av_frame_new_side_data(out, AV_FRAME_DATA_MOTION_VECTORS,
- 2 * mb_count * sizeof(AVMotionVector));
+ nb_dirs * mb_count * sizeof(AVMotionVector));
if (!sd) {
av_frame_free(&out);
return AVERROR(ENOMEM);
@@ -913,6 +918,19 @@ static int mestimate_d3d12_filter_frame(AVFilterLink *inlink, AVFrame *frame)
return ff_filter_frame(ctx->outputs[0], out);
}
+static int mestimate_d3d12_request_frame(AVFilterLink *outlink)
+{
+ AVFilterContext *ctx = outlink->src;
+ MEstimateD3D12Context *s = ctx->priv;
+ int ret;
+
+ ret = ff_request_frame(ctx->inputs[0]);
+ if (ret == AVERROR_EOF && s->next_frame)
+ ret = mestimate_d3d12_filter_frame(ctx->inputs[0], NULL);
+
+ return ret;
+}
+
static av_cold void mestimate_d3d12_uninit(AVFilterContext *ctx)
{
MEstimateD3D12Context *s = ctx->priv;
@@ -952,9 +970,10 @@ static const AVFilterPad mestimate_d3d12_inputs[] = {
static const AVFilterPad mestimate_d3d12_outputs[] = {
{
- .name = "default",
- .type = AVMEDIA_TYPE_VIDEO,
- .config_props = mestimate_d3d12_config_props,
+ .name = "default",
+ .type = AVMEDIA_TYPE_VIDEO,
+ .config_props = mestimate_d3d12_config_props,
+ .request_frame = mestimate_d3d12_request_frame,
},
};
diff --git a/tests/ref/fate/filter-metadata-cropdetect1 b/tests/ref/fate/filter-metadata-cropdetect1
index 7deebb306c..9ec9e43e72 100644
--- a/tests/ref/fate/filter-metadata-cropdetect1
+++ b/tests/ref/fate/filter-metadata-cropdetect1
@@ -7,3 +7,4 @@ pts=5005|tag:lavfi.cropdetect.y=316|tag:lavfi.cropdetect.x1=20|tag:lavfi.cropdet
pts=6006|tag:lavfi.cropdetect.y=122|tag:lavfi.cropdetect.x1=0|tag:lavfi.cropdetect.x2=885|tag:lavfi.cropdetect.y1=115|tag:lavfi.cropdetect.y2=621|tag:lavfi.cropdetect.w=880|tag:lavfi.cropdetect.h=496|tag:lavfi.cropdetect.x=4|tag:lavfi.cropdetect.limit=0.094118|
pts=7007|tag:lavfi.cropdetect.y=122|tag:lavfi.cropdetect.x1=0|tag:lavfi.cropdetect.x2=885|tag:lavfi.cropdetect.y1=115|tag:lavfi.cropdetect.y2=621|tag:lavfi.cropdetect.w=880|tag:lavfi.cropdetect.h=496|tag:lavfi.cropdetect.x=4|tag:lavfi.cropdetect.limit=0.094118|
pts=8008|tag:lavfi.cropdetect.y=122|tag:lavfi.cropdetect.x1=0|tag:lavfi.cropdetect.x2=885|tag:lavfi.cropdetect.y1=115|tag:lavfi.cropdetect.y2=621|tag:lavfi.cropdetect.w=880|tag:lavfi.cropdetect.h=496|tag:lavfi.cropdetect.x=4|tag:lavfi.cropdetect.limit=0.094118|
+pts=9009|tag:lavfi.cropdetect.y=122|tag:lavfi.cropdetect.x1=0|tag:lavfi.cropdetect.x2=885|tag:lavfi.cropdetect.y1=115|tag:lavfi.cropdetect.y2=621|tag:lavfi.cropdetect.w=880|tag:lavfi.cropdetect.h=496|tag:lavfi.cropdetect.x=4|tag:lavfi.cropdetect.limit=0.094118|
diff --git a/tests/ref/fate/filter-metadata-cropdetect2 b/tests/ref/fate/filter-metadata-cropdetect2
index 42806c16b7..ba6d2851c1 100644
--- a/tests/ref/fate/filter-metadata-cropdetect2
+++ b/tests/ref/fate/filter-metadata-cropdetect2
@@ -7,3 +7,4 @@ pts=2560|tag:lavfi.cropdetect.y=22|tag:lavfi.cropdetect.x1=21|tag:lavfi.cropdete
pts=3072|tag:lavfi.cropdetect.y=22|tag:lavfi.cropdetect.x1=21|tag:lavfi.cropdetect.x2=817|tag:lavfi.cropdetect.y1=15|tag:lavfi.cropdetect.y2=937|tag:lavfi.cropdetect.w=784|tag:lavfi.cropdetect.h=912|tag:lavfi.cropdetect.x=28|tag:lavfi.cropdetect.limit=0.094118|
pts=3584|tag:lavfi.cropdetect.y=40|tag:lavfi.cropdetect.x1=21|tag:lavfi.cropdetect.x2=817|tag:lavfi.cropdetect.y1=38|tag:lavfi.cropdetect.y2=937|tag:lavfi.cropdetect.w=784|tag:lavfi.cropdetect.h=896|tag:lavfi.cropdetect.x=28|tag:lavfi.cropdetect.limit=0.094118|
pts=4096|tag:lavfi.cropdetect.y=22|tag:lavfi.cropdetect.x1=21|tag:lavfi.cropdetect.x2=817|tag:lavfi.cropdetect.y1=15|tag:lavfi.cropdetect.y2=937|tag:lavfi.cropdetect.w=784|tag:lavfi.cropdetect.h=912|tag:lavfi.cropdetect.x=28|tag:lavfi.cropdetect.limit=0.094118|
+pts=4608|tag:lavfi.cropdetect.y=22|tag:lavfi.cropdetect.x1=21|tag:lavfi.cropdetect.x2=1221|tag:lavfi.cropdetect.y1=15|tag:lavfi.cropdetect.y2=1116|tag:lavfi.cropdetect.w=1200|tag:lavfi.cropdetect.h=1088|tag:lavfi.cropdetect.x=22|tag:lavfi.cropdetect.limit=0.094118|
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]