[PR] avfilter/fruc_vulkan: A framerate up-conversion filter using the hardware acceleerated VK_NV_optical_flow extension (PR #23704)
philipl via ffmpeg-devel <[email protected]> Sat, 04 Jul 2026 22:43:04 -0000
| Newsgroups | gmane.comp.video.ffmpeg.devel |
|---|---|
| Message-ID | <178320498535.59.660064450930184251@29965ddac10e> |
PR #23704 opened by philipl URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23704 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23704.patch Too many years ago, I created a filter using nvidia's nvoffruc library to provide hardware accelerated frame rater conversion, which was cute, but unmergeable due to nvidia's poor licensing of their library. They made some vague noises about trying to address this, but it never happened, and they never bothered updating the library so it doesn't work on Blackwell hardware (it includes compiled cuda kernels, so an update would be required for sm_120). @Lynne asked me if I'd be willing to write a Vulkan extension using VK_NV_optical_flow to provide a filter with the same hardware acceleration, but without license encumbrance. Unfortunately, using this extension also requires you to provide your own interpolation implementation, as it only gives you the optical flow. I've never had enough time to sit down and make it happen until now, so here we are. As far as extensions go, it pushes the boundaries a little bit, as it involves cross-queue synchronisation, which might be a pain to wrap your head around (it certainly is for me). Other notable oddities are their use of a custom image format for the optical flow data which can't be directly sampled (even though it claims it can be), and the fact that the optical flow engine creates bogus flows in regions where nothing is going on - requiring heuristics to decide when to ignore the flows. The interpolation implementation is pretty simple, although not complete naive. There are many paths we could go down to make it smarter, or to offer multiple options. Folks who are much more knowledgeable than me can take that on - think of this primarily as getting the scaffolding into place to allow for interesting future work. From a performance perspective, the optical flow calculation itself is the dominant cost, so you can request very high interpolation rates with no real effect on throughput. You just have to tune the optical flow configuration to run fast enough for your content. In particular, you can't do much with 4k content without turning down both the grid size and the perf setting (ie: grid=2:perf=medium is the setting I'd recommend). I ran this all through the validation layers, and it is clean, in so far as anything is clean, given the limitations in the layers and the nvidia drivers. From d1aef9a9773a9fbb6ba9bc7cfdfc5f6916e33e31 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sat, 20 Jun 2026 21:06:48 -0700 Subject: [PATCH 01/14] avutil/hwcontext_vulkan: enable VK_NV_optical_flow Enable VK_NV_optical_flow if it is present, as an optional extension. The extension was added in 1.3.230 which is older than the minimum required header release, so we don't need #ifdef guards. --- libavutil/hwcontext_vulkan.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 20b6ed46f8..0a432cc9bb 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -121,6 +121,7 @@ typedef struct VulkanDeviceFeatures { #ifdef VK_KHR_internally_synchronized_queues VkPhysicalDeviceInternallySynchronizedQueuesFeaturesKHR internal_queue_sync; #endif + VkPhysicalDeviceOpticalFlowFeaturesNV optical_flow; } VulkanDeviceFeatures; typedef struct VulkanDevicePriv { @@ -298,6 +299,9 @@ static void device_features_init(AVHWDeviceContext *ctx, VulkanDeviceFeatures *f FF_VK_STRUCT_EXT(s, &feats->device, &feats->internal_queue_sync, FF_VK_EXT_INTERNAL_QUEUE_SYNC, VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_INTERNALLY_SYNCHRONIZED_QUEUES_FEATURES_KHR); #endif + + FF_VK_STRUCT_EXT(s, &feats->device, &feats->optical_flow, FF_VK_EXT_OPTICAL_FLOW, + VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_OPTICAL_FLOW_FEATURES_NV); } /* Copy all needed device features */ @@ -407,6 +411,8 @@ static void device_features_copy_needed(VulkanDeviceFeatures *dst, VulkanDeviceF COPY_VAL(internal_queue_sync.internallySynchronizedQueues); #endif + COPY_VAL(optical_flow.opticalFlow); + #undef COPY_VAL } @@ -730,6 +736,7 @@ static const VulkanOptExtension optional_device_exts[] = { #ifdef VK_KHR_internally_synchronized_queues { VK_KHR_INTERNALLY_SYNCHRONIZED_QUEUES_EXTENSION_NAME, FF_VK_EXT_INTERNAL_QUEUE_SYNC }, #endif + { VK_NV_OPTICAL_FLOW_EXTENSION_NAME, FF_VK_EXT_OPTICAL_FLOW }, /* Imports/exports */ { VK_KHR_EXTERNAL_MEMORY_FD_EXTENSION_NAME, FF_VK_EXT_EXTERNAL_FD_MEMORY }, @@ -1697,6 +1704,9 @@ static int setup_queue_families(AVHWDeviceContext *ctx, VkDeviceCreateInfo *cd) #endif PICK_QF(VK_QUEUE_VIDEO_DECODE_BIT_KHR, VK_VIDEO_CODEC_OPERATION_DECODE_AV1_BIT_KHR); + if (p->vkctx.extensions & FF_VK_EXT_OPTICAL_FLOW) + PICK_QF(VK_QUEUE_OPTICAL_FLOW_BIT_NV, VK_VIDEO_CODEC_OPERATION_NONE_KHR); + av_free(qf); av_free(qf_vid); -- 2.52.0 From c30aaeb7f0f73f18538861a69cae2d24c61ba915 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sun, 28 Jun 2026 13:23:10 -0700 Subject: [PATCH 02/14] avfilter/fruc_vulkan: copy vf_framerate.c verbatim to vf_fruc_vulkan.c The fruc_vulkan filter will re-use the frame timing logic from vf_framerate - it works so we don't need to do something different. To maximise reviewability, this commit simply copies the file into place, so that all the changes in the subsequent commits can be more easily understood. --- libavfilter/vf_fruc_vulkan.c | 447 +++++++++++++++++++++++++++++++++++ 1 file changed, 447 insertions(+) create mode 100644 libavfilter/vf_fruc_vulkan.c diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c new file mode 100644 index 0000000000..468042e7db --- /dev/null +++ b/libavfilter/vf_fruc_vulkan.c @@ -0,0 +1,447 @@ +/* + * Copyright (C) 2012 Mark Himsley + * + * get_scene_score() Copyright (c) 2011 Stefano Sabatini + * taken from libavfilter/vf_select.c + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * filter for upsampling or downsampling a progressive source + */ + +#define DEBUG + +#include "libavutil/avassert.h" +#include "libavutil/imgutils.h" +#include "libavutil/internal.h" +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" + +#include "avfilter.h" +#include "video.h" +#include "filters.h" +#include "framerate.h" +#include "scene_sad.h" + +#define OFFSET(x) offsetof(FrameRateContext, x) +#define V AV_OPT_FLAG_VIDEO_PARAM +#define F AV_OPT_FLAG_FILTERING_PARAM +#define FRAMERATE_FLAG_SCD 01 + +static const AVOption framerate_options[] = { + {"fps", "required output frames per second rate", OFFSET(dest_frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="50"}, 0, INT_MAX, V|F }, + + {"interp_start", "point to start linear interpolation", OFFSET(interp_start), AV_OPT_TYPE_INT, {.i64=15}, 0, 255, V|F }, + {"interp_end", "point to end linear interpolation", OFFSET(interp_end), AV_OPT_TYPE_INT, {.i64=240}, 0, 255, V|F }, + {"scene", "scene change level", OFFSET(scene_score), AV_OPT_TYPE_DOUBLE, {.dbl=8.2}, 0, 100., V|F }, + + {"flags", "set flags", OFFSET(flags), AV_OPT_TYPE_FLAGS, {.i64=1}, 0, INT_MAX, V|F, .unit = "flags" }, + {"scene_change_detect", "enable scene change detection", 0, AV_OPT_TYPE_CONST, {.i64=FRAMERATE_FLAG_SCD}, INT_MIN, INT_MAX, V|F, .unit = "flags" }, + {"scd", "enable scene change detection", 0, AV_OPT_TYPE_CONST, {.i64=FRAMERATE_FLAG_SCD}, INT_MIN, INT_MAX, V|F, .unit = "flags" }, + + {NULL} +}; + +AVFILTER_DEFINE_CLASS(framerate); + +static double get_scene_score(AVFilterContext *ctx, AVFrame *crnt, AVFrame *next) +{ + FrameRateContext *s = ctx->priv; + double ret = 0; + + ff_dlog(ctx, "get_scene_score()\n"); + + if (crnt->height == next->height && + crnt->width == next->width) { + uint64_t sad; + double mafd, diff; + + ff_dlog(ctx, "get_scene_score() process\n"); + s->sad(crnt->data[0], crnt->linesize[0], next->data[0], next->linesize[0], crnt->width, crnt->height, &sad); + mafd = (double)sad * 100.0 / (crnt->width * crnt->height) / (1 << s->bitdepth); + diff = fabs(mafd - s->prev_mafd); + ret = av_clipf(FFMIN(mafd, diff), 0, 100.0); + s->prev_mafd = mafd; + } + ff_dlog(ctx, "get_scene_score() result is:%f\n", ret); + return ret; +} + +typedef struct ThreadData { + AVFrame *copy_src1, *copy_src2; + uint16_t src1_factor, src2_factor; +} ThreadData; + +static int filter_slice(AVFilterContext *ctx, void *arg, int job, int nb_jobs) +{ + FrameRateContext *s = ctx->priv; + ThreadData *td = arg; + AVFrame *work = s->work; + AVFrame *src1 = td->copy_src1; + AVFrame *src2 = td->copy_src2; + uint16_t src1_factor = td->src1_factor; + uint16_t src2_factor = td->src2_factor; + int plane; + + for (plane = 0; plane < 4 && src1->data[plane] && src2->data[plane]; plane++) { + const int start = (s->height[plane] * job ) / nb_jobs; + const int end = (s->height[plane] * (job+1)) / nb_jobs; + uint8_t *src1_data = src1->data[plane] + start * src1->linesize[plane]; + uint8_t *src2_data = src2->data[plane] + start * src2->linesize[plane]; + uint8_t *dst_data = work->data[plane] + start * work->linesize[plane]; + + s->blend(src1_data, src1->linesize[plane], src2_data, src2->linesize[plane], + dst_data, work->linesize[plane], s->line_size[plane], end - start, + src1_factor, src2_factor, s->blend_factor_max >> 1); + } + + return 0; +} + +static int blend_frames(AVFilterContext *ctx, int interpolate) +{ + FrameRateContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + double interpolate_scene_score = 0; + + if ((s->flags & FRAMERATE_FLAG_SCD)) { + if (s->score >= 0.0) + interpolate_scene_score = s->score; + else + interpolate_scene_score = s->score = get_scene_score(ctx, s->f0, s->f1); + ff_dlog(ctx, "blend_frames() interpolate scene score:%f\n", interpolate_scene_score); + } + // decide if the shot-change detection allows us to blend two frames + if (interpolate_scene_score < s->scene_score) { + ThreadData td; + td.copy_src1 = s->f0; + td.copy_src2 = s->f1; + td.src2_factor = interpolate; + td.src1_factor = s->blend_factor_max - td.src2_factor; + + // get work-space for output frame + s->work = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!s->work) + return AVERROR(ENOMEM); + + av_frame_copy_props(s->work, s->f0); + + ff_dlog(ctx, "blend_frames() INTERPOLATE to create work frame\n"); + ff_filter_execute(ctx, filter_slice, &td, NULL, + FFMIN(FFMAX(1, outlink->h >> 2), ff_filter_get_nb_threads(ctx))); + return 1; + } + return 0; +} + +static int process_work_frame(AVFilterContext *ctx) +{ + FrameRateContext *s = ctx->priv; + int64_t work_pts; + int64_t interpolate, interpolate8; + int ret; + + if (!s->f1) + return 0; + if (!s->f0 && !s->flush) + return 0; + + work_pts = s->start_pts + av_rescale_q(s->n, av_inv_q(s->dest_frame_rate), s->dest_time_base); + + if (work_pts >= s->pts1 && !s->flush) + return 0; + + if (!s->f0) { + av_assert1(s->flush); + s->work = s->f1; + s->f1 = NULL; + } else { + if (work_pts >= s->pts1 + s->delta && s->flush) + return 0; + + interpolate = av_rescale(work_pts - s->pts0, s->blend_factor_max, s->delta); + interpolate8 = av_rescale(work_pts - s->pts0, 256, s->delta); + ff_dlog(ctx, "process_work_frame() interpolate: %"PRId64"/256\n", interpolate8); + if (interpolate >= s->blend_factor_max || interpolate8 > s->interp_end) { + s->work = av_frame_clone(s->f1); + } else if (interpolate <= 0 || interpolate8 < s->interp_start) { + s->work = av_frame_clone(s->f0); + } else { + ret = blend_frames(ctx, interpolate); + if (ret < 0) + return ret; + if (ret == 0) + s->work = av_frame_clone(interpolate > (s->blend_factor_max >> 1) ? s->f1 : s->f0); + } + } + + if (!s->work) + return AVERROR(ENOMEM); + + s->work->pts = work_pts; + s->n++; + + return 1; +} + +static av_cold int init(AVFilterContext *ctx) +{ + FrameRateContext *s = ctx->priv; + s->start_pts = AV_NOPTS_VALUE; + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + FrameRateContext *s = ctx->priv; + av_frame_free(&s->f0); + av_frame_free(&s->f1); +} + +static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_YUV410P, + AV_PIX_FMT_YUV411P, AV_PIX_FMT_YUVJ411P, + AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUVJ420P, + AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUVJ422P, + AV_PIX_FMT_YUV440P, AV_PIX_FMT_YUVJ440P, + AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUVJ444P, + AV_PIX_FMT_YUV420P9, AV_PIX_FMT_YUV420P10, AV_PIX_FMT_YUV420P12, + AV_PIX_FMT_YUV422P9, AV_PIX_FMT_YUV422P10, AV_PIX_FMT_YUV422P12, + AV_PIX_FMT_YUV444P9, AV_PIX_FMT_YUV444P10, AV_PIX_FMT_YUV444P12, + AV_PIX_FMT_NONE +}; + +#define BLEND_FRAME_FUNC(nbits) \ +static void blend_frames##nbits##_c(BLEND_FUNC_PARAMS) \ +{ \ + int line, pixel; \ + uint##nbits##_t *dstw = (uint##nbits##_t *)dst; \ + uint##nbits##_t *src1w = (uint##nbits##_t *)src1; \ + uint##nbits##_t *src2w = (uint##nbits##_t *)src2; \ + int bytes = nbits / 8; \ + width /= bytes; \ + src1_linesize /= bytes; \ + src2_linesize /= bytes; \ + dst_linesize /= bytes; \ + for (line = 0; line < height; line++) { \ + for (pixel = 0; pixel < width; pixel++) \ + dstw[pixel] = ((src1w[pixel] * factor1) + \ + (src2w[pixel] * factor2) + half) \ + >> BLEND_FACTOR_DEPTH(nbits); \ + src1w += src1_linesize; \ + src2w += src2_linesize; \ + dstw += dst_linesize; \ + } \ +} +BLEND_FRAME_FUNC(8) +BLEND_FRAME_FUNC(16) + +void ff_framerate_init(FrameRateContext *s) +{ + if (s->bitdepth == 8) { + s->blend_factor_max = 1 << BLEND_FACTOR_DEPTH(8); + s->blend = blend_frames8_c; + } else { + s->blend_factor_max = 1 << BLEND_FACTOR_DEPTH(16); + s->blend = blend_frames16_c; + } +#if ARCH_X86 && HAVE_X86ASM + ff_framerate_init_x86(s); +#endif +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + FrameRateContext *s = ctx->priv; + const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(inlink->format); + int plane; + + s->vsub = pix_desc->log2_chroma_h; + for (plane = 0; plane < 4; plane++) { + s->line_size[plane] = av_image_get_linesize(inlink->format, inlink->w, plane); + s->height[plane] = inlink->h >> ((plane == 1 || plane == 2) ? s->vsub : 0); + } + + s->bitdepth = pix_desc->comp[0].depth; + + s->sad = ff_scene_sad_get_fn(s->bitdepth); + if (!s->sad) + return AVERROR(EINVAL); + + s->srce_time_base = inlink->time_base; + + ff_framerate_init(s); + + return 0; +} + +static int activate(AVFilterContext *ctx) +{ + int ret, status; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + FrameRateContext *s = ctx->priv; + AVFrame *inpicref; + int64_t pts; + + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); + +retry: + ret = process_work_frame(ctx); + if (ret < 0) + return ret; + else if (ret == 1) + return ff_filter_frame(outlink, s->work); + + ret = ff_inlink_consume_frame(inlink, &inpicref); + if (ret < 0) + return ret; + + if (inpicref) { + if (inpicref->flags & AV_FRAME_FLAG_INTERLACED) + av_log(ctx, AV_LOG_WARNING, "Interlaced frame found - the output will not be correct.\n"); + + if (inpicref->pts == AV_NOPTS_VALUE) { + av_log(ctx, AV_LOG_WARNING, "Ignoring frame without PTS.\n"); + av_frame_free(&inpicref); + } + } + + if (inpicref) { + pts = av_rescale_q(inpicref->pts, s->srce_time_base, s->dest_time_base); + + if (s->f1 && pts == s->pts1) { + av_log(ctx, AV_LOG_WARNING, "Ignoring frame with same PTS.\n"); + av_frame_free(&inpicref); + } + } + + if (inpicref) { + av_frame_free(&s->f0); + s->f0 = s->f1; + s->pts0 = s->pts1; + s->f1 = inpicref; + s->pts1 = pts; + s->delta = s->pts1 - s->pts0; + s->score = -1.0; + + if (s->delta < 0) { + av_log(ctx, AV_LOG_WARNING, "PTS discontinuity.\n"); + s->start_pts = s->pts1; + s->n = 0; + av_frame_free(&s->f0); + } + + if (s->start_pts == AV_NOPTS_VALUE) + s->start_pts = s->pts1; + + goto retry; + } + + if (ff_inlink_acknowledge_status(inlink, &status, &pts)) { + if (!s->flush) { + s->flush = 1; + goto retry; + } + ff_outlink_set_status(outlink, status, pts); + return 0; + } + + FF_FILTER_FORWARD_WANTED(outlink, inlink); + + return FFERROR_NOT_READY; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + FilterLink *l = ff_filter_link(outlink); + FrameRateContext *s = ctx->priv; + int exact; + + ff_dlog(ctx, "config_output()\n"); + + ff_dlog(ctx, + "config_output() input time base:%u/%u (%f)\n", + ctx->inputs[0]->time_base.num,ctx->inputs[0]->time_base.den, + av_q2d(ctx->inputs[0]->time_base)); + + // make sure timebase is small enough to hold the framerate + + exact = av_reduce(&s->dest_time_base.num, &s->dest_time_base.den, + av_gcd((int64_t)s->srce_time_base.num * s->dest_frame_rate.num, + (int64_t)s->srce_time_base.den * s->dest_frame_rate.den ), + (int64_t)s->srce_time_base.den * s->dest_frame_rate.num, INT_MAX); + + av_log(ctx, AV_LOG_INFO, + "time base:%u/%u -> %u/%u exact:%d\n", + s->srce_time_base.num, s->srce_time_base.den, + s->dest_time_base.num, s->dest_time_base.den, exact); + if (!exact) { + av_log(ctx, AV_LOG_WARNING, "Timebase conversion is not exact\n"); + } + + l->frame_rate = s->dest_frame_rate; + outlink->time_base = s->dest_time_base; + + ff_dlog(ctx, + "config_output() output time base:%u/%u (%f) w:%d h:%d\n", + outlink->time_base.num, outlink->time_base.den, + av_q2d(outlink->time_base), + outlink->w, outlink->h); + + + av_log(ctx, AV_LOG_INFO, "fps -> fps:%u/%u scene score:%f interpolate start:%d end:%d\n", + s->dest_frame_rate.num, s->dest_frame_rate.den, + s->scene_score, s->interp_start, s->interp_end); + + return 0; +} + +static const AVFilterPad framerate_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_input, + }, +}; + +static const AVFilterPad framerate_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +const FFFilter ff_vf_framerate = { + .p.name = "framerate", + .p.description = NULL_IF_CONFIG_SMALL("Upsamples or downsamples progressive source between specified frame rates."), + .p.priv_class = &framerate_class, + .p.flags = AVFILTER_FLAG_SLICE_THREADS, + .priv_size = sizeof(FrameRateContext), + .init = init, + .uninit = uninit, + FILTER_INPUTS(framerate_inputs), + FILTER_OUTPUTS(framerate_outputs), + FILTER_PIXFMTS_ARRAY(pix_fmts), + .activate = activate, +}; -- 2.52.0 From aefd210491ffbe99804676db478ec5569146e9bb Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sun, 28 Jun 2026 13:23:58 -0700 Subject: [PATCH 03/14] avfilter/fruc_vulkan: wire up filter as a copy of vf_framerate This is the minimal level of renames required to get the new filter to function as a clone of vf_framerate. The context type is inlined from the header, but there no actual functional changes. --- configure | 1 + libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_fruc_vulkan.c | 106 ++++++++++++++++++++++++----------- 4 files changed, 76 insertions(+), 33 deletions(-) diff --git a/configure b/configure index 070dd3f40f..4b81ea966d 100755 --- a/configure +++ b/configure @@ -4195,6 +4195,7 @@ find_rect_filter_deps="avcodec avformat gpl" flip_vulkan_filter_deps="vulkan spirv_compiler" flite_filter_deps="libflite threads" framerate_filter_select="scene_sad" +fruc_vulkan_filter_select="scene_sad" freezedetect_filter_select="scene_sad" frei0r_deps_any="libdl LoadLibrary" frei0r_filter_deps="frei0r" diff --git a/libavfilter/Makefile b/libavfilter/Makefile index cbae5f4ffd..a8e585fa11 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -332,6 +332,7 @@ OBJS-$(CONFIG_FREEZEFRAMES_FILTER) += vf_freezeframes.o OBJS-$(CONFIG_FREI0R_FILTER) += vf_frei0r.o OBJS-$(CONFIG_FSPP_FILTER) += vf_fspp.o vf_fsppdsp.o qp_table.o OBJS-$(CONFIG_FSYNC_FILTER) += vf_fsync.o +OBJS-$(CONFIG_FRUC_VULKAN_FILTER) += vf_fruc_vulkan.o OBJS-$(CONFIG_GBLUR_FILTER) += vf_gblur.o OBJS-$(CONFIG_GBLUR_VULKAN_FILTER) += vf_gblur_vulkan.o vulkan.o vulkan_filter.o OBJS-$(CONFIG_GEQ_FILTER) += vf_geq.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 402b843649..ac801e3163 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -306,6 +306,7 @@ extern const FFFilter ff_vf_freezeframes; extern const FFFilter ff_vf_frei0r; extern const FFFilter ff_vf_fspp; extern const FFFilter ff_vf_fsync; +extern const FFFilter ff_vf_fruc_vulkan; extern const FFFilter ff_vf_gblur; extern const FFFilter ff_vf_gblur_vulkan; extern const FFFilter ff_vf_geq; diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c index 468042e7db..4244571191 100644 --- a/libavfilter/vf_fruc_vulkan.c +++ b/libavfilter/vf_fruc_vulkan.c @@ -23,11 +23,9 @@ /** * @file - * filter for upsampling or downsampling a progressive source + * Frame rate up-conversion filter. */ -#define DEBUG - #include "libavutil/avassert.h" #include "libavutil/imgutils.h" #include "libavutil/internal.h" @@ -37,15 +35,60 @@ #include "avfilter.h" #include "video.h" #include "filters.h" -#include "framerate.h" #include "scene_sad.h" -#define OFFSET(x) offsetof(FrameRateContext, x) +#define BLEND_FUNC_PARAMS const uint8_t *src1, ptrdiff_t src1_linesize, \ + const uint8_t *src2, ptrdiff_t src2_linesize, \ + uint8_t *dst, ptrdiff_t dst_linesize, \ + ptrdiff_t width, ptrdiff_t height, \ + int factor1, int factor2, int half + +#define BLEND_FACTOR_DEPTH(n) (n-1) + +typedef void (*blend_func)(BLEND_FUNC_PARAMS); + +typedef struct FRUCVulkanContext { + const AVClass *class; + // parameters + AVRational dest_frame_rate; ///< output frames per second + int flags; ///< flags affecting frame rate conversion algorithm + double scene_score; ///< score that denotes a scene change has happened + int interp_start; ///< start of range to apply linear interpolation + int interp_end; ///< end of range to apply linear interpolation + + int line_size[4]; ///< bytes of pixel data per line for each plane + int height[4]; ///< height of each plane + int vsub; + + AVRational srce_time_base; ///< timebase of source + AVRational dest_time_base; ///< timebase of destination + + ff_scene_sad_fn sad; ///< Sum of the absolute difference function (scene detect only) + double prev_mafd; ///< previous MAFD (scene detect only) + + int blend_factor_max; + int bitdepth; + AVFrame *work; + + AVFrame *f0; ///< last frame + AVFrame *f1; ///< current frame + int64_t pts0; ///< last frame pts in dest_time_base + int64_t pts1; ///< current frame pts in dest_time_base + int64_t delta; ///< pts1 to pts0 delta + double score; ///< scene change score (f0 to f1) + int flush; ///< 1 if the filter is being flushed + int64_t start_pts; ///< pts of the first output frame + int64_t n; ///< output frame counter + + blend_func blend; +} FRUCVulkanContext; + +#define OFFSET(x) offsetof(FRUCVulkanContext, x) #define V AV_OPT_FLAG_VIDEO_PARAM #define F AV_OPT_FLAG_FILTERING_PARAM -#define FRAMERATE_FLAG_SCD 01 +#define FRUC_VULKAN_FLAG_SCD 01 -static const AVOption framerate_options[] = { +static const AVOption fruc_vulkan_options[] = { {"fps", "required output frames per second rate", OFFSET(dest_frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="50"}, 0, INT_MAX, V|F }, {"interp_start", "point to start linear interpolation", OFFSET(interp_start), AV_OPT_TYPE_INT, {.i64=15}, 0, 255, V|F }, @@ -53,17 +96,17 @@ static const AVOption framerate_options[] = { {"scene", "scene change level", OFFSET(scene_score), AV_OPT_TYPE_DOUBLE, {.dbl=8.2}, 0, 100., V|F }, {"flags", "set flags", OFFSET(flags), AV_OPT_TYPE_FLAGS, {.i64=1}, 0, INT_MAX, V|F, .unit = "flags" }, - {"scene_change_detect", "enable scene change detection", 0, AV_OPT_TYPE_CONST, {.i64=FRAMERATE_FLAG_SCD}, INT_MIN, INT_MAX, V|F, .unit = "flags" }, - {"scd", "enable scene change detection", 0, AV_OPT_TYPE_CONST, {.i64=FRAMERATE_FLAG_SCD}, INT_MIN, INT_MAX, V|F, .unit = "flags" }, + {"scene_change_detect", "enable scene change detection", 0, AV_OPT_TYPE_CONST, {.i64=FRUC_VULKAN_FLAG_SCD}, INT_MIN, INT_MAX, V|F, .unit = "flags" }, + {"scd", "enable scene change detection", 0, AV_OPT_TYPE_CONST, {.i64=FRUC_VULKAN_FLAG_SCD}, INT_MIN, INT_MAX, V|F, .unit = "flags" }, {NULL} }; -AVFILTER_DEFINE_CLASS(framerate); +AVFILTER_DEFINE_CLASS(fruc_vulkan); static double get_scene_score(AVFilterContext *ctx, AVFrame *crnt, AVFrame *next) { - FrameRateContext *s = ctx->priv; + FRUCVulkanContext *s = ctx->priv; double ret = 0; ff_dlog(ctx, "get_scene_score()\n"); @@ -91,7 +134,7 @@ typedef struct ThreadData { static int filter_slice(AVFilterContext *ctx, void *arg, int job, int nb_jobs) { - FrameRateContext *s = ctx->priv; + FRUCVulkanContext *s = ctx->priv; ThreadData *td = arg; AVFrame *work = s->work; AVFrame *src1 = td->copy_src1; @@ -117,11 +160,11 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int job, int nb_jobs) static int blend_frames(AVFilterContext *ctx, int interpolate) { - FrameRateContext *s = ctx->priv; + FRUCVulkanContext *s = ctx->priv; AVFilterLink *outlink = ctx->outputs[0]; double interpolate_scene_score = 0; - if ((s->flags & FRAMERATE_FLAG_SCD)) { + if ((s->flags & FRUC_VULKAN_FLAG_SCD)) { if (s->score >= 0.0) interpolate_scene_score = s->score; else @@ -153,7 +196,7 @@ static int blend_frames(AVFilterContext *ctx, int interpolate) static int process_work_frame(AVFilterContext *ctx) { - FrameRateContext *s = ctx->priv; + FRUCVulkanContext *s = ctx->priv; int64_t work_pts; int64_t interpolate, interpolate8; int ret; @@ -203,14 +246,14 @@ static int process_work_frame(AVFilterContext *ctx) static av_cold int init(AVFilterContext *ctx) { - FrameRateContext *s = ctx->priv; + FRUCVulkanContext *s = ctx->priv; s->start_pts = AV_NOPTS_VALUE; return 0; } static av_cold void uninit(AVFilterContext *ctx) { - FrameRateContext *s = ctx->priv; + FRUCVulkanContext *s = ctx->priv; av_frame_free(&s->f0); av_frame_free(&s->f1); } @@ -253,7 +296,7 @@ static void blend_frames##nbits##_c(BLEND_FUNC_PARAMS) \ BLEND_FRAME_FUNC(8) BLEND_FRAME_FUNC(16) -void ff_framerate_init(FrameRateContext *s) +static void fruc_vulkan_blend_init(FRUCVulkanContext *s) { if (s->bitdepth == 8) { s->blend_factor_max = 1 << BLEND_FACTOR_DEPTH(8); @@ -262,15 +305,12 @@ void ff_framerate_init(FrameRateContext *s) s->blend_factor_max = 1 << BLEND_FACTOR_DEPTH(16); s->blend = blend_frames16_c; } -#if ARCH_X86 && HAVE_X86ASM - ff_framerate_init_x86(s); -#endif } static int config_input(AVFilterLink *inlink) { AVFilterContext *ctx = inlink->dst; - FrameRateContext *s = ctx->priv; + FRUCVulkanContext *s = ctx->priv; const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(inlink->format); int plane; @@ -288,7 +328,7 @@ static int config_input(AVFilterLink *inlink) s->srce_time_base = inlink->time_base; - ff_framerate_init(s); + fruc_vulkan_blend_init(s); return 0; } @@ -298,7 +338,7 @@ static int activate(AVFilterContext *ctx) int ret, status; AVFilterLink *inlink = ctx->inputs[0]; AVFilterLink *outlink = ctx->outputs[0]; - FrameRateContext *s = ctx->priv; + FRUCVulkanContext *s = ctx->priv; AVFrame *inpicref; int64_t pts; @@ -374,7 +414,7 @@ static int config_output(AVFilterLink *outlink) { AVFilterContext *ctx = outlink->src; FilterLink *l = ff_filter_link(outlink); - FrameRateContext *s = ctx->priv; + FRUCVulkanContext *s = ctx->priv; int exact; ff_dlog(ctx, "config_output()\n"); @@ -416,7 +456,7 @@ static int config_output(AVFilterLink *outlink) return 0; } -static const AVFilterPad framerate_inputs[] = { +static const AVFilterPad fruc_vulkan_inputs[] = { { .name = "default", .type = AVMEDIA_TYPE_VIDEO, @@ -424,7 +464,7 @@ static const AVFilterPad framerate_inputs[] = { }, }; -static const AVFilterPad framerate_outputs[] = { +static const AVFilterPad fruc_vulkan_outputs[] = { { .name = "default", .type = AVMEDIA_TYPE_VIDEO, @@ -432,16 +472,16 @@ static const AVFilterPad framerate_outputs[] = { }, }; -const FFFilter ff_vf_framerate = { - .p.name = "framerate", +const FFFilter ff_vf_fruc_vulkan = { + .p.name = "fruc_vulkan", .p.description = NULL_IF_CONFIG_SMALL("Upsamples or downsamples progressive source between specified frame rates."), - .p.priv_class = &framerate_class, + .p.priv_class = &fruc_vulkan_class, .p.flags = AVFILTER_FLAG_SLICE_THREADS, - .priv_size = sizeof(FrameRateContext), + .priv_size = sizeof(FRUCVulkanContext), .init = init, .uninit = uninit, - FILTER_INPUTS(framerate_inputs), - FILTER_OUTPUTS(framerate_outputs), + FILTER_INPUTS(fruc_vulkan_inputs), + FILTER_OUTPUTS(fruc_vulkan_outputs), FILTER_PIXFMTS_ARRAY(pix_fmts), .activate = activate, }; -- 2.52.0 From 9c6d1b9cbffd5736e27ef3a43c618a71739aedf5 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sun, 28 Jun 2026 15:29:15 -0700 Subject: [PATCH 04/14] avfilter/fruc_vulkan: evaluate the output frame rate as an expression This change enhances the `fps` parameter to support the `source_fps` expressions from `vf_fps`, which are highly useful if you want to request a multiple of the base frame rate. The NTSC/PAL constants from `vf_fps` are not brought over. These are very unlikely to be useful targets for a filter that is increasing the framerate of a video. --- libavfilter/vf_fruc_vulkan.c | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c index 4244571191..2118e4eced 100644 --- a/libavfilter/vf_fruc_vulkan.c +++ b/libavfilter/vf_fruc_vulkan.c @@ -27,6 +27,7 @@ */ #include "libavutil/avassert.h" +#include "libavutil/eval.h" #include "libavutil/imgutils.h" #include "libavutil/internal.h" #include "libavutil/opt.h" @@ -37,6 +38,16 @@ #include "filters.h" #include "scene_sad.h" +static const char *const var_names[] = { + "source_fps", + NULL +}; + +enum var_name { + VAR_SOURCE_FPS, + VARS_NB +}; + #define BLEND_FUNC_PARAMS const uint8_t *src1, ptrdiff_t src1_linesize, \ const uint8_t *src2, ptrdiff_t src2_linesize, \ uint8_t *dst, ptrdiff_t dst_linesize, \ @@ -50,6 +61,7 @@ typedef void (*blend_func)(BLEND_FUNC_PARAMS); typedef struct FRUCVulkanContext { const AVClass *class; // parameters + char *requested_frame_rate; ///< output fps as an expression AVRational dest_frame_rate; ///< output frames per second int flags; ///< flags affecting frame rate conversion algorithm double scene_score; ///< score that denotes a scene change has happened @@ -89,7 +101,8 @@ typedef struct FRUCVulkanContext { #define FRUC_VULKAN_FLAG_SCD 01 static const AVOption fruc_vulkan_options[] = { - {"fps", "required output frames per second rate", OFFSET(dest_frame_rate), AV_OPT_TYPE_VIDEO_RATE, {.str="50"}, 0, INT_MAX, V|F }, + { "fps", "A string describing the desired output frame rate", + OFFSET(requested_frame_rate), AV_OPT_TYPE_STRING, { .str = "60" }, 0, 0, V|F }, {"interp_start", "point to start linear interpolation", OFFSET(interp_start), AV_OPT_TYPE_INT, {.i64=15}, 0, 255, V|F }, {"interp_end", "point to end linear interpolation", OFFSET(interp_end), AV_OPT_TYPE_INT, {.i64=240}, 0, 255, V|F }, @@ -413,8 +426,12 @@ retry: static int config_output(AVFilterLink *outlink) { AVFilterContext *ctx = outlink->src; + AVFilterLink *inlink = ctx->inputs[0]; + FilterLink *il = ff_filter_link(inlink); FilterLink *l = ff_filter_link(outlink); FRUCVulkanContext *s = ctx->priv; + double var_values[VARS_NB], res; + int err; int exact; ff_dlog(ctx, "config_output()\n"); @@ -424,6 +441,22 @@ static int config_output(AVFilterLink *outlink) ctx->inputs[0]->time_base.num,ctx->inputs[0]->time_base.den, av_q2d(ctx->inputs[0]->time_base)); + // The fps option is an expression evaluated against the source frame rate + var_values[VAR_SOURCE_FPS] = av_q2d(il->frame_rate); + err = av_expr_parse_and_eval(&res, s->requested_frame_rate, + var_names, var_values, + NULL, NULL, NULL, NULL, NULL, 0, ctx); + if (err < 0) + return err; + + s->dest_frame_rate = av_d2q(res, INT_MAX); + if (s->dest_frame_rate.num <= 0 || s->dest_frame_rate.den <= 0) { + av_log(ctx, AV_LOG_ERROR, + "Invalid output frame rate '%s' (must evaluate to a positive value)\n", + s->requested_frame_rate); + return AVERROR(EINVAL); + } + // make sure timebase is small enough to hold the framerate exact = av_reduce(&s->dest_time_base.num, &s->dest_time_base.den, -- 2.52.0 From ac422a9c7b25b452925e5eb73c1f66bb46ecb56f Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sun, 28 Jun 2026 15:37:21 -0700 Subject: [PATCH 05/14] avfilter/fruc_vulkan: fall back to a plain timebase when reduction degenerates The output timebase is derived, as in vf_fps / vf_framerate, by reducing the source timebase against the requested frame rate. That reduction can collapse to a degenerate 0/0 rational when the source timebase shares no useful factors with the requested rate, leaving an unusable output timebase that breaks the downstream PTS rescaling. Detect the degenerate result and fall back to the plain 1/fps timebase derived directly from the requested rate, so config_output() always yields a valid timebase. --- libavfilter/vf_fruc_vulkan.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c index 2118e4eced..bd8a26d61b 100644 --- a/libavfilter/vf_fruc_vulkan.c +++ b/libavfilter/vf_fruc_vulkan.c @@ -464,6 +464,16 @@ static int config_output(AVFilterLink *outlink) (int64_t)s->srce_time_base.den * s->dest_frame_rate.den ), (int64_t)s->srce_time_base.den * s->dest_frame_rate.num, INT_MAX); + /* The source-timebase-derived reduction above can collapse to a degenerate + * 0/0 rational (e.g. when the source timebase shares no useful factors with + * the requested rate), which would make the output timebase unusable. Fall + * back to the plain 1/fps timebase in that case so a valid timebase is + * always produced. */ + if (!s->dest_time_base.num || !s->dest_time_base.den) { + exact = av_reduce(&s->dest_time_base.num, &s->dest_time_base.den, + s->dest_frame_rate.den, s->dest_frame_rate.num, INT_MAX); + } + av_log(ctx, AV_LOG_INFO, "time base:%u/%u -> %u/%u exact:%d\n", s->srce_time_base.num, s->srce_time_base.den, -- 2.52.0 From 4198a17d4564057e94ca99e0edce560e67006bdc Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sun, 28 Jun 2026 15:37:21 -0700 Subject: [PATCH 06/14] avfilter/fruc_vulkan: turn the filter into a Vulkan filter This change introduces the basic Vulkan filter machinery, while preserving the original vf_framerate timing logic. At this stage, the filter simply picks the temporally closest input frame for the output. --- configure | 2 +- libavfilter/Makefile | 2 +- libavfilter/vf_fruc_vulkan.c | 315 ++++++++--------------------------- 3 files changed, 67 insertions(+), 252 deletions(-) diff --git a/configure b/configure index 4b81ea966d..d85ce1db2b 100755 --- a/configure +++ b/configure @@ -4195,7 +4195,7 @@ find_rect_filter_deps="avcodec avformat gpl" flip_vulkan_filter_deps="vulkan spirv_compiler" flite_filter_deps="libflite threads" framerate_filter_select="scene_sad" -fruc_vulkan_filter_select="scene_sad" +fruc_vulkan_filter_deps="vulkan spirv_compiler" freezedetect_filter_select="scene_sad" frei0r_deps_any="libdl LoadLibrary" frei0r_filter_deps="frei0r" diff --git a/libavfilter/Makefile b/libavfilter/Makefile index a8e585fa11..78135b4986 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -332,7 +332,7 @@ OBJS-$(CONFIG_FREEZEFRAMES_FILTER) += vf_freezeframes.o OBJS-$(CONFIG_FREI0R_FILTER) += vf_frei0r.o OBJS-$(CONFIG_FSPP_FILTER) += vf_fspp.o vf_fsppdsp.o qp_table.o OBJS-$(CONFIG_FSYNC_FILTER) += vf_fsync.o -OBJS-$(CONFIG_FRUC_VULKAN_FILTER) += vf_fruc_vulkan.o +OBJS-$(CONFIG_FRUC_VULKAN_FILTER) += vf_fruc_vulkan.o vulkan.o vulkan_filter.o OBJS-$(CONFIG_GBLUR_FILTER) += vf_gblur.o OBJS-$(CONFIG_GBLUR_VULKAN_FILTER) += vf_gblur_vulkan.o vulkan.o vulkan_filter.o OBJS-$(CONFIG_GEQ_FILTER) += vf_geq.o diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c index bd8a26d61b..d14fc9e8c3 100644 --- a/libavfilter/vf_fruc_vulkan.c +++ b/libavfilter/vf_fruc_vulkan.c @@ -1,8 +1,7 @@ /* - * Copyright (C) 2012 Mark Himsley + * Copyright (C) 2026 Philip Langdale <[email protected]> * - * get_scene_score() Copyright (c) 2011 Stefano Sabatini - * taken from libavfilter/vf_select.c + * Based on vf_framerate - Copyright (C) 2012 Mark Himsley * * This file is part of FFmpeg. * @@ -23,20 +22,20 @@ /** * @file - * Frame rate up-conversion filter. + * Frame rate up-conversion filter that synthesises intermediate frames using + * the NVIDIA Vulkan optical flow extension (VK_NV_optical_flow). */ #include "libavutil/avassert.h" #include "libavutil/eval.h" -#include "libavutil/imgutils.h" #include "libavutil/internal.h" +#include "libavutil/mem.h" #include "libavutil/opt.h" #include "libavutil/pixdesc.h" -#include "avfilter.h" -#include "video.h" +#include "vulkan_filter.h" #include "filters.h" -#include "scene_sad.h" +#include "video.h" static const char *const var_names[] = { "source_fps", @@ -48,38 +47,18 @@ enum var_name { VARS_NB }; -#define BLEND_FUNC_PARAMS const uint8_t *src1, ptrdiff_t src1_linesize, \ - const uint8_t *src2, ptrdiff_t src2_linesize, \ - uint8_t *dst, ptrdiff_t dst_linesize, \ - ptrdiff_t width, ptrdiff_t height, \ - int factor1, int factor2, int half - -#define BLEND_FACTOR_DEPTH(n) (n-1) - -typedef void (*blend_func)(BLEND_FUNC_PARAMS); - typedef struct FRUCVulkanContext { - const AVClass *class; + FFVulkanContext vkctx; + + int initialized; + // parameters char *requested_frame_rate; ///< output fps as an expression AVRational dest_frame_rate; ///< output frames per second - int flags; ///< flags affecting frame rate conversion algorithm - double scene_score; ///< score that denotes a scene change has happened - int interp_start; ///< start of range to apply linear interpolation - int interp_end; ///< end of range to apply linear interpolation - - int line_size[4]; ///< bytes of pixel data per line for each plane - int height[4]; ///< height of each plane - int vsub; AVRational srce_time_base; ///< timebase of source AVRational dest_time_base; ///< timebase of destination - ff_scene_sad_fn sad; ///< Sum of the absolute difference function (scene detect only) - double prev_mafd; ///< previous MAFD (scene detect only) - - int blend_factor_max; - int bitdepth; AVFrame *work; AVFrame *f0; ///< last frame @@ -87,123 +66,28 @@ typedef struct FRUCVulkanContext { int64_t pts0; ///< last frame pts in dest_time_base int64_t pts1; ///< current frame pts in dest_time_base int64_t delta; ///< pts1 to pts0 delta - double score; ///< scene change score (f0 to f1) int flush; ///< 1 if the filter is being flushed int64_t start_pts; ///< pts of the first output frame int64_t n; ///< output frame counter - - blend_func blend; } FRUCVulkanContext; #define OFFSET(x) offsetof(FRUCVulkanContext, x) -#define V AV_OPT_FLAG_VIDEO_PARAM -#define F AV_OPT_FLAG_FILTERING_PARAM -#define FRUC_VULKAN_FLAG_SCD 01 +#define FLAGS (AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM) static const AVOption fruc_vulkan_options[] = { { "fps", "A string describing the desired output frame rate", - OFFSET(requested_frame_rate), AV_OPT_TYPE_STRING, { .str = "60" }, 0, 0, V|F }, - - {"interp_start", "point to start linear interpolation", OFFSET(interp_start), AV_OPT_TYPE_INT, {.i64=15}, 0, 255, V|F }, - {"interp_end", "point to end linear interpolation", OFFSET(interp_end), AV_OPT_TYPE_INT, {.i64=240}, 0, 255, V|F }, - {"scene", "scene change level", OFFSET(scene_score), AV_OPT_TYPE_DOUBLE, {.dbl=8.2}, 0, 100., V|F }, - - {"flags", "set flags", OFFSET(flags), AV_OPT_TYPE_FLAGS, {.i64=1}, 0, INT_MAX, V|F, .unit = "flags" }, - {"scene_change_detect", "enable scene change detection", 0, AV_OPT_TYPE_CONST, {.i64=FRUC_VULKAN_FLAG_SCD}, INT_MIN, INT_MAX, V|F, .unit = "flags" }, - {"scd", "enable scene change detection", 0, AV_OPT_TYPE_CONST, {.i64=FRUC_VULKAN_FLAG_SCD}, INT_MIN, INT_MAX, V|F, .unit = "flags" }, - - {NULL} + OFFSET(requested_frame_rate), AV_OPT_TYPE_STRING, { .str = "60" }, 0, 0, FLAGS }, + { NULL } }; AVFILTER_DEFINE_CLASS(fruc_vulkan); -static double get_scene_score(AVFilterContext *ctx, AVFrame *crnt, AVFrame *next) +static av_cold int init_filter(AVFilterContext *avctx) { - FRUCVulkanContext *s = ctx->priv; - double ret = 0; + FRUCVulkanContext *s = avctx->priv; - ff_dlog(ctx, "get_scene_score()\n"); + s->initialized = 1; - if (crnt->height == next->height && - crnt->width == next->width) { - uint64_t sad; - double mafd, diff; - - ff_dlog(ctx, "get_scene_score() process\n"); - s->sad(crnt->data[0], crnt->linesize[0], next->data[0], next->linesize[0], crnt->width, crnt->height, &sad); - mafd = (double)sad * 100.0 / (crnt->width * crnt->height) / (1 << s->bitdepth); - diff = fabs(mafd - s->prev_mafd); - ret = av_clipf(FFMIN(mafd, diff), 0, 100.0); - s->prev_mafd = mafd; - } - ff_dlog(ctx, "get_scene_score() result is:%f\n", ret); - return ret; -} - -typedef struct ThreadData { - AVFrame *copy_src1, *copy_src2; - uint16_t src1_factor, src2_factor; -} ThreadData; - -static int filter_slice(AVFilterContext *ctx, void *arg, int job, int nb_jobs) -{ - FRUCVulkanContext *s = ctx->priv; - ThreadData *td = arg; - AVFrame *work = s->work; - AVFrame *src1 = td->copy_src1; - AVFrame *src2 = td->copy_src2; - uint16_t src1_factor = td->src1_factor; - uint16_t src2_factor = td->src2_factor; - int plane; - - for (plane = 0; plane < 4 && src1->data[plane] && src2->data[plane]; plane++) { - const int start = (s->height[plane] * job ) / nb_jobs; - const int end = (s->height[plane] * (job+1)) / nb_jobs; - uint8_t *src1_data = src1->data[plane] + start * src1->linesize[plane]; - uint8_t *src2_data = src2->data[plane] + start * src2->linesize[plane]; - uint8_t *dst_data = work->data[plane] + start * work->linesize[plane]; - - s->blend(src1_data, src1->linesize[plane], src2_data, src2->linesize[plane], - dst_data, work->linesize[plane], s->line_size[plane], end - start, - src1_factor, src2_factor, s->blend_factor_max >> 1); - } - - return 0; -} - -static int blend_frames(AVFilterContext *ctx, int interpolate) -{ - FRUCVulkanContext *s = ctx->priv; - AVFilterLink *outlink = ctx->outputs[0]; - double interpolate_scene_score = 0; - - if ((s->flags & FRUC_VULKAN_FLAG_SCD)) { - if (s->score >= 0.0) - interpolate_scene_score = s->score; - else - interpolate_scene_score = s->score = get_scene_score(ctx, s->f0, s->f1); - ff_dlog(ctx, "blend_frames() interpolate scene score:%f\n", interpolate_scene_score); - } - // decide if the shot-change detection allows us to blend two frames - if (interpolate_scene_score < s->scene_score) { - ThreadData td; - td.copy_src1 = s->f0; - td.copy_src2 = s->f1; - td.src2_factor = interpolate; - td.src1_factor = s->blend_factor_max - td.src2_factor; - - // get work-space for output frame - s->work = ff_get_video_buffer(outlink, outlink->w, outlink->h); - if (!s->work) - return AVERROR(ENOMEM); - - av_frame_copy_props(s->work, s->f0); - - ff_dlog(ctx, "blend_frames() INTERPOLATE to create work frame\n"); - ff_filter_execute(ctx, filter_slice, &td, NULL, - FFMIN(FFMAX(1, outlink->h >> 2), ff_filter_get_nb_threads(ctx))); - return 1; - } return 0; } @@ -211,41 +95,30 @@ static int process_work_frame(AVFilterContext *ctx) { FRUCVulkanContext *s = ctx->priv; int64_t work_pts; - int64_t interpolate, interpolate8; - int ret; + int64_t interpolate8; if (!s->f1) return 0; if (!s->f0 && !s->flush) return 0; - work_pts = s->start_pts + av_rescale_q(s->n, av_inv_q(s->dest_frame_rate), s->dest_time_base); + work_pts = s->start_pts + av_rescale_q(s->n, av_inv_q(s->dest_frame_rate), + s->dest_time_base); if (work_pts >= s->pts1 && !s->flush) return 0; if (!s->f0) { av_assert1(s->flush); - s->work = s->f1; - s->f1 = NULL; + s->work = av_frame_clone(s->f1); } else { if (work_pts >= s->pts1 + s->delta && s->flush) return 0; - interpolate = av_rescale(work_pts - s->pts0, s->blend_factor_max, s->delta); + /* No motion compensation yet: emit the temporally nearest source + * frame. Genuine interpolation replaces this in a later change. */ interpolate8 = av_rescale(work_pts - s->pts0, 256, s->delta); - ff_dlog(ctx, "process_work_frame() interpolate: %"PRId64"/256\n", interpolate8); - if (interpolate >= s->blend_factor_max || interpolate8 > s->interp_end) { - s->work = av_frame_clone(s->f1); - } else if (interpolate <= 0 || interpolate8 < s->interp_start) { - s->work = av_frame_clone(s->f0); - } else { - ret = blend_frames(ctx, interpolate); - if (ret < 0) - return ret; - if (ret == 0) - s->work = av_frame_clone(interpolate > (s->blend_factor_max >> 1) ? s->f1 : s->f0); - } + s->work = av_frame_clone(interpolate8 >= 128 ? s->f1 : s->f0); } if (!s->work) @@ -257,95 +130,6 @@ static int process_work_frame(AVFilterContext *ctx) return 1; } -static av_cold int init(AVFilterContext *ctx) -{ - FRUCVulkanContext *s = ctx->priv; - s->start_pts = AV_NOPTS_VALUE; - return 0; -} - -static av_cold void uninit(AVFilterContext *ctx) -{ - FRUCVulkanContext *s = ctx->priv; - av_frame_free(&s->f0); - av_frame_free(&s->f1); -} - -static const enum AVPixelFormat pix_fmts[] = { - AV_PIX_FMT_YUV410P, - AV_PIX_FMT_YUV411P, AV_PIX_FMT_YUVJ411P, - AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUVJ420P, - AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUVJ422P, - AV_PIX_FMT_YUV440P, AV_PIX_FMT_YUVJ440P, - AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUVJ444P, - AV_PIX_FMT_YUV420P9, AV_PIX_FMT_YUV420P10, AV_PIX_FMT_YUV420P12, - AV_PIX_FMT_YUV422P9, AV_PIX_FMT_YUV422P10, AV_PIX_FMT_YUV422P12, - AV_PIX_FMT_YUV444P9, AV_PIX_FMT_YUV444P10, AV_PIX_FMT_YUV444P12, - AV_PIX_FMT_NONE -}; - -#define BLEND_FRAME_FUNC(nbits) \ -static void blend_frames##nbits##_c(BLEND_FUNC_PARAMS) \ -{ \ - int line, pixel; \ - uint##nbits##_t *dstw = (uint##nbits##_t *)dst; \ - uint##nbits##_t *src1w = (uint##nbits##_t *)src1; \ - uint##nbits##_t *src2w = (uint##nbits##_t *)src2; \ - int bytes = nbits / 8; \ - width /= bytes; \ - src1_linesize /= bytes; \ - src2_linesize /= bytes; \ - dst_linesize /= bytes; \ - for (line = 0; line < height; line++) { \ - for (pixel = 0; pixel < width; pixel++) \ - dstw[pixel] = ((src1w[pixel] * factor1) + \ - (src2w[pixel] * factor2) + half) \ - >> BLEND_FACTOR_DEPTH(nbits); \ - src1w += src1_linesize; \ - src2w += src2_linesize; \ - dstw += dst_linesize; \ - } \ -} -BLEND_FRAME_FUNC(8) -BLEND_FRAME_FUNC(16) - -static void fruc_vulkan_blend_init(FRUCVulkanContext *s) -{ - if (s->bitdepth == 8) { - s->blend_factor_max = 1 << BLEND_FACTOR_DEPTH(8); - s->blend = blend_frames8_c; - } else { - s->blend_factor_max = 1 << BLEND_FACTOR_DEPTH(16); - s->blend = blend_frames16_c; - } -} - -static int config_input(AVFilterLink *inlink) -{ - AVFilterContext *ctx = inlink->dst; - FRUCVulkanContext *s = ctx->priv; - const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(inlink->format); - int plane; - - s->vsub = pix_desc->log2_chroma_h; - for (plane = 0; plane < 4; plane++) { - s->line_size[plane] = av_image_get_linesize(inlink->format, inlink->w, plane); - s->height[plane] = inlink->h >> ((plane == 1 || plane == 2) ? s->vsub : 0); - } - - s->bitdepth = pix_desc->comp[0].depth; - - s->sad = ff_scene_sad_get_fn(s->bitdepth); - if (!s->sad) - return AVERROR(EINVAL); - - s->srce_time_base = inlink->time_base; - - fruc_vulkan_blend_init(s); - - return 0; -} - static int activate(AVFilterContext *ctx) { int ret, status; @@ -394,7 +178,6 @@ retry: s->f1 = inpicref; s->pts1 = pts; s->delta = s->pts1 - s->pts0; - s->score = -1.0; if (s->delta < 0) { av_log(ctx, AV_LOG_WARNING, "PTS discontinuity.\n"); @@ -423,12 +206,22 @@ retry: return FFERROR_NOT_READY; } +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + FRUCVulkanContext *s = ctx->priv; + + s->srce_time_base = inlink->time_base; + + return ff_vk_filter_config_input(inlink); +} + static int config_output(AVFilterLink *outlink) { AVFilterContext *ctx = outlink->src; AVFilterLink *inlink = ctx->inputs[0]; FilterLink *il = ff_filter_link(inlink); - FilterLink *l = ff_filter_link(outlink); + FilterLink *ol = ff_filter_link(outlink); FRUCVulkanContext *s = ctx->priv; double var_values[VARS_NB], res; int err; @@ -482,7 +275,11 @@ static int config_output(AVFilterLink *outlink) av_log(ctx, AV_LOG_WARNING, "Timebase conversion is not exact\n"); } - l->frame_rate = s->dest_frame_rate; + err = ff_vk_filter_config_output(outlink); + if (err < 0) + return err; + + ol->frame_rate = s->dest_frame_rate; outlink->time_base = s->dest_time_base; ff_dlog(ctx, @@ -491,12 +288,29 @@ static int config_output(AVFilterLink *outlink) av_q2d(outlink->time_base), outlink->w, outlink->h); + return init_filter(ctx); +} - av_log(ctx, AV_LOG_INFO, "fps -> fps:%u/%u scene score:%f interpolate start:%d end:%d\n", - s->dest_frame_rate.num, s->dest_frame_rate.den, - s->scene_score, s->interp_start, s->interp_end); +static av_cold int init(AVFilterContext *avctx) +{ + FRUCVulkanContext *s = avctx->priv; - return 0; + s->start_pts = AV_NOPTS_VALUE; + + return ff_vk_filter_init(avctx); +} + +static av_cold void uninit(AVFilterContext *avctx) +{ + FRUCVulkanContext *s = avctx->priv; + FFVulkanContext *vkctx = &s->vkctx; + + ff_vk_uninit(vkctx); + + av_frame_free(&s->f0); + av_frame_free(&s->f1); + + s->initialized = 0; } static const AVFilterPad fruc_vulkan_inputs[] = { @@ -517,14 +331,15 @@ static const AVFilterPad fruc_vulkan_outputs[] = { const FFFilter ff_vf_fruc_vulkan = { .p.name = "fruc_vulkan", - .p.description = NULL_IF_CONFIG_SMALL("Upsamples or downsamples progressive source between specified frame rates."), + .p.description = NULL_IF_CONFIG_SMALL("Frame rate up-conversion using the Vulkan NV optical flow extension"), .p.priv_class = &fruc_vulkan_class, - .p.flags = AVFILTER_FLAG_SLICE_THREADS, + .p.flags = AVFILTER_FLAG_HWDEVICE, .priv_size = sizeof(FRUCVulkanContext), .init = init, .uninit = uninit, FILTER_INPUTS(fruc_vulkan_inputs), FILTER_OUTPUTS(fruc_vulkan_outputs), - FILTER_PIXFMTS_ARRAY(pix_fmts), + FILTER_SINGLE_PIXFMT(AV_PIX_FMT_VULKAN), .activate = activate, + .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE, }; -- 2.52.0 From adfed419a51d8e76e0064f68dcd4e3b4018baf61 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Wed, 24 Jun 2026 07:59:37 -0700 Subject: [PATCH 07/14] avfilter/fruc_vulkan: add the initial optical-flow pipeline Add the Vulkan optical-flow machinery on top of the filter skeleton: * a grayscale-extraction compute shader * the VK_NV_optical_flow session and its persistent input/flow images * the cross-queue (compute -> optical flow -> compute) submission with binary-semaphore ordering * an interpolation compute shader dispatched per output frame which blends each new frame instead of repeating the nearest source frame Notable characteristics: * At this stage we are doing a simple proportional blend that doesn't actually look at the optical flow information. That will happen in the next commit * The optical flow is stored in a weird nvidia specific fixed point image format. Yay? * Optical flow is calculated based purely on Luma. For YUV format, this is trivially obtained, but for RGB input, we should put some effort into getting accurate Luma. But we only need relative Luma differences, so we don't have to use the exactly correct colourspace conversion; I've just hard-coded BT.709 * We're doing host-synchronisation for initial simplicity, but this will get replaced in subsequent changes --- doc/filters.texi | 17 + libavfilter/vf_fruc_vulkan.c | 831 +++++++++++++++++- libavfilter/vulkan/Makefile | 2 + libavfilter/vulkan/fruc_grayscale.comp.glsl | 54 ++ libavfilter/vulkan/fruc_interpolate.comp.glsl | 65 ++ 5 files changed, 963 insertions(+), 6 deletions(-) create mode 100644 libavfilter/vulkan/fruc_grayscale.comp.glsl create mode 100644 libavfilter/vulkan/fruc_interpolate.comp.glsl diff --git a/doc/filters.texi b/doc/filters.texi index aa0059f9cc..7fa22de494 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -29391,6 +29391,23 @@ Flips an image horizontally. Flips an image along both the vertical and horizontal axis. +@section fruc_vulkan + +Frame rate up-conversion using the NVIDIA Vulkan optical flow extension +(@code{VK_NV_optical_flow}), implemented on the GPU using Vulkan. + +For each pair of consecutive input frames the filter computes a forward and +backward optical flow field on the device's optical flow engine, and uses those +fields to synthesise motion-compensated intermediate frames at the requested +output frame rate. The frame timing logic mirrors the @code{framerate} filter. + +@table @option +@item fps +A string describing the desired output frame rate, evaluated as an expression. +The following constants are available: @code{source_fps}. The default value is +@code{60}. +@end table + @section gblur_vulkan Apply Gaussian blur filter on Vulkan frames. diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c index d14fc9e8c3..b0b0018ec4 100644 --- a/libavfilter/vf_fruc_vulkan.c +++ b/libavfilter/vf_fruc_vulkan.c @@ -37,6 +37,11 @@ #include "filters.h" #include "video.h" +extern const unsigned char ff_fruc_grayscale_comp_spv_data[]; +extern const unsigned int ff_fruc_grayscale_comp_spv_len; +extern const unsigned char ff_fruc_interpolate_comp_spv_data[]; +extern const unsigned int ff_fruc_interpolate_comp_spv_len; + static const char *const var_names[] = { "source_fps", NULL @@ -47,11 +52,67 @@ enum var_name { VARS_NB }; +typedef struct GrayscalePushData { + float luma_weights[4]; +} GrayscalePushData; + +typedef struct InterpolatePushData { + float t; + float luma_size[2]; + int32_t planes; +} InterpolatePushData; + typedef struct FRUCVulkanContext { FFVulkanContext vkctx; int initialized; + AVVulkanDeviceQueueFamily *qf; ///< compute queue family + AVVulkanDeviceQueueFamily *qf_of; ///< optical flow queue family + + /* Queue families that share the optical flow images, so the images can be + * written on the compute queue and read/written on the optical flow queue + * without explicit queue family ownership transfers. */ + uint32_t share_qfs[2]; + int nb_share_qfs; + FFVkExecPool e; ///< compute execution pool + FFVkExecPool e_of; ///< optical flow execution pool + + FFVulkanShader grayscale; + FFVulkanShader interpolate; + VkSampler sampler; ///< linear sampler for the video planes + VkSampler flow_sampler; ///< nearest sampler for the flow vectors + + /* Binary semaphores used to order the cross-queue submissions and, more + * importantly, to make each stage's writes visible to the next: the host + * fence waits alone do not provide cross-queue memory visibility. */ + VkSemaphore sem_gray; ///< grayscale (compute) -> optical flow + VkSemaphore sem_flow; ///< optical flow -> interpolation (compute) + + /* Optical flow session and associated images. */ + VkOpticalFlowSessionNV session; + VkOpticalFlowGridSizeFlagsNV grid_bit; + int grid_size; + VkFormat input_format; ///< grayscale input format + VkFormat flow_format; ///< flow vector format + + int width; ///< luma width + int height; ///< luma height + int flow_width; + int flow_height; + float luma_weights[4]; ///< RGB->Y weights for the grayscale pass + + VkImage gray_img[2]; ///< grayscale inputs (INPUT, REFERENCE) + VkDeviceMemory gray_mem[2]; + VkImageView gray_view[2]; + + VkImage flow_img[2]; ///< [0] forward, [1] backward + VkDeviceMemory flow_mem[2]; + VkImageView flow_view[2]; ///< native (SFIXED5) view, bound to the OF session + VkImageView flow_sint_view[2]; ///< R16G16_SINT reinterpret view for sampling + + int flow_valid; ///< flow computed for current (f0, f1) pair + // parameters char *requested_frame_rate; ///< output fps as an expression AVRational dest_frame_rate; ///< output frames per second @@ -82,20 +143,722 @@ static const AVOption fruc_vulkan_options[] = { AVFILTER_DEFINE_CLASS(fruc_vulkan); +static VkFormat pick_of_format(FRUCVulkanContext *s, VkOpticalFlowUsageFlagsNV usage, + VkFormat preferred) +{ + FFVulkanContext *vkctx = &s->vkctx; + FFVulkanFunctions *vk = &vkctx->vkfn; + VkOpticalFlowImageFormatInfoNV info = { + .sType = VK_STRUCTURE_TYPE_OPTICAL_FLOW_IMAGE_FORMAT_INFO_NV, + .usage = usage, + }; + VkOpticalFlowImageFormatPropertiesNV *props; + VkFormat result = VK_FORMAT_UNDEFINED; + uint32_t count = 0; + + vk->GetPhysicalDeviceOpticalFlowImageFormatsNV(vkctx->hwctx->phys_dev, &info, + &count, NULL); + if (!count) + return VK_FORMAT_UNDEFINED; + + props = av_calloc(count, sizeof(*props)); + if (!props) + return VK_FORMAT_UNDEFINED; + for (uint32_t i = 0; i < count; i++) + props[i].sType = VK_STRUCTURE_TYPE_OPTICAL_FLOW_IMAGE_FORMAT_PROPERTIES_NV; + + vk->GetPhysicalDeviceOpticalFlowImageFormatsNV(vkctx->hwctx->phys_dev, &info, + &count, props); + + result = props[0].format; + for (uint32_t i = 0; i < count; i++) { + av_log(s, AV_LOG_VERBOSE, "Optical flow usage 0x%x supports format %d\n", + usage, props[i].format); + if (props[i].format == preferred) { + result = preferred; + break; + } + } + + av_free(props); + return result; +} + +static int create_of_image(FRUCVulkanContext *s, VkImage *img, VkDeviceMemory *mem, + VkImageView *view, VkFormat format, int width, int height, + VkOpticalFlowUsageFlagsNV of_usage, VkImageUsageFlags usage, + VkImageCreateFlags create_flags) +{ + FFVulkanContext *vkctx = &s->vkctx; + FFVulkanFunctions *vk = &vkctx->vkfn; + AVVulkanDeviceContext *hwctx = vkctx->hwctx; + VkResult ret; + int err; + + VkOpticalFlowImageFormatInfoNV of_info = { + .sType = VK_STRUCTURE_TYPE_OPTICAL_FLOW_IMAGE_FORMAT_INFO_NV, + .usage = of_usage, + }; + VkImageCreateInfo create_info = { + .sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, + .pNext = &of_info, + .flags = create_flags, + .imageType = VK_IMAGE_TYPE_2D, + .format = format, + .extent = { width, height, 1 }, + .mipLevels = 1, + .arrayLayers = 1, + .samples = VK_SAMPLE_COUNT_1_BIT, + .tiling = VK_IMAGE_TILING_OPTIMAL, + .usage = usage, + .initialLayout = VK_IMAGE_LAYOUT_UNDEFINED, + .sharingMode = s->nb_share_qfs > 1 ? VK_SHARING_MODE_CONCURRENT : + VK_SHARING_MODE_EXCLUSIVE, + .queueFamilyIndexCount = s->nb_share_qfs, + .pQueueFamilyIndices = s->share_qfs, + }; + VkImageMemoryRequirementsInfo2 req_desc = { + .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_REQUIREMENTS_INFO_2, + }; + VkMemoryRequirements2 req = { + .sType = VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2, + }; + VkImageViewCreateInfo view_info = { + .sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO, + .viewType = VK_IMAGE_VIEW_TYPE_2D, + .format = format, + .components = ff_comp_identity_map, + .subresourceRange = { + .aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, + .levelCount = 1, + .layerCount = 1, + }, + }; + + ret = vk->CreateImage(hwctx->act_dev, &create_info, hwctx->alloc, img); + if (ret != VK_SUCCESS) { + av_log(s, AV_LOG_ERROR, "Failed to create optical flow image: %s\n", + ff_vk_ret2str(ret)); + return AVERROR_EXTERNAL; + } + + req_desc.image = *img; + vk->GetImageMemoryRequirements2(hwctx->act_dev, &req_desc, &req); + + err = ff_vk_alloc_mem(vkctx, &req.memoryRequirements, + VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT, NULL, NULL, mem); + if (err < 0) + return err; + + ret = vk->BindImageMemory2(hwctx->act_dev, 1, &(VkBindImageMemoryInfo) { + .sType = VK_STRUCTURE_TYPE_BIND_IMAGE_MEMORY_INFO, + .image = *img, + .memory = *mem, + .memoryOffset = 0, + }); + if (ret != VK_SUCCESS) { + av_log(s, AV_LOG_ERROR, "Failed to bind optical flow image memory: %s\n", + ff_vk_ret2str(ret)); + return AVERROR_EXTERNAL; + } + + view_info.image = *img; + ret = vk->CreateImageView(hwctx->act_dev, &view_info, hwctx->alloc, view); + if (ret != VK_SUCCESS) { + av_log(s, AV_LOG_ERROR, "Failed to create optical flow image view: %s\n", + ff_vk_ret2str(ret)); + return AVERROR_EXTERNAL; + } + + return 0; +} + +/* Transition the persistent optical flow images to VK_IMAGE_LAYOUT_GENERAL, + * which is the layout they remain in for the lifetime of the filter. */ +static int init_image_layouts(FRUCVulkanContext *s) +{ + FFVulkanContext *vkctx = &s->vkctx; + FFVulkanFunctions *vk = &vkctx->vkfn; + FFVkExecContext *exec = ff_vk_exec_get(vkctx, &s->e); + VkImage imgs[4] = { s->gray_img[0], s->gray_img[1], + s->flow_img[0], s->flow_img[1] }; + VkImageMemoryBarrier2 bar[4]; + int err; + + ff_vk_exec_start(vkctx, exec); + + for (int i = 0; i < 4; i++) { + bar[i] = (VkImageMemoryBarrier2) { + .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, + .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .srcAccessMask = 0, + .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, + .oldLayout = VK_IMAGE_LAYOUT_UNDEFINED, + .newLayout = VK_IMAGE_LAYOUT_GENERAL, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = imgs[i], + .subresourceRange = { + .aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, + .levelCount = 1, + .layerCount = 1, + }, + }; + } + + vk->CmdPipelineBarrier2(exec->buf, &(VkDependencyInfo) { + .sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, + .pImageMemoryBarriers = bar, + .imageMemoryBarrierCount = 4, + }); + + err = ff_vk_exec_submit(vkctx, exec); + if (err < 0) + return err; + ff_vk_exec_wait(vkctx, exec); + + return 0; +} + static av_cold int init_filter(AVFilterContext *avctx) { + int err; FRUCVulkanContext *s = avctx->priv; + FFVulkanContext *vkctx = &s->vkctx; + FFVulkanFunctions *vk = &vkctx->vkfn; + const int planes = av_pix_fmt_count_planes(vkctx->output_format); + VkOpticalFlowGridSizeFlagsNV grids; + VkResult ret; + + /* config_output() may run more than once (link reconfiguration); the + * Vulkan resources are allocated once and never reallocated here, so a + * second entry would leak the previous set. */ + if (s->initialized) + return 0; + + s->width = vkctx->output_width; + s->height = vkctx->output_height; + + /* Optical flow tracks luma; plane 0 already is luma for YUV. For RGB, derive + * it so the flow follows brightness. The value only feeds the flow engine and + * is never written out, so a fixed BT.709 matrix suffices for all RGB inputs. */ + { + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(vkctx->output_format); + if (desc && (desc->flags & AV_PIX_FMT_FLAG_RGB)) { + s->luma_weights[0] = 0.2126f; + s->luma_weights[1] = 0.7152f; + s->luma_weights[2] = 0.0722f; + s->luma_weights[3] = 0.0f; + } else { + s->luma_weights[0] = 1.0f; + s->luma_weights[1] = 0.0f; + s->luma_weights[2] = 0.0f; + s->luma_weights[3] = 0.0f; + } + } + + if (!(vkctx->extensions & FF_VK_EXT_OPTICAL_FLOW)) { + av_log(avctx, AV_LOG_ERROR, "Vulkan device does not support the " + "VK_NV_optical_flow extension\n"); + return AVERROR(ENOTSUP); + } + + s->qf = ff_vk_qf_find(vkctx, VK_QUEUE_COMPUTE_BIT, 0); + if (!s->qf) { + av_log(avctx, AV_LOG_ERROR, "Device has no compute queues\n"); + return AVERROR(ENOTSUP); + } + + s->qf_of = ff_vk_qf_find(vkctx, VK_QUEUE_OPTICAL_FLOW_BIT_NV, 0); + if (!s->qf_of) { + av_log(avctx, AV_LOG_ERROR, "Device has no optical flow queues\n"); + return AVERROR(ENOTSUP); + } + + s->share_qfs[0] = s->qf->idx; + s->nb_share_qfs = 1; + if (s->qf_of->idx != s->qf->idx) + s->share_qfs[s->nb_share_qfs++] = s->qf_of->idx; + + RET(ff_vk_exec_pool_init(vkctx, s->qf, &s->e, s->qf->num, 0, 0, 0, NULL)); + RET(ff_vk_exec_pool_init(vkctx, s->qf_of, &s->e_of, 1, 0, 0, 0, NULL)); + RET(ff_vk_init_sampler(vkctx, &s->sampler, 0, VK_FILTER_LINEAR)); + /* Flow is sampled through an integer view and its format has no linear + * filtering support, so use nearest. Requires normalised coords and as we + * have only one mip level, the mipmap filtering mode is irrelevant. */ + RET(ff_vk_init_sampler(vkctx, &s->flow_sampler, 0, VK_FILTER_NEAREST)); + + { + VkSemaphoreCreateInfo sem_info = { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO, + }; + if (vk->CreateSemaphore(vkctx->hwctx->act_dev, &sem_info, + vkctx->hwctx->alloc, &s->sem_gray) != VK_SUCCESS || + vk->CreateSemaphore(vkctx->hwctx->act_dev, &sem_info, + vkctx->hwctx->alloc, &s->sem_flow) != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Failed to create synchronization semaphores\n"); + return AVERROR_EXTERNAL; + } + } + + /* Select the optical flow image formats and grid size. */ + s->input_format = pick_of_format(s, VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV, + VK_FORMAT_R8_UNORM); + if (s->input_format != VK_FORMAT_R8_UNORM) { + av_log(avctx, AV_LOG_ERROR, "Optical flow R8 input format unavailable\n"); + return AVERROR(ENOTSUP); + } + /* The engine emits flow vectors as signed fixed point (SFIXED5); theoretically + * it could be something else, but this is all we've seen on real hardware. */ + s->flow_format = pick_of_format(s, VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV, + VK_FORMAT_R16G16_SFIXED5_NV); + if (s->flow_format != VK_FORMAT_R16G16_SFIXED5_NV) { + av_log(avctx, AV_LOG_ERROR, "Optical flow SFIXED5 vector format " + "unavailable (got %d)\n", s->flow_format); + return AVERROR(ENOTSUP); + } + + grids = vkctx->optical_flow_props.supportedOutputGridSizes; + if (grids & VK_OPTICAL_FLOW_GRID_SIZE_1X1_BIT_NV) { + s->grid_size = 1; + s->grid_bit = VK_OPTICAL_FLOW_GRID_SIZE_1X1_BIT_NV; + } else if (grids & VK_OPTICAL_FLOW_GRID_SIZE_2X2_BIT_NV) { + s->grid_size = 2; + s->grid_bit = VK_OPTICAL_FLOW_GRID_SIZE_2X2_BIT_NV; + } else if (grids & VK_OPTICAL_FLOW_GRID_SIZE_4X4_BIT_NV) { + s->grid_size = 4; + s->grid_bit = VK_OPTICAL_FLOW_GRID_SIZE_4X4_BIT_NV; + } else if (grids & VK_OPTICAL_FLOW_GRID_SIZE_8X8_BIT_NV) { + s->grid_size = 8; + s->grid_bit = VK_OPTICAL_FLOW_GRID_SIZE_8X8_BIT_NV; + } else { + av_log(avctx, AV_LOG_ERROR, "No supported optical flow output grid size\n"); + return AVERROR(ENOTSUP); + } + + s->flow_width = (s->width + s->grid_size - 1) / s->grid_size; + s->flow_height = (s->height + s->grid_size - 1) / s->grid_size; + + av_log(avctx, AV_LOG_INFO, "optical flow: grid %d, flow %dx%d, bidir=%d, " + "min %dx%d max %dx%d\n", s->grid_size, s->flow_width, s->flow_height, + vkctx->optical_flow_props.bidirectionalFlowSupported, + vkctx->optical_flow_props.minWidth, vkctx->optical_flow_props.minHeight, + vkctx->optical_flow_props.maxWidth, vkctx->optical_flow_props.maxHeight); + + /* Create the persistent optical flow images. */ + RET(create_of_image(s, &s->gray_img[0], &s->gray_mem[0], &s->gray_view[0], + s->input_format, s->width, s->height, + VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV, + VK_IMAGE_USAGE_STORAGE_BIT, 0)); + RET(create_of_image(s, &s->gray_img[1], &s->gray_mem[1], &s->gray_view[1], + s->input_format, s->width, s->height, + VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV, + VK_IMAGE_USAGE_STORAGE_BIT, 0)); + /* The flow images must be created mutable, as we need to sample the raw + * integers through an R16G16_SINT view. SFIXED5 advertises SAMPLED_IMAGE, + * but if you try and use a float sampler, it will read the values as + * R16G16_SFLOAT, resulting in garbage. So we have to read the raw bits and + * rescale them (value/32) ourselves. */ + RET(create_of_image(s, &s->flow_img[0], &s->flow_mem[0], &s->flow_view[0], + s->flow_format, s->flow_width, s->flow_height, + VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV, + VK_IMAGE_USAGE_SAMPLED_BIT, VK_IMAGE_CREATE_MUTABLE_FORMAT_BIT)); + RET(create_of_image(s, &s->flow_img[1], &s->flow_mem[1], &s->flow_view[1], + s->flow_format, s->flow_width, s->flow_height, + VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV, + VK_IMAGE_USAGE_SAMPLED_BIT, VK_IMAGE_CREATE_MUTABLE_FORMAT_BIT)); + + for (int i = 0; i < 2; i++) { + VkImageViewCreateInfo view_info = { + .sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO, + .image = s->flow_img[i], + .viewType = VK_IMAGE_VIEW_TYPE_2D, + .format = VK_FORMAT_R16G16_SINT, + .components = ff_comp_identity_map, + .subresourceRange = { + .aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, + .levelCount = 1, + .layerCount = 1, + }, + }; + if (vk->CreateImageView(vkctx->hwctx->act_dev, &view_info, + vkctx->hwctx->alloc, &s->flow_sint_view[i]) != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Failed to create flow SINT view\n"); + return AVERROR_EXTERNAL; + } + } + + RET(init_image_layouts(s)); + + /* Create the optical flow session, requesting forward and backward flow. */ + ret = vk->CreateOpticalFlowSessionNV(vkctx->hwctx->act_dev, + &(VkOpticalFlowSessionCreateInfoNV) { + .sType = VK_STRUCTURE_TYPE_OPTICAL_FLOW_SESSION_CREATE_INFO_NV, + .width = s->width, + .height = s->height, + .imageFormat = s->input_format, + .flowVectorFormat = s->flow_format, + .outputGridSize = s->grid_bit, + .performanceLevel = VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_SLOW_NV, + .flags = VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NV, + }, vkctx->hwctx->alloc, &s->session); + if (ret != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Failed to create optical flow session: %s\n", + ff_vk_ret2str(ret)); + return AVERROR_EXTERNAL; + } + + vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, s->session, + VK_OPTICAL_FLOW_SESSION_BINDING_POINT_INPUT_NV, + s->gray_view[0], VK_IMAGE_LAYOUT_GENERAL); + vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, s->session, + VK_OPTICAL_FLOW_SESSION_BINDING_POINT_REFERENCE_NV, + s->gray_view[1], VK_IMAGE_LAYOUT_GENERAL); + vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, s->session, + VK_OPTICAL_FLOW_SESSION_BINDING_POINT_FLOW_VECTOR_NV, + s->flow_view[0], VK_IMAGE_LAYOUT_GENERAL); + vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, s->session, + VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_FLOW_VECTOR_NV, + s->flow_view[1], VK_IMAGE_LAYOUT_GENERAL); + + /* Grayscale extraction shader. */ + ff_vk_shader_load(&s->grayscale, VK_SHADER_STAGE_COMPUTE_BIT, NULL, + (uint32_t []) { 32, 32, 1 }, 0); + ff_vk_shader_add_push_const(&s->grayscale, 0, sizeof(GrayscalePushData), + VK_SHADER_STAGE_COMPUTE_BIT); + { + const FFVulkanDescriptorSetBinding desc[] = { + { + .type = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, + .dimensions = 2, + .elems = 2, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + .samplers = DUP_SAMPLER(s->sampler), + }, + { + .type = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, + .mem_layout = "r8", + .mem_quali = "writeonly", + .dimensions = 2, + .elems = 2, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + }, + }; + RET(ff_vk_shader_add_descriptor_set(vkctx, &s->grayscale, desc, 2, 0, 0)); + } + RET(ff_vk_shader_link(vkctx, &s->grayscale, + ff_fruc_grayscale_comp_spv_data, + ff_fruc_grayscale_comp_spv_len, "main")); + RET(ff_vk_shader_register_exec(vkctx, &s->e, &s->grayscale)); + + /* Motion compensated interpolation shader. */ + ff_vk_shader_load(&s->interpolate, VK_SHADER_STAGE_COMPUTE_BIT, NULL, + (uint32_t []) { 32, 32, 1 }, 0); + ff_vk_shader_add_push_const(&s->interpolate, 0, sizeof(InterpolatePushData), + VK_SHADER_STAGE_COMPUTE_BIT); + { + const FFVulkanDescriptorSetBinding desc[] = { + { + .type = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, + .dimensions = 2, + .elems = planes, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + .samplers = DUP_SAMPLER(s->sampler), + }, + { + .type = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, + .dimensions = 2, + .elems = planes, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + .samplers = DUP_SAMPLER(s->sampler), + }, + { + .type = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, + .mem_quali = "writeonly", + .dimensions = 2, + .elems = planes, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + }, + { + .type = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, + .dimensions = 2, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + .samplers = DUP_SAMPLER(s->flow_sampler), + }, + { + .type = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, + .dimensions = 2, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + .samplers = DUP_SAMPLER(s->flow_sampler), + }, + }; + RET(ff_vk_shader_add_descriptor_set(vkctx, &s->interpolate, desc, 5, 0, 0)); + } + RET(ff_vk_shader_link(vkctx, &s->interpolate, + ff_fruc_interpolate_comp_spv_data, + ff_fruc_interpolate_comp_spv_len, "main")); + RET(ff_vk_shader_register_exec(vkctx, &s->e, &s->interpolate)); s->initialized = 1; +fail: + return err; +} + +static void of_image_barrier(VkImageMemoryBarrier2 *bar, VkImage img, + VkPipelineStageFlags2 src_stage, VkAccessFlags2 src_access, + VkPipelineStageFlags2 dst_stage, VkAccessFlags2 dst_access) +{ + *bar = (VkImageMemoryBarrier2) { + .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, + .srcStageMask = src_stage, + .srcAccessMask = src_access, + .dstStageMask = dst_stage, + .dstAccessMask = dst_access, + .oldLayout = VK_IMAGE_LAYOUT_GENERAL, + .newLayout = VK_IMAGE_LAYOUT_GENERAL, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = img, + .subresourceRange = { + .aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, + .levelCount = 1, + .layerCount = 1, + }, + }; +} + +/* Compute the forward and backward optical flow between f0 and f1. */ +static int compute_flow(AVFilterContext *avctx) +{ + int err; + FRUCVulkanContext *s = avctx->priv; + FFVulkanContext *vkctx = &s->vkctx; + FFVulkanFunctions *vk = &vkctx->vkfn; + FFVkExecContext *exec; + VkImageView f0_views[AV_NUM_DATA_POINTERS]; + VkImageView f1_views[AV_NUM_DATA_POINTERS]; + /* f0 and f1 each contribute one barrier per VkImage (a multi-image + * sw_format such as planar RGB or a separate alpha plane has several), + * plus the two single-image grayscale targets. */ + VkImageMemoryBarrier2 img_bar[2 * AV_NUM_DATA_POINTERS + 2]; + int nb_img_bar; + + /* --- Grayscale extraction on the compute queue. --- */ + exec = ff_vk_exec_get(vkctx, &s->e); + ff_vk_exec_start(vkctx, exec); + + RET(ff_vk_exec_add_dep_frame(vkctx, exec, s->f0, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + RET(ff_vk_exec_add_dep_frame(vkctx, exec, s->f1, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + /* Signal so the optical flow queue can see the grayscale writes. */ + RET(ff_vk_exec_add_dep_bool_sem(vkctx, exec, &s->sem_gray, 1, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, 0)); + RET(ff_vk_create_imageviews(vkctx, exec, f0_views, s->f0, FF_VK_REP_FLOAT)); + RET(ff_vk_create_imageviews(vkctx, exec, f1_views, s->f1, FF_VK_REP_FLOAT)); + + ff_vk_shader_update_img(vkctx, exec, &s->grayscale, 0, 0, 0, + f0_views[0], VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, + s->sampler); + ff_vk_shader_update_img(vkctx, exec, &s->grayscale, 0, 0, 1, + f1_views[0], VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, + s->sampler); + ff_vk_shader_update_img(vkctx, exec, &s->grayscale, 0, 1, 0, + s->gray_view[0], VK_IMAGE_LAYOUT_GENERAL, VK_NULL_HANDLE); + ff_vk_shader_update_img(vkctx, exec, &s->grayscale, 0, 1, 1, + s->gray_view[1], VK_IMAGE_LAYOUT_GENERAL, VK_NULL_HANDLE); + + ff_vk_exec_bind_shader(vkctx, exec, &s->grayscale); + ff_vk_shader_update_push_const(vkctx, exec, &s->grayscale, + VK_SHADER_STAGE_COMPUTE_BIT, 0, + sizeof(GrayscalePushData), &s->luma_weights); + + nb_img_bar = 0; + ff_vk_frame_barrier(vkctx, exec, s->f0, img_bar, &nb_img_bar, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, + VK_ACCESS_SHADER_READ_BIT, + VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, + VK_QUEUE_FAMILY_IGNORED); + ff_vk_frame_barrier(vkctx, exec, s->f1, img_bar, &nb_img_bar, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, + VK_ACCESS_SHADER_READ_BIT, + VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, + VK_QUEUE_FAMILY_IGNORED); + of_image_barrier(&img_bar[nb_img_bar++], s->gray_img[0], + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, 0, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_WRITE_BIT); + of_image_barrier(&img_bar[nb_img_bar++], s->gray_img[1], + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, 0, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_WRITE_BIT); + + vk->CmdPipelineBarrier2(exec->buf, &(VkDependencyInfo) { + .sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, + .pImageMemoryBarriers = img_bar, + .imageMemoryBarrierCount = nb_img_bar, + }); + + vk->CmdDispatch(exec->buf, + FFALIGN(s->width, s->grayscale.lg_size[0]) / s->grayscale.lg_size[0], + FFALIGN(s->height, s->grayscale.lg_size[1]) / s->grayscale.lg_size[1], + 1); + + RET(ff_vk_exec_submit(vkctx, exec)); + ff_vk_exec_wait(vkctx, exec); + + /* --- Optical flow execution on the optical flow queue. --- */ + exec = ff_vk_exec_get(vkctx, &s->e_of); + ff_vk_exec_start(vkctx, exec); + + /* Wait for the grayscale writes, signal once the flow has been written. */ + RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_gray, 0, + VK_PIPELINE_STAGE_2_OPTICAL_FLOW_BIT_NV)); + RET(ff_vk_exec_add_dep_bool_sem(vkctx, exec, &s->sem_flow, 1, + VK_PIPELINE_STAGE_2_OPTICAL_FLOW_BIT_NV, 0)); + + /* All optical flow images permanently reside in VK_IMAGE_LAYOUT_GENERAL and + * the cross-queue memory visibility is handled by the semaphores, so no + * image barriers are required on the optical flow queue. */ + vk->CmdOpticalFlowExecuteNV(exec->buf, s->session, + &(VkOpticalFlowExecuteInfoNV) { + .sType = VK_STRUCTURE_TYPE_OPTICAL_FLOW_EXECUTE_INFO_NV, + }); + + RET(ff_vk_exec_submit(vkctx, exec)); + ff_vk_exec_wait(vkctx, exec); + + s->flow_valid = 1; return 0; + +fail: + ff_vk_exec_discard_deps(vkctx, exec); + return err; +} + +/* Produce the motion compensated output frame at temporal position t. */ +static int interpolate_frame(AVFilterContext *avctx, AVFrame *out, float t) +{ + int err; + FRUCVulkanContext *s = avctx->priv; + FFVulkanContext *vkctx = &s->vkctx; + FFVulkanFunctions *vk = &vkctx->vkfn; + FFVkExecContext *exec; + VkImageView f0_views[AV_NUM_DATA_POINTERS]; + VkImageView f1_views[AV_NUM_DATA_POINTERS]; + VkImageView out_views[AV_NUM_DATA_POINTERS]; + /* out, f0 and f1 each contribute one barrier per VkImage (a multi-image + * sw_format such as planar RGB or a separate alpha plane has several), + * plus the two single-image flow fields. */ + VkImageMemoryBarrier2 img_bar[3 * AV_NUM_DATA_POINTERS + 2]; + int nb_img_bar; + int computed = 0; + InterpolatePushData pd = { + .t = t, + .luma_size = { s->width, s->height }, + .planes = av_pix_fmt_count_planes(vkctx->output_format), + }; + + if (!s->flow_valid) { + RET(compute_flow(avctx)); + computed = 1; + } + + exec = ff_vk_exec_get(vkctx, &s->e); + ff_vk_exec_start(vkctx, exec); + + /* Wait on the flow only when it was (re)computed for this submission, to + * consume the single signal of the binary semaphore exactly once. */ + if (computed) + RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_flow, 0, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + + RET(ff_vk_exec_add_dep_frame(vkctx, exec, out, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + RET(ff_vk_exec_add_dep_frame(vkctx, exec, s->f0, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + RET(ff_vk_exec_add_dep_frame(vkctx, exec, s->f1, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + + RET(ff_vk_create_imageviews(vkctx, exec, f0_views, s->f0, FF_VK_REP_FLOAT)); + RET(ff_vk_create_imageviews(vkctx, exec, f1_views, s->f1, FF_VK_REP_FLOAT)); + RET(ff_vk_create_imageviews(vkctx, exec, out_views, out, FF_VK_REP_FLOAT)); + + ff_vk_shader_update_img_array(vkctx, exec, &s->interpolate, s->f0, f0_views, + 0, 0, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, + s->sampler); + ff_vk_shader_update_img_array(vkctx, exec, &s->interpolate, s->f1, f1_views, + 0, 1, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, + s->sampler); + ff_vk_shader_update_img_array(vkctx, exec, &s->interpolate, out, out_views, + 0, 2, VK_IMAGE_LAYOUT_GENERAL, VK_NULL_HANDLE); + ff_vk_shader_update_img(vkctx, exec, &s->interpolate, 0, 3, 0, + s->flow_sint_view[0], VK_IMAGE_LAYOUT_GENERAL, s->flow_sampler); + ff_vk_shader_update_img(vkctx, exec, &s->interpolate, 0, 4, 0, + s->flow_sint_view[1], VK_IMAGE_LAYOUT_GENERAL, s->flow_sampler); + + ff_vk_exec_bind_shader(vkctx, exec, &s->interpolate); + ff_vk_shader_update_push_const(vkctx, exec, &s->interpolate, + VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof(pd), &pd); + + nb_img_bar = 0; + ff_vk_frame_barrier(vkctx, exec, out, img_bar, &nb_img_bar, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, + VK_ACCESS_SHADER_WRITE_BIT, + VK_IMAGE_LAYOUT_GENERAL, + VK_QUEUE_FAMILY_IGNORED); + ff_vk_frame_barrier(vkctx, exec, s->f0, img_bar, &nb_img_bar, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, + VK_ACCESS_SHADER_READ_BIT, + VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, + VK_QUEUE_FAMILY_IGNORED); + ff_vk_frame_barrier(vkctx, exec, s->f1, img_bar, &nb_img_bar, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, + VK_ACCESS_SHADER_READ_BIT, + VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, + VK_QUEUE_FAMILY_IGNORED); + of_image_barrier(&img_bar[nb_img_bar++], s->flow_img[0], + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, 0, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_READ_BIT); + of_image_barrier(&img_bar[nb_img_bar++], s->flow_img[1], + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, 0, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_READ_BIT); + + vk->CmdPipelineBarrier2(exec->buf, &(VkDependencyInfo) { + .sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, + .pImageMemoryBarriers = img_bar, + .imageMemoryBarrierCount = nb_img_bar, + }); + + vk->CmdDispatch(exec->buf, + FFALIGN(s->width, s->interpolate.lg_size[0]) / s->interpolate.lg_size[0], + FFALIGN(s->height, s->interpolate.lg_size[1]) / s->interpolate.lg_size[1], + 1); + + return ff_vk_exec_submit(vkctx, exec); + +fail: + ff_vk_exec_discard_deps(vkctx, exec); + return err; } static int process_work_frame(AVFilterContext *ctx) { FRUCVulkanContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; int64_t work_pts; int64_t interpolate8; + int ret; if (!s->f1) return 0; @@ -111,19 +874,37 @@ static int process_work_frame(AVFilterContext *ctx) if (!s->f0) { av_assert1(s->flush); s->work = av_frame_clone(s->f1); + if (!s->work) + return AVERROR(ENOMEM); } else { if (work_pts >= s->pts1 + s->delta && s->flush) return 0; - /* No motion compensation yet: emit the temporally nearest source - * frame. Genuine interpolation replaces this in a later change. */ interpolate8 = av_rescale(work_pts - s->pts0, 256, s->delta); - s->work = av_frame_clone(interpolate8 >= 128 ? s->f1 : s->f0); + if (interpolate8 >= 256) { + s->work = av_frame_clone(s->f1); + } else if (interpolate8 <= 0) { + s->work = av_frame_clone(s->f0); + } else { + float t = (float)(work_pts - s->pts0) / (float)s->delta; + s->work = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!s->work) + return AVERROR(ENOMEM); + ret = av_frame_copy_props(s->work, s->f0); + if (ret < 0) { + av_frame_free(&s->work); + return ret; + } + ret = interpolate_frame(ctx, s->work, t); + if (ret < 0) { + av_frame_free(&s->work); + return ret; + } + } + if (!s->work) + return AVERROR(ENOMEM); } - if (!s->work) - return AVERROR(ENOMEM); - s->work->pts = work_pts; s->n++; @@ -178,6 +959,7 @@ retry: s->f1 = inpicref; s->pts1 = pts; s->delta = s->pts1 - s->pts0; + s->flow_valid = 0; if (s->delta < 0) { av_log(ctx, AV_LOG_WARNING, "PTS discontinuity.\n"); @@ -304,6 +1086,43 @@ static av_cold void uninit(AVFilterContext *avctx) { FRUCVulkanContext *s = avctx->priv; FFVulkanContext *vkctx = &s->vkctx; + FFVulkanFunctions *vk = &vkctx->vkfn; + + if (s->initialized) { + if (s->sem_gray) + vk->DestroySemaphore(vkctx->hwctx->act_dev, s->sem_gray, vkctx->hwctx->alloc); + if (s->sem_flow) + vk->DestroySemaphore(vkctx->hwctx->act_dev, s->sem_flow, vkctx->hwctx->alloc); + if (s->session) + vk->DestroyOpticalFlowSessionNV(vkctx->hwctx->act_dev, s->session, + vkctx->hwctx->alloc); + for (int i = 0; i < 2; i++) { + if (s->gray_view[i]) + vk->DestroyImageView(vkctx->hwctx->act_dev, s->gray_view[i], vkctx->hwctx->alloc); + if (s->gray_img[i]) + vk->DestroyImage(vkctx->hwctx->act_dev, s->gray_img[i], vkctx->hwctx->alloc); + if (s->gray_mem[i]) + vk->FreeMemory(vkctx->hwctx->act_dev, s->gray_mem[i], vkctx->hwctx->alloc); + if (s->flow_view[i]) + vk->DestroyImageView(vkctx->hwctx->act_dev, s->flow_view[i], vkctx->hwctx->alloc); + if (s->flow_sint_view[i]) + vk->DestroyImageView(vkctx->hwctx->act_dev, s->flow_sint_view[i], vkctx->hwctx->alloc); + if (s->flow_img[i]) + vk->DestroyImage(vkctx->hwctx->act_dev, s->flow_img[i], vkctx->hwctx->alloc); + if (s->flow_mem[i]) + vk->FreeMemory(vkctx->hwctx->act_dev, s->flow_mem[i], vkctx->hwctx->alloc); + } + } + + ff_vk_exec_pool_free(vkctx, &s->e); + ff_vk_exec_pool_free(vkctx, &s->e_of); + ff_vk_shader_free(vkctx, &s->grayscale); + ff_vk_shader_free(vkctx, &s->interpolate); + + if (s->sampler) + vk->DestroySampler(vkctx->hwctx->act_dev, s->sampler, vkctx->hwctx->alloc); + if (s->flow_sampler) + vk->DestroySampler(vkctx->hwctx->act_dev, s->flow_sampler, vkctx->hwctx->alloc); ff_vk_uninit(vkctx); diff --git a/libavfilter/vulkan/Makefile b/libavfilter/vulkan/Makefile index 2cfe9cfa93..a6a7c18886 100644 --- a/libavfilter/vulkan/Makefile +++ b/libavfilter/vulkan/Makefile @@ -12,6 +12,8 @@ OBJS-$(CONFIG_SCALE_VULKAN_FILTER) += vulkan/debayer.comp.spv.o OBJS-$(CONFIG_SCDET_VULKAN_FILTER) += vulkan/scdet.comp.spv.o OBJS-$(CONFIG_OVERLAY_VULKAN_FILTER) += vulkan/overlay.comp.spv.o OBJS-$(CONFIG_FLIP_VULKAN_FILTER) += vulkan/flip.comp.spv.o +OBJS-$(CONFIG_FRUC_VULKAN_FILTER) += vulkan/fruc_grayscale.comp.spv.o \ + vulkan/fruc_interpolate.comp.spv.o OBJS-$(CONFIG_TRANSPOSE_VULKAN_FILTER) += vulkan/transpose.comp.spv.o OBJS-$(CONFIG_V360_VULKAN_FILTER) += vulkan/v360.comp.spv.o OBJS-$(CONFIG_INTERLACE_VULKAN_FILTER) += vulkan/interlace.comp.spv.o diff --git a/libavfilter/vulkan/fruc_grayscale.comp.glsl b/libavfilter/vulkan/fruc_grayscale.comp.glsl new file mode 100644 index 0000000000..971209b879 --- /dev/null +++ b/libavfilter/vulkan/fruc_grayscale.comp.glsl @@ -0,0 +1,54 @@ +/* + * Copyright (C) 2026 Philip Langdale <[email protected]> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#pragma shader_stage(compute) + +#extension GL_EXT_scalar_block_layout : require + +layout (local_size_x_id = 253, local_size_y_id = 254, local_size_z_id = 255) in; + +/* The two source frames, sampled with normalized coordinates. Only the first + * plane (luma for YUV inputs) is bound. */ +layout (set = 0, binding = 0) uniform sampler2D in_frames[2]; + +/* The single channel grayscale images consumed by the optical flow session. */ +layout (set = 0, binding = 1, r8) uniform writeonly image2D out_gray[2]; + +/* Luma weights: (1,0,0,0) selects plane 0 for YUV (already luma); RGB->Y + * coefficients for RGB. Only feeds the flow engine as a matching signal and is + * never output, so the host passes fixed BT.709 without consulting metadata. */ +layout (push_constant, scalar) uniform pushConstants { + vec4 luma_weights; +}; + +void main() +{ + ivec2 pos = ivec2(gl_GlobalInvocationID.xy); + + for (int i = 0; i < 2; i++) { + ivec2 size = imageSize(out_gray[i]); + if (any(greaterThanEqual(pos, size))) + continue; + + vec2 uv = (vec2(pos) + 0.5) / vec2(size); + float luma = dot(texture(in_frames[i], uv).xyz, luma_weights.xyz) + luma_weights.w; + imageStore(out_gray[i], pos, vec4(luma)); + } +} diff --git a/libavfilter/vulkan/fruc_interpolate.comp.glsl b/libavfilter/vulkan/fruc_interpolate.comp.glsl new file mode 100644 index 0000000000..f9c417c4fe --- /dev/null +++ b/libavfilter/vulkan/fruc_interpolate.comp.glsl @@ -0,0 +1,65 @@ +/* + * Copyright (C) 2026 Philip Langdale <[email protected]> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#pragma shader_stage(compute) + +#extension GL_EXT_shader_image_load_formatted : require +#extension GL_EXT_scalar_block_layout : require +#extension GL_EXT_nonuniform_qualifier : require + +layout (local_size_x_id = 253, local_size_y_id = 254, local_size_z_id = 255) in; + +layout (set = 0, binding = 0) uniform sampler2D f0_img[]; +layout (set = 0, binding = 1) uniform sampler2D f1_img[]; +layout (set = 0, binding = 2) uniform writeonly image2D out_img[]; + +/* Forward (f0 -> f1) and backward (f1 -> f0) flow fields. These are bound for + * descriptor-set compatibility with the C code but are not consulted yet: this + * initial implementation performs a plain temporal blend. Motion-compensated + * warping using these fields is introduced in a subsequent change. */ +layout (set = 0, binding = 3) uniform isampler2D flow_fwd; +layout (set = 0, binding = 4) uniform isampler2D flow_bwd; + +layout (push_constant, scalar) uniform pushConstants { + float t; /* interpolation position in [0, 1] between f0 and f1 */ + vec2 luma_size; /* dimensions the flow vectors are expressed against */ + int planes; +}; + +void main() +{ + ivec2 pos = ivec2(gl_GlobalInvocationID.xy); + + for (int i = 0; i < planes; i++) { + /* Inputs are sampled images, so query size with textureSize() (imageSize() + * is for storage images). Content is top-left aligned, so this is also the + * size to normalize coordinates against; the output may be padded larger. */ + ivec2 size = textureSize(f0_img[i], 0); + if (any(greaterThanEqual(pos, size))) + continue; + + vec2 base = (vec2(pos) + 0.5) / vec2(size); + + vec4 c0 = texture(f0_img[i], base); + vec4 c1 = texture(f1_img[i], base); + + imageStore(out_img[i], pos, mix(c0, c1, t)); + } +} -- 2.52.0 From 7ee7bfd2084ace36944ba158f8808655c3b9ceb1 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sun, 21 Jun 2026 10:20:36 -0700 Subject: [PATCH 08/14] avfilter/fruc_vulkan: expose optical flow perf level and grid size Add the two options that tune the NV optical flow session: * perf: performance level (slow/medium/fast), previously hard-coded to slow. Trades motion-estimation quality for speed. * grid: output grid size in pixels (auto/1/2/4/8), previously always the finest grid the device supported. A coarser grid computes fewer flow vectors, so it is faster and uses less memory at the cost of detail. These settings are important as you can't handle high resolution input in realtime at the highest quality settings. For example, 2160p24 movie content needs reduced settings (eg: pref=medium,grid=2) for the filter to run comfortably. --- doc/filters.texi | 19 +++++++++ libavfilter/vf_fruc_vulkan.c | 79 ++++++++++++++++++++++++++++-------- 2 files changed, 81 insertions(+), 17 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index 7fa22de494..bc156eff1b 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -29406,6 +29406,25 @@ output frame rate. The frame timing logic mirrors the @code{framerate} filter. A string describing the desired output frame rate, evaluated as an expression. The following constants are available: @code{source_fps}. The default value is @code{60}. + +@item perf +Optical flow performance level, trading motion-estimation quality for speed. +This is the dominant cost at high resolutions. Possible values are: +@table @samp +@item slow +Highest quality, slowest. This is the default. +@item medium +Balanced quality and speed. +@item fast +Lowest quality, fastest. +@end table + +@item grid +Optical flow output grid size in pixels. A coarser grid computes fewer flow +vectors, making it faster and less memory hungry at the cost of flow detail. +Possible values are @code{auto} (the default; selects the finest grid the +device supports, usually 1x1), @code{1}, @code{2}, @code{4} and @code{8}. Only +grid sizes the device advertises are accepted. @end table @section gblur_vulkan diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c index b0b0018ec4..fe10a45ee8 100644 --- a/libavfilter/vf_fruc_vulkan.c +++ b/libavfilter/vf_fruc_vulkan.c @@ -96,6 +96,10 @@ typedef struct FRUCVulkanContext { VkFormat input_format; ///< grayscale input format VkFormat flow_format; ///< flow vector format + /* Tuning options. */ + int perf_level; ///< VkOpticalFlowPerformanceLevelNV + int opt_grid_size; ///< requested grid in pixels (0 = finest) + int width; ///< luma width int height; ///< luma height int flow_width; @@ -138,6 +142,24 @@ typedef struct FRUCVulkanContext { static const AVOption fruc_vulkan_options[] = { { "fps", "A string describing the desired output frame rate", OFFSET(requested_frame_rate), AV_OPT_TYPE_STRING, { .str = "60" }, 0, 0, FLAGS }, + { "perf", "Optical flow performance level (quality versus speed)", + OFFSET(perf_level), AV_OPT_TYPE_INT, { .i64 = VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_SLOW_NV }, + VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_SLOW_NV, VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_FAST_NV, + FLAGS, .unit = "perf" }, + { "slow", "Highest quality, slowest", 0, AV_OPT_TYPE_CONST, + { .i64 = VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_SLOW_NV }, 0, 0, FLAGS, .unit = "perf" }, + { "medium", "Balanced quality and speed", 0, AV_OPT_TYPE_CONST, + { .i64 = VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_MEDIUM_NV }, 0, 0, FLAGS, .unit = "perf" }, + { "fast", "Lowest quality, fastest", 0, AV_OPT_TYPE_CONST, + { .i64 = VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_FAST_NV }, 0, 0, FLAGS, .unit = "perf" }, + { "grid", "Optical flow output grid size in pixels (coarser is faster)", + OFFSET(opt_grid_size), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 8, FLAGS, .unit = "grid" }, + { "auto", "Finest grid the device supports", 0, AV_OPT_TYPE_CONST, + { .i64 = 0 }, 0, 0, FLAGS, .unit = "grid" }, + { "1", "1x1", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, .unit = "grid" }, + { "2", "2x2", 0, AV_OPT_TYPE_CONST, { .i64 = 2 }, 0, 0, FLAGS, .unit = "grid" }, + { "4", "4x4", 0, AV_OPT_TYPE_CONST, { .i64 = 4 }, 0, 0, FLAGS, .unit = "grid" }, + { "8", "8x8", 0, AV_OPT_TYPE_CONST, { .i64 = 8 }, 0, 0, FLAGS, .unit = "grid" }, { NULL } }; @@ -419,29 +441,52 @@ static av_cold int init_filter(AVFilterContext *avctx) return AVERROR(ENOTSUP); } + static const struct { + int size; + VkOpticalFlowGridSizeFlagsNV bit; + } grid_map[] = { + { 1, VK_OPTICAL_FLOW_GRID_SIZE_1X1_BIT_NV }, + { 2, VK_OPTICAL_FLOW_GRID_SIZE_2X2_BIT_NV }, + { 4, VK_OPTICAL_FLOW_GRID_SIZE_4X4_BIT_NV }, + { 8, VK_OPTICAL_FLOW_GRID_SIZE_8X8_BIT_NV }, + }; + grids = vkctx->optical_flow_props.supportedOutputGridSizes; - if (grids & VK_OPTICAL_FLOW_GRID_SIZE_1X1_BIT_NV) { - s->grid_size = 1; - s->grid_bit = VK_OPTICAL_FLOW_GRID_SIZE_1X1_BIT_NV; - } else if (grids & VK_OPTICAL_FLOW_GRID_SIZE_2X2_BIT_NV) { - s->grid_size = 2; - s->grid_bit = VK_OPTICAL_FLOW_GRID_SIZE_2X2_BIT_NV; - } else if (grids & VK_OPTICAL_FLOW_GRID_SIZE_4X4_BIT_NV) { - s->grid_size = 4; - s->grid_bit = VK_OPTICAL_FLOW_GRID_SIZE_4X4_BIT_NV; - } else if (grids & VK_OPTICAL_FLOW_GRID_SIZE_8X8_BIT_NV) { - s->grid_size = 8; - s->grid_bit = VK_OPTICAL_FLOW_GRID_SIZE_8X8_BIT_NV; + if (s->opt_grid_size) { + /* Honour an explicit grid request, erroring if the device lacks it. */ + VkOpticalFlowGridSizeFlagsNV want = 0; + for (int i = 0; i < FF_ARRAY_ELEMS(grid_map); i++) + if (grid_map[i].size == s->opt_grid_size) + want = grid_map[i].bit; + if (!want || !(grids & want)) { + av_log(avctx, AV_LOG_ERROR, "Requested optical flow grid size %d is not " + "supported by the device (supported mask 0x%x)\n", + s->opt_grid_size, grids); + return AVERROR(ENOTSUP); + } + s->grid_size = s->opt_grid_size; + s->grid_bit = want; } else { - av_log(avctx, AV_LOG_ERROR, "No supported optical flow output grid size\n"); - return AVERROR(ENOTSUP); + /* Auto: pick the finest (smallest) grid the device supports. */ + s->grid_size = 0; + for (int i = 0; i < FF_ARRAY_ELEMS(grid_map); i++) { + if (grids & grid_map[i].bit) { + s->grid_size = grid_map[i].size; + s->grid_bit = grid_map[i].bit; + break; + } + } + if (!s->grid_size) { + av_log(avctx, AV_LOG_ERROR, "No supported optical flow output grid size\n"); + return AVERROR(ENOTSUP); + } } s->flow_width = (s->width + s->grid_size - 1) / s->grid_size; s->flow_height = (s->height + s->grid_size - 1) / s->grid_size; - av_log(avctx, AV_LOG_INFO, "optical flow: grid %d, flow %dx%d, bidir=%d, " - "min %dx%d max %dx%d\n", s->grid_size, s->flow_width, s->flow_height, + av_log(avctx, AV_LOG_INFO, "optical flow: perf %d, grid %d, flow %dx%d, bidir=%d, " + "min %dx%d max %dx%d\n", s->perf_level, s->grid_size, s->flow_width, s->flow_height, vkctx->optical_flow_props.bidirectionalFlowSupported, vkctx->optical_flow_props.minWidth, vkctx->optical_flow_props.minHeight, vkctx->optical_flow_props.maxWidth, vkctx->optical_flow_props.maxHeight); @@ -500,7 +545,7 @@ static av_cold int init_filter(AVFilterContext *avctx) .imageFormat = s->input_format, .flowVectorFormat = s->flow_format, .outputGridSize = s->grid_bit, - .performanceLevel = VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_SLOW_NV, + .performanceLevel = s->perf_level, .flags = VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NV, }, vkctx->hwctx->alloc, &s->session); if (ret != VK_SUCCESS) { -- 2.52.0 From e7f07b6d771a0b99419a3067c419410f75b26ad9 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Mon, 22 Jun 2026 09:24:15 -0700 Subject: [PATCH 09/14] avfilter/fruc_vulkan: copy pass-through frames into the output frames context Passing through unchanged input frames creates problems for subsequent filters when using vulkan video decode. These frames remain associated with the decoder's frame context. This limits the possible usages for the frames, as well as extending the lifetime of frames that are part of the DPB, which can cause decoder starvation if the filtering pipeline is deep enough. Instead, we need to do deep copies of these frames so that they are fully independent of the decoder and can be consumed correctly by subsequent filters. This requires exposing vkCmdCopyImage in the Vulkan function loader. --- libavfilter/vf_fruc_vulkan.c | 115 ++++++++++++++++++++++++++++++++--- libavutil/vulkan_functions.h | 1 + 2 files changed, 109 insertions(+), 7 deletions(-) diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c index fe10a45ee8..f270b42fdf 100644 --- a/libavfilter/vf_fruc_vulkan.c +++ b/libavfilter/vf_fruc_vulkan.c @@ -897,6 +897,105 @@ fail: return err; } +/* Copy a source frame into a freshly allocated output frame on the GPU. Used for + * pass-through (boundary / flush) outputs instead of cloning the input: a clone + * keeps the input's frames context, which can differ from the filter's output + * context that the downstream link is configured for. */ +static int copy_frame(AVFilterContext *avctx, AVFrame *out, AVFrame *src) +{ + int err; + FRUCVulkanContext *s = avctx->priv; + FFVulkanContext *vkctx = &s->vkctx; + FFVulkanFunctions *vk = &vkctx->vkfn; + FFVkExecContext *exec; + AVVkFrame *src_vk = (AVVkFrame *)src->data[0]; + AVVkFrame *out_vk = (AVVkFrame *)out->data[0]; + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(vkctx->output_format); + const int nb_planes = av_pix_fmt_count_planes(vkctx->output_format); + const int src_nb_images = ff_vk_count_images(src_vk); + const int out_nb_images = ff_vk_count_images(out_vk); + /* src and out each contribute one barrier per VkImage; a multi-image + * sw_format such as planar RGB or a separate alpha plane has several. */ + VkImageMemoryBarrier2 img_bar[2 * AV_NUM_DATA_POINTERS]; + int nb_img_bar = 0; + + exec = ff_vk_exec_get(vkctx, &s->e); + ff_vk_exec_start(vkctx, exec); + + RET(ff_vk_exec_add_dep_frame(vkctx, exec, src, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT)); + RET(ff_vk_exec_add_dep_frame(vkctx, exec, out, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT)); + + ff_vk_frame_barrier(vkctx, exec, src, img_bar, &nb_img_bar, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COPY_BIT, + VK_ACCESS_2_TRANSFER_READ_BIT, + VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, + VK_QUEUE_FAMILY_IGNORED); + ff_vk_frame_barrier(vkctx, exec, out, img_bar, &nb_img_bar, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_COPY_BIT, + VK_ACCESS_2_TRANSFER_WRITE_BIT, + VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, + VK_QUEUE_FAMILY_IGNORED); + + vk->CmdPipelineBarrier2(exec->buf, &(VkDependencyInfo) { + .sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, + .pImageMemoryBarriers = img_bar, + .imageMemoryBarrierCount = nb_img_bar, + }); + + for (int i = 0; i < nb_planes; i++) { + /* Plane texel extent: plane 0 (and non-planar / alpha) is full size, + * chroma planes are subsampled. Mirrors hwcontext_vulkan's get_plane_wh. */ + int sub = i && i != 3 && (desc->flags & AV_PIX_FMT_FLAG_PLANAR) && + !(desc->flags & AV_PIX_FMT_FLAG_RGB); + uint32_t w = sub ? AV_CEIL_RSHIFT(s->width, desc->log2_chroma_w) : s->width; + uint32_t h = sub ? AV_CEIL_RSHIFT(s->height, desc->log2_chroma_h) : s->height; + VkImageCopy region = { + .srcSubresource = { .aspectMask = ff_vk_aspect_flag(src, i), .layerCount = 1 }, + .dstSubresource = { .aspectMask = ff_vk_aspect_flag(out, i), .layerCount = 1 }, + .extent = { w, h, 1 }, + }; + vk->CmdCopyImage(exec->buf, + src_vk->img[FFMIN(i, src_nb_images - 1)], + VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL, + out_vk->img[FFMIN(i, out_nb_images - 1)], + VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, + 1, ®ion); + } + + return ff_vk_exec_submit(vkctx, exec); + +fail: + ff_vk_exec_discard_deps(vkctx, exec); + return err; +} + +/* Allocate an output-context frame and fill it with a copy of src. */ +static int passthrough_frame(AVFilterContext *ctx, AVFrame **work, AVFrame *src) +{ + AVFilterLink *outlink = ctx->outputs[0]; + int ret; + + *work = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!*work) + return AVERROR(ENOMEM); + ret = av_frame_copy_props(*work, src); + if (ret < 0) + goto fail; + ret = copy_frame(ctx, *work, src); + if (ret < 0) + goto fail; + return 0; +fail: + av_frame_free(work); + return ret; +} + static int process_work_frame(AVFilterContext *ctx) { FRUCVulkanContext *s = ctx->priv; @@ -918,18 +1017,22 @@ static int process_work_frame(AVFilterContext *ctx) if (!s->f0) { av_assert1(s->flush); - s->work = av_frame_clone(s->f1); - if (!s->work) - return AVERROR(ENOMEM); + ret = passthrough_frame(ctx, &s->work, s->f1); + if (ret < 0) + return ret; } else { if (work_pts >= s->pts1 + s->delta && s->flush) return 0; interpolate8 = av_rescale(work_pts - s->pts0, 256, s->delta); if (interpolate8 >= 256) { - s->work = av_frame_clone(s->f1); + ret = passthrough_frame(ctx, &s->work, s->f1); + if (ret < 0) + return ret; } else if (interpolate8 <= 0) { - s->work = av_frame_clone(s->f0); + ret = passthrough_frame(ctx, &s->work, s->f0); + if (ret < 0) + return ret; } else { float t = (float)(work_pts - s->pts0) / (float)s->delta; s->work = ff_get_video_buffer(outlink, outlink->w, outlink->h); @@ -946,8 +1049,6 @@ static int process_work_frame(AVFilterContext *ctx) return ret; } } - if (!s->work) - return AVERROR(ENOMEM); } s->work->pts = work_pts; diff --git a/libavutil/vulkan_functions.h b/libavutil/vulkan_functions.h index 86dff046b2..29205d3154 100644 --- a/libavutil/vulkan_functions.h +++ b/libavutil/vulkan_functions.h @@ -157,6 +157,7 @@ typedef uint64_t FFVulkanExtensions; MACRO(1, 1, FF_VK_EXT_NO_FLAG, CmdPipelineBarrier) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, CmdCopyBufferToImage) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, CmdCopyImageToBuffer) \ + MACRO(1, 1, FF_VK_EXT_NO_FLAG, CmdCopyImage) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, CmdClearColorImage) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, CmdCopyBuffer) \ MACRO(1, 1, FF_VK_EXT_NO_FLAG, CmdUpdateBuffer) \ -- 2.52.0 From 20c1bd65ed59a1379d1b32ab734bb58fe6f44e95 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Wed, 24 Jun 2026 07:59:46 -0700 Subject: [PATCH 10/14] avfilter/fruc_vulkan: warp frames with the computed optical flow Replace the placeholder temporal blend in the interpolation shader with a motion-compensated algorithm. This is finally the point where we are actually using the optical flow information. For each output pixel the source position on the f0 and f1 grids is recovered by fixed-point Picard iteration before the samples are blended. The number of iterations and the relaxation factor are established through empirical testing on a couple of ad-hoc samples, representing highly uniform and non-uniform motion respectively. The approach here is hardly profound, and not meant to be. There are for sure many more advanced algorithms for doing motion interpolation from optical flow, and we can happily replace it with, or add, something more advanced once all the machinery is in place. --- libavfilter/vulkan/fruc_interpolate.comp.glsl | 34 +++++++++++++++---- 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/libavfilter/vulkan/fruc_interpolate.comp.glsl b/libavfilter/vulkan/fruc_interpolate.comp.glsl index f9c417c4fe..fbe7b13d17 100644 --- a/libavfilter/vulkan/fruc_interpolate.comp.glsl +++ b/libavfilter/vulkan/fruc_interpolate.comp.glsl @@ -30,19 +30,32 @@ layout (set = 0, binding = 0) uniform sampler2D f0_img[]; layout (set = 0, binding = 1) uniform sampler2D f1_img[]; layout (set = 0, binding = 2) uniform writeonly image2D out_img[]; -/* Forward (f0 -> f1) and backward (f1 -> f0) flow fields. These are bound for - * descriptor-set compatibility with the C code but are not consulted yet: this - * initial implementation performs a plain temporal blend. Motion-compensated - * warping using these fields is introduced in a subsequent change. */ +/* Forward (f0 -> f1) and backward (f1 -> f0) flow fields. The NV optical flow + * engine writes signed fixed point (S10.5) vectors, sampled here as raw + * integers; dividing by 32 (2^5) yields correct values. */ layout (set = 0, binding = 3) uniform isampler2D flow_fwd; layout (set = 0, binding = 4) uniform isampler2D flow_bwd; +#define FLOW_FIXED_POINT_SCALE (1.0 / 32.0) + +/* These Picard values were established empirically on a couple of different + * samples, but one could easily imagine reaching a different conclusion from + * different data. */ +#define PICARD_ITERS 6 +#define PICARD_OMEGA 0.5 + layout (push_constant, scalar) uniform pushConstants { float t; /* interpolation position in [0, 1] between f0 and f1 */ vec2 luma_size; /* dimensions the flow vectors are expressed against */ int planes; }; +/* Normalized forward/backward flow at a normalized position. The flow vectors + * are relative to the original Luma, so must be converted to a resolution independent + * displacement that can be applied to other planes with different dimensions. */ +vec2 flow_at_fwd(vec2 p) { return vec2(texture(flow_fwd, p).xy) * FLOW_FIXED_POINT_SCALE / luma_size; } +vec2 flow_at_bwd(vec2 p) { return vec2(texture(flow_bwd, p).xy) * FLOW_FIXED_POINT_SCALE / luma_size; } + void main() { ivec2 pos = ivec2(gl_GlobalInvocationID.xy); @@ -57,8 +70,17 @@ void main() vec2 base = (vec2(pos) + 0.5) / vec2(size); - vec4 c0 = texture(f0_img[i], base); - vec4 c1 = texture(f1_img[i], base); + /* The flow is anchored on the f0/f1 grids, not the intermediate frame. + * Recover the source position on each grid by Picard iteration. */ + vec2 s0 = base; + vec2 s1 = base; + for (int k = 0; k < PICARD_ITERS; k++) { + s0 = mix(s0, base - t * flow_at_fwd(s0), PICARD_OMEGA); + s1 = mix(s1, base - (1.0 - t) * flow_at_bwd(s1), PICARD_OMEGA); + } + + vec4 c0 = texture(f0_img[i], s0); + vec4 c1 = texture(f1_img[i], s1); imageStore(out_img[i], pos, mix(c0, c1, t)); } -- 2.52.0 From 319a53fce9d970bf8c759854e101629da8ffaa90 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sun, 28 Jun 2026 13:03:15 -0700 Subject: [PATCH 11/14] avutil/vulkan: add ff_vk_exec_add_dep_signal_sem Add a helper to register a timeline-semaphore signal with an explicit value on an execution context, mirroring ff_vk_exec_add_dep_wait_sem() on the wait side. The fruc_vulkan filter will use this to chain timeline semaphores across its pipelined cross-queue submissions, where an interpolation must signal a specific (generation-keyed) value rather than the implicit binary signal. --- libavutil/vulkan.c | 17 +++++++++++++++++ libavutil/vulkan.h | 3 +++ 2 files changed, 20 insertions(+) diff --git a/libavutil/vulkan.c b/libavutil/vulkan.c index 07580dda3b..bee4d05544 100644 --- a/libavutil/vulkan.c +++ b/libavutil/vulkan.c @@ -731,6 +731,23 @@ int ff_vk_exec_add_dep_wait_sem(FFVulkanContext *s, FFVkExecContext *e, return 0; } +int ff_vk_exec_add_dep_signal_sem(FFVulkanContext *s, FFVkExecContext *e, + VkSemaphore sem, uint64_t val, + VkPipelineStageFlagBits2 stage) +{ + VkSemaphoreSubmitInfo *sem_sig; + ARR_REALLOC(e, sem_sig, &e->sem_sig_alloc, e->sem_sig_cnt); + + e->sem_sig[e->sem_sig_cnt++] = (VkSemaphoreSubmitInfo) { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_SUBMIT_INFO, + .semaphore = sem, + .value = val, + .stageMask = stage, + }; + + return 0; +} + int ff_vk_exec_add_dep_bool_sem(FFVulkanContext *s, FFVkExecContext *e, VkSemaphore *sem, int nb, VkPipelineStageFlagBits2 stage, diff --git a/libavutil/vulkan.h b/libavutil/vulkan.h index 1a2fcc74f1..e78c959d52 100644 --- a/libavutil/vulkan.h +++ b/libavutil/vulkan.h @@ -518,6 +518,9 @@ int ff_vk_exec_add_dep_buf(FFVulkanContext *s, FFVkExecContext *e, int ff_vk_exec_add_dep_wait_sem(FFVulkanContext *s, FFVkExecContext *e, VkSemaphore sem, uint64_t val, VkPipelineStageFlagBits2 stage); +int ff_vk_exec_add_dep_signal_sem(FFVulkanContext *s, FFVkExecContext *e, + VkSemaphore sem, uint64_t val, + VkPipelineStageFlagBits2 stage); int ff_vk_exec_add_dep_bool_sem(FFVulkanContext *s, FFVkExecContext *e, VkSemaphore *sem, int nb, VkPipelineStageFlagBits2 stage, -- 2.52.0 From 5b5e60b483dc06789a027de0e4ec17976633a9bd Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sun, 28 Jun 2026 13:03:33 -0700 Subject: [PATCH 12/14] avfilter/fruc_vulkan: pipeline the cross-queue submissions The grayscale, optical flow and interpolation stages are currently serialised with host fence waits (ff_vk_exec_wait) after each of the the grayscale and optical flow submissions. We can remove these waits and manage everything with timeline semaphores. In particular, we must correctly handle the case where more than one frame is being interpolated per optical flow pair. In these situations, we have multiple interpolations waiting on the optical flow generation, and then we have the next optical flow generation waiting on all the previous interpolations. This requires the ff_vk_exec_add_dep_signal_sem() we added in the previous commit. We also introduce a second context to the optical flow exec pool so that the next command buffer can be recorded without a host-wait for the previous one to retire - but of course, optical flow is serial so the queued one won't actually execute until the previous one completes. Finally, we free the execution pools at the start of uninit, before destroying the semaphores and images they reference: ff_vk_exec_pool_free waits out every still-submitted command buffer, and those pooled submissions wait on and signal the timeline semaphores, so destroying an in-use semaphore first would leave a fence permanently unsignaled and deadlock teardown. --- libavfilter/vf_fruc_vulkan.c | 107 ++++++++++++++++++++++++++--------- 1 file changed, 81 insertions(+), 26 deletions(-) diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c index f270b42fdf..db7e563e4b 100644 --- a/libavfilter/vf_fruc_vulkan.c +++ b/libavfilter/vf_fruc_vulkan.c @@ -83,11 +83,21 @@ typedef struct FRUCVulkanContext { VkSampler sampler; ///< linear sampler for the video planes VkSampler flow_sampler; ///< nearest sampler for the flow vectors - /* Binary semaphores used to order the cross-queue submissions and, more - * importantly, to make each stage's writes visible to the next: the host - * fence waits alone do not provide cross-queue memory visibility. */ + /* Timeline semaphores order the pipelined cross-queue submissions and make + * each stage's writes visible to the next. Timeline is required for two + * reasons: + * * Each optical flow source pair may yield multiple interpolations, and + * each interpolation should wait on the flow. Multi-wait requires timeline + * semaphores + * * We don't know how many interpolations will be done from a single optical + * flow pair ahead of time, so we cannot simply signal completion after the + * last one. Instead we increment a timeline semaphore after each one and + * then the next flow calculation waits on the final timeline value. */ VkSemaphore sem_gray; ///< grayscale (compute) -> optical flow VkSemaphore sem_flow; ///< optical flow -> interpolation (compute) + VkSemaphore sem_interp; ///< interpolation reads -> next pair optical flow + uint64_t gen; ///< source pair generation (sem_gray/sem_flow value) + uint64_t interp_value; ///< monotonic interpolation counter (sem_interp value) /* Optical flow session and associated images. */ VkOpticalFlowSessionNV session; @@ -404,7 +414,11 @@ static av_cold int init_filter(AVFilterContext *avctx) s->share_qfs[s->nb_share_qfs++] = s->qf_of->idx; RET(ff_vk_exec_pool_init(vkctx, s->qf, &s->e, s->qf->num, 0, 0, 0, NULL)); - RET(ff_vk_exec_pool_init(vkctx, s->qf_of, &s->e_of, 1, 0, 0, 0, NULL)); + /* Use more than one optical flow context so that the optical flow execution + * for the next frame pair can be recorded and submitted without first + * host-waiting the previous pair's execution to retire its command buffer. */ + RET(ff_vk_exec_pool_init(vkctx, s->qf_of, &s->e_of, + FFMIN(s->qf_of->num, 2), 0, 0, 0, NULL)); RET(ff_vk_init_sampler(vkctx, &s->sampler, 0, VK_FILTER_LINEAR)); /* Flow is sampled through an integer view and its format has no linear * filtering support, so use nearest. Requires normalised coords and as we @@ -412,13 +426,21 @@ static av_cold int init_filter(AVFilterContext *avctx) RET(ff_vk_init_sampler(vkctx, &s->flow_sampler, 0, VK_FILTER_NEAREST)); { + VkSemaphoreTypeCreateInfo sem_type_info = { + .sType = VK_STRUCTURE_TYPE_SEMAPHORE_TYPE_CREATE_INFO, + .semaphoreType = VK_SEMAPHORE_TYPE_TIMELINE, + .initialValue = 0, + }; VkSemaphoreCreateInfo sem_info = { .sType = VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO, + .pNext = &sem_type_info, }; if (vk->CreateSemaphore(vkctx->hwctx->act_dev, &sem_info, vkctx->hwctx->alloc, &s->sem_gray) != VK_SUCCESS || vk->CreateSemaphore(vkctx->hwctx->act_dev, &sem_info, - vkctx->hwctx->alloc, &s->sem_flow) != VK_SUCCESS) { + vkctx->hwctx->alloc, &s->sem_flow) != VK_SUCCESS || + vk->CreateSemaphore(vkctx->hwctx->act_dev, &sem_info, + vkctx->hwctx->alloc, &s->sem_interp) != VK_SUCCESS) { av_log(avctx, AV_LOG_ERROR, "Failed to create synchronization semaphores\n"); return AVERROR_EXTERNAL; } @@ -690,6 +712,9 @@ static int compute_flow(AVFilterContext *avctx) VkImageMemoryBarrier2 img_bar[2 * AV_NUM_DATA_POINTERS + 2]; int nb_img_bar; + /* This pair's generation; sem_gray and sem_flow are signalled with it. */ + s->gen++; + /* --- Grayscale extraction on the compute queue. --- */ exec = ff_vk_exec_get(vkctx, &s->e); ff_vk_exec_start(vkctx, exec); @@ -700,9 +725,14 @@ static int compute_flow(AVFilterContext *avctx) RET(ff_vk_exec_add_dep_frame(vkctx, exec, s->f1, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); - /* Signal so the optical flow queue can see the grayscale writes. */ - RET(ff_vk_exec_add_dep_bool_sem(vkctx, exec, &s->sem_gray, 1, - VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, 0)); + /* The grayscale images are a single instance reused every pair, and the + * previous pair's optical flow reads them. Wait for that read to retire + * (sem_flow at the previous generation) before overwriting them. */ + if (s->gen > 1) + RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_flow, s->gen - 1, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + RET(ff_vk_exec_add_dep_signal_sem(vkctx, exec, s->sem_gray, s->gen, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); RET(ff_vk_create_imageviews(vkctx, exec, f0_views, s->f0, FF_VK_REP_FLOAT)); RET(ff_vk_create_imageviews(vkctx, exec, f1_views, s->f1, FF_VK_REP_FLOAT)); @@ -753,29 +783,37 @@ static int compute_flow(AVFilterContext *avctx) FFALIGN(s->height, s->grayscale.lg_size[1]) / s->grayscale.lg_size[1], 1); + /* sem_gray orders the optical flow submission after this one + * and makes the grayscale writes visible across the queue boundary. */ RET(ff_vk_exec_submit(vkctx, exec)); - ff_vk_exec_wait(vkctx, exec); /* --- Optical flow execution on the optical flow queue. --- */ exec = ff_vk_exec_get(vkctx, &s->e_of); ff_vk_exec_start(vkctx, exec); /* Wait for the grayscale writes, signal once the flow has been written. */ - RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_gray, 0, + RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_gray, s->gen, VK_PIPELINE_STAGE_2_OPTICAL_FLOW_BIT_NV)); - RET(ff_vk_exec_add_dep_bool_sem(vkctx, exec, &s->sem_flow, 1, - VK_PIPELINE_STAGE_2_OPTICAL_FLOW_BIT_NV, 0)); + /* The flow images are a single instance reused every pair. The previous + * pair's interpolations sampled them; wait for the last such read to retire + * (the highest interpolation value signalled so far, which belongs to the + * previous pair since this pair has produced none yet) before overwriting + * them. A value of 0 is the initial state and is satisfied immediately. */ + RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_interp, s->interp_value, + VK_PIPELINE_STAGE_2_OPTICAL_FLOW_BIT_NV)); + RET(ff_vk_exec_add_dep_signal_sem(vkctx, exec, s->sem_flow, s->gen, + VK_PIPELINE_STAGE_2_OPTICAL_FLOW_BIT_NV)); - /* All optical flow images permanently reside in VK_IMAGE_LAYOUT_GENERAL and - * the cross-queue memory visibility is handled by the semaphores, so no - * image barriers are required on the optical flow queue. */ + /* The flow images stay in VK_IMAGE_LAYOUT_GENERAL and the semaphores handle + * cross-queue visibility, so no image barriers are needed here. */ vk->CmdOpticalFlowExecuteNV(exec->buf, s->session, &(VkOpticalFlowExecuteInfoNV) { .sType = VK_STRUCTURE_TYPE_OPTICAL_FLOW_EXECUTE_INFO_NV, }); + /* sem_flow orders the interpolation after the flow execution + * and makes the flow writes visible on the compute queue. */ RET(ff_vk_exec_submit(vkctx, exec)); - ff_vk_exec_wait(vkctx, exec); s->flow_valid = 1; return 0; @@ -801,26 +839,34 @@ static int interpolate_frame(AVFilterContext *avctx, AVFrame *out, float t) * plus the two single-image flow fields. */ VkImageMemoryBarrier2 img_bar[3 * AV_NUM_DATA_POINTERS + 2]; int nb_img_bar; - int computed = 0; InterpolatePushData pd = { .t = t, .luma_size = { s->width, s->height }, .planes = av_pix_fmt_count_planes(vkctx->output_format), }; + /* Not via RET: the fail label discards deps on exec, which is not yet + * acquired here, and compute_flow cleans up its own exec on failure. */ if (!s->flow_valid) { - RET(compute_flow(avctx)); - computed = 1; + err = compute_flow(avctx); + if (err < 0) + return err; } exec = ff_vk_exec_get(vkctx, &s->e); ff_vk_exec_start(vkctx, exec); - /* Wait on the flow only when it was (re)computed for this submission, to - * consume the single signal of the binary semaphore exactly once. */ - if (computed) - RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_flow, 0, - VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + /* Every interpolation of this pair waits on the same flow result, keyed by + * the pair generation. */ + RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_flow, s->gen, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + /* Chain sem_interp: wait the previous value, signal the next. Keeps the + * signals monotonic and lets the next pair's optical flow fence on the final + * value before overwriting the flow images. */ + RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_interp, s->interp_value, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + RET(ff_vk_exec_add_dep_signal_sem(vkctx, exec, s->sem_interp, ++s->interp_value, + VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); RET(ff_vk_exec_add_dep_frame(vkctx, exec, out, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, @@ -1234,11 +1280,22 @@ static av_cold void uninit(AVFilterContext *avctx) FFVulkanContext *vkctx = &s->vkctx; FFVulkanFunctions *vk = &vkctx->vkfn; + /* Free the execution pools first: this waits for every submitted command + * buffer to retire (ff_vk_exec_pool_free fences each context). The pooled + * submissions wait on and signal the timeline semaphores and reference the + * optical flow images below, so those objects must outlive the wait — destroying + * an in-use semaphore would leave a submission's fence permanently unsignaled + * and deadlock the wait. */ + ff_vk_exec_pool_free(vkctx, &s->e); + ff_vk_exec_pool_free(vkctx, &s->e_of); + if (s->initialized) { if (s->sem_gray) vk->DestroySemaphore(vkctx->hwctx->act_dev, s->sem_gray, vkctx->hwctx->alloc); if (s->sem_flow) vk->DestroySemaphore(vkctx->hwctx->act_dev, s->sem_flow, vkctx->hwctx->alloc); + if (s->sem_interp) + vk->DestroySemaphore(vkctx->hwctx->act_dev, s->sem_interp, vkctx->hwctx->alloc); if (s->session) vk->DestroyOpticalFlowSessionNV(vkctx->hwctx->act_dev, s->session, vkctx->hwctx->alloc); @@ -1260,8 +1317,6 @@ static av_cold void uninit(AVFilterContext *avctx) } } - ff_vk_exec_pool_free(vkctx, &s->e); - ff_vk_exec_pool_free(vkctx, &s->e_of); ff_vk_shader_free(vkctx, &s->grayscale); ff_vk_shader_free(vkctx, &s->interpolate); -- 2.52.0 From affc3bf642fe09e21d7b83b183f4776b2f5359c6 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Wed, 24 Jun 2026 08:06:44 -0700 Subject: [PATCH 13/14] avfilter/fruc_vulkan: double-buffer optical flow resources The grayscale inputs, flow images and optical flow session were a single shared instance, so each source pair's optical flow execution had to wait for the previous pair's interpolations to finish reading the flow images before it could overwrite them, serialising the optical flow engine against the compute queue. Double-buffer the per-pair resources across two slots (the optical flow session bakes in its image bindings, so each slot owns its own session, grayscale and flow images), indexed by the pair generation. One pair's optical flow execution can then run while the previous slot's flow images are still being sampled by the earlier pair's interpolations. The write-after-read fence on slot reuse is tracked per slot (interp_done) instead of against the single shared instance. As the optical flow engine is a single serial unit, more slots can't increase throughput. --- libavfilter/vf_fruc_vulkan.c | 319 +++++++++++++++++++---------------- 1 file changed, 177 insertions(+), 142 deletions(-) diff --git a/libavfilter/vf_fruc_vulkan.c b/libavfilter/vf_fruc_vulkan.c index db7e563e4b..1bcbb5137e 100644 --- a/libavfilter/vf_fruc_vulkan.c +++ b/libavfilter/vf_fruc_vulkan.c @@ -62,6 +62,30 @@ typedef struct InterpolatePushData { int32_t planes; } InterpolatePushData; +/* Double-buffer the per-pair optical flow resources so one pair's flow execution + * overlaps the previous pair's interpolations instead of stalling on shared + * images. The session bakes in its image bindings, so each slot owns its session, + * grayscale and flow images. Two suffice: the flow engine is serial. */ +#define FRUC_NB_SLOTS 2 + +typedef struct FRUCFlowSlot { + VkOpticalFlowSessionNV session; + + VkImage gray_img[2]; ///< grayscale inputs (INPUT, REFERENCE) + VkDeviceMemory gray_mem[2]; + VkImageView gray_view[2]; + + VkImage flow_img[2]; ///< [0] forward, [1] backward + VkDeviceMemory flow_mem[2]; + VkImageView flow_view[2]; ///< native (SFIXED5) view, bound to the OF session + VkImageView flow_sint_view[2]; ///< R16G16_SINT reinterpret view for sampling + + /* sem_interp value reached by the pair that last used this slot; the next + * pair to reuse it waits here before overwriting the flow images. Zero (the + * initial value) is satisfied immediately, covering the first use. */ + uint64_t interp_done; +} FRUCFlowSlot; + typedef struct FRUCVulkanContext { FFVulkanContext vkctx; @@ -99,8 +123,7 @@ typedef struct FRUCVulkanContext { uint64_t gen; ///< source pair generation (sem_gray/sem_flow value) uint64_t interp_value; ///< monotonic interpolation counter (sem_interp value) - /* Optical flow session and associated images. */ - VkOpticalFlowSessionNV session; + /* Optical flow session parameters (images live per-slot, see slots[]). */ VkOpticalFlowGridSizeFlagsNV grid_bit; int grid_size; VkFormat input_format; ///< grayscale input format @@ -116,14 +139,8 @@ typedef struct FRUCVulkanContext { int flow_height; float luma_weights[4]; ///< RGB->Y weights for the grayscale pass - VkImage gray_img[2]; ///< grayscale inputs (INPUT, REFERENCE) - VkDeviceMemory gray_mem[2]; - VkImageView gray_view[2]; - - VkImage flow_img[2]; ///< [0] forward, [1] backward - VkDeviceMemory flow_mem[2]; - VkImageView flow_view[2]; ///< native (SFIXED5) view, bound to the OF session - VkImageView flow_sint_view[2]; ///< R16G16_SINT reinterpret view for sampling + /* Double-buffered optical flow resources, indexed by (gen % FRUC_NB_SLOTS). */ + FRUCFlowSlot slots[FRUC_NB_SLOTS]; int flow_valid; ///< flow computed for current (f0, f1) pair @@ -312,37 +329,42 @@ static int init_image_layouts(FRUCVulkanContext *s) FFVulkanContext *vkctx = &s->vkctx; FFVulkanFunctions *vk = &vkctx->vkfn; FFVkExecContext *exec = ff_vk_exec_get(vkctx, &s->e); - VkImage imgs[4] = { s->gray_img[0], s->gray_img[1], - s->flow_img[0], s->flow_img[1] }; - VkImageMemoryBarrier2 bar[4]; + VkImageMemoryBarrier2 bar[4 * FRUC_NB_SLOTS]; + int nb_bar = 0; int err; ff_vk_exec_start(vkctx, exec); - for (int i = 0; i < 4; i++) { - bar[i] = (VkImageMemoryBarrier2) { - .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, - .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, - .srcAccessMask = 0, - .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, - .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, - .oldLayout = VK_IMAGE_LAYOUT_UNDEFINED, - .newLayout = VK_IMAGE_LAYOUT_GENERAL, - .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, - .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, - .image = imgs[i], - .subresourceRange = { - .aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, - .levelCount = 1, - .layerCount = 1, - }, - }; + for (int slot = 0; slot < FRUC_NB_SLOTS; slot++) { + FRUCFlowSlot *fs = &s->slots[slot]; + VkImage imgs[4] = { fs->gray_img[0], fs->gray_img[1], + fs->flow_img[0], fs->flow_img[1] }; + + for (int i = 0; i < 4; i++) { + bar[nb_bar++] = (VkImageMemoryBarrier2) { + .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, + .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .srcAccessMask = 0, + .dstStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT | VK_ACCESS_2_MEMORY_WRITE_BIT, + .oldLayout = VK_IMAGE_LAYOUT_UNDEFINED, + .newLayout = VK_IMAGE_LAYOUT_GENERAL, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = imgs[i], + .subresourceRange = { + .aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, + .levelCount = 1, + .layerCount = 1, + }, + }; + } } vk->CmdPipelineBarrier2(exec->buf, &(VkDependencyInfo) { .sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, .pImageMemoryBarriers = bar, - .imageMemoryBarrierCount = 4, + .imageMemoryBarrierCount = nb_bar, }); err = ff_vk_exec_submit(vkctx, exec); @@ -414,11 +436,11 @@ static av_cold int init_filter(AVFilterContext *avctx) s->share_qfs[s->nb_share_qfs++] = s->qf_of->idx; RET(ff_vk_exec_pool_init(vkctx, s->qf, &s->e, s->qf->num, 0, 0, 0, NULL)); - /* Use more than one optical flow context so that the optical flow execution - * for the next frame pair can be recorded and submitted without first + /* One optical flow context per slot so that the optical flow execution for + * the next frame pair can be recorded and submitted without first * host-waiting the previous pair's execution to retire its command buffer. */ RET(ff_vk_exec_pool_init(vkctx, s->qf_of, &s->e_of, - FFMIN(s->qf_of->num, 2), 0, 0, 0, NULL)); + FFMIN(s->qf_of->num, FRUC_NB_SLOTS), 0, 0, 0, NULL)); RET(ff_vk_init_sampler(vkctx, &s->sampler, 0, VK_FILTER_LINEAR)); /* Flow is sampled through an integer view and its format has no linear * filtering support, so use nearest. Requires normalised coords and as we @@ -513,82 +535,86 @@ static av_cold int init_filter(AVFilterContext *avctx) vkctx->optical_flow_props.minWidth, vkctx->optical_flow_props.minHeight, vkctx->optical_flow_props.maxWidth, vkctx->optical_flow_props.maxHeight); - /* Create the persistent optical flow images. */ - RET(create_of_image(s, &s->gray_img[0], &s->gray_mem[0], &s->gray_view[0], - s->input_format, s->width, s->height, - VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV, - VK_IMAGE_USAGE_STORAGE_BIT, 0)); - RET(create_of_image(s, &s->gray_img[1], &s->gray_mem[1], &s->gray_view[1], - s->input_format, s->width, s->height, - VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV, - VK_IMAGE_USAGE_STORAGE_BIT, 0)); - /* The flow images must be created mutable, as we need to sample the raw - * integers through an R16G16_SINT view. SFIXED5 advertises SAMPLED_IMAGE, - * but if you try and use a float sampler, it will read the values as - * R16G16_SFLOAT, resulting in garbage. So we have to read the raw bits and - * rescale them (value/32) ourselves. */ - RET(create_of_image(s, &s->flow_img[0], &s->flow_mem[0], &s->flow_view[0], - s->flow_format, s->flow_width, s->flow_height, - VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV, - VK_IMAGE_USAGE_SAMPLED_BIT, VK_IMAGE_CREATE_MUTABLE_FORMAT_BIT)); - RET(create_of_image(s, &s->flow_img[1], &s->flow_mem[1], &s->flow_view[1], - s->flow_format, s->flow_width, s->flow_height, - VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV, - VK_IMAGE_USAGE_SAMPLED_BIT, VK_IMAGE_CREATE_MUTABLE_FORMAT_BIT)); + /* Create the persistent optical flow images and sessions, one set per slot + * so consecutive source pairs round-robin between independent resources. */ + for (int slot = 0; slot < FRUC_NB_SLOTS; slot++) { + FRUCFlowSlot *fs = &s->slots[slot]; - for (int i = 0; i < 2; i++) { - VkImageViewCreateInfo view_info = { - .sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO, - .image = s->flow_img[i], - .viewType = VK_IMAGE_VIEW_TYPE_2D, - .format = VK_FORMAT_R16G16_SINT, - .components = ff_comp_identity_map, - .subresourceRange = { - .aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, - .levelCount = 1, - .layerCount = 1, - }, - }; - if (vk->CreateImageView(vkctx->hwctx->act_dev, &view_info, - vkctx->hwctx->alloc, &s->flow_sint_view[i]) != VK_SUCCESS) { - av_log(avctx, AV_LOG_ERROR, "Failed to create flow SINT view\n"); + RET(create_of_image(s, &fs->gray_img[0], &fs->gray_mem[0], &fs->gray_view[0], + s->input_format, s->width, s->height, + VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV, + VK_IMAGE_USAGE_STORAGE_BIT, 0)); + RET(create_of_image(s, &fs->gray_img[1], &fs->gray_mem[1], &fs->gray_view[1], + s->input_format, s->width, s->height, + VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV, + VK_IMAGE_USAGE_STORAGE_BIT, 0)); + /* The flow images must be created mutable, as we need to sample the raw + * integers through an R16G16_SINT view. SFIXED5 advertises SAMPLED_IMAGE, + * but if you try and use a float sampler, it will read the values as + * R16G16_SFLOAT, resulting in garbage. So we have to read the raw bits and + * rescale them (value/32) ourselves. */ + RET(create_of_image(s, &fs->flow_img[0], &fs->flow_mem[0], &fs->flow_view[0], + s->flow_format, s->flow_width, s->flow_height, + VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV, + VK_IMAGE_USAGE_SAMPLED_BIT, VK_IMAGE_CREATE_MUTABLE_FORMAT_BIT)); + RET(create_of_image(s, &fs->flow_img[1], &fs->flow_mem[1], &fs->flow_view[1], + s->flow_format, s->flow_width, s->flow_height, + VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV, + VK_IMAGE_USAGE_SAMPLED_BIT, VK_IMAGE_CREATE_MUTABLE_FORMAT_BIT)); + + for (int i = 0; i < 2; i++) { + VkImageViewCreateInfo view_info = { + .sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO, + .image = fs->flow_img[i], + .viewType = VK_IMAGE_VIEW_TYPE_2D, + .format = VK_FORMAT_R16G16_SINT, + .components = ff_comp_identity_map, + .subresourceRange = { + .aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, + .levelCount = 1, + .layerCount = 1, + }, + }; + if (vk->CreateImageView(vkctx->hwctx->act_dev, &view_info, + vkctx->hwctx->alloc, &fs->flow_sint_view[i]) != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Failed to create flow SINT view\n"); + return AVERROR_EXTERNAL; + } + } + + ret = vk->CreateOpticalFlowSessionNV(vkctx->hwctx->act_dev, + &(VkOpticalFlowSessionCreateInfoNV) { + .sType = VK_STRUCTURE_TYPE_OPTICAL_FLOW_SESSION_CREATE_INFO_NV, + .width = s->width, + .height = s->height, + .imageFormat = s->input_format, + .flowVectorFormat = s->flow_format, + .outputGridSize = s->grid_bit, + .performanceLevel = s->perf_level, + .flags = VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NV, + }, vkctx->hwctx->alloc, &fs->session); + if (ret != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Failed to create optical flow session: %s\n", + ff_vk_ret2str(ret)); return AVERROR_EXTERNAL; } + + vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, fs->session, + VK_OPTICAL_FLOW_SESSION_BINDING_POINT_INPUT_NV, + fs->gray_view[0], VK_IMAGE_LAYOUT_GENERAL); + vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, fs->session, + VK_OPTICAL_FLOW_SESSION_BINDING_POINT_REFERENCE_NV, + fs->gray_view[1], VK_IMAGE_LAYOUT_GENERAL); + vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, fs->session, + VK_OPTICAL_FLOW_SESSION_BINDING_POINT_FLOW_VECTOR_NV, + fs->flow_view[0], VK_IMAGE_LAYOUT_GENERAL); + vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, fs->session, + VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_FLOW_VECTOR_NV, + fs->flow_view[1], VK_IMAGE_LAYOUT_GENERAL); } RET(init_image_layouts(s)); - /* Create the optical flow session, requesting forward and backward flow. */ - ret = vk->CreateOpticalFlowSessionNV(vkctx->hwctx->act_dev, - &(VkOpticalFlowSessionCreateInfoNV) { - .sType = VK_STRUCTURE_TYPE_OPTICAL_FLOW_SESSION_CREATE_INFO_NV, - .width = s->width, - .height = s->height, - .imageFormat = s->input_format, - .flowVectorFormat = s->flow_format, - .outputGridSize = s->grid_bit, - .performanceLevel = s->perf_level, - .flags = VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NV, - }, vkctx->hwctx->alloc, &s->session); - if (ret != VK_SUCCESS) { - av_log(avctx, AV_LOG_ERROR, "Failed to create optical flow session: %s\n", - ff_vk_ret2str(ret)); - return AVERROR_EXTERNAL; - } - - vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, s->session, - VK_OPTICAL_FLOW_SESSION_BINDING_POINT_INPUT_NV, - s->gray_view[0], VK_IMAGE_LAYOUT_GENERAL); - vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, s->session, - VK_OPTICAL_FLOW_SESSION_BINDING_POINT_REFERENCE_NV, - s->gray_view[1], VK_IMAGE_LAYOUT_GENERAL); - vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, s->session, - VK_OPTICAL_FLOW_SESSION_BINDING_POINT_FLOW_VECTOR_NV, - s->flow_view[0], VK_IMAGE_LAYOUT_GENERAL); - vk->BindOpticalFlowSessionImageNV(vkctx->hwctx->act_dev, s->session, - VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_FLOW_VECTOR_NV, - s->flow_view[1], VK_IMAGE_LAYOUT_GENERAL); - /* Grayscale extraction shader. */ ff_vk_shader_load(&s->grayscale, VK_SHADER_STAGE_COMPUTE_BIT, NULL, (uint32_t []) { 32, 32, 1 }, 0); @@ -715,6 +741,9 @@ static int compute_flow(AVFilterContext *avctx) /* This pair's generation; sem_gray and sem_flow are signalled with it. */ s->gen++; + /* Round-robin slot for this pair's optical flow resources. */ + FRUCFlowSlot *fs = &s->slots[s->gen % FRUC_NB_SLOTS]; + /* --- Grayscale extraction on the compute queue. --- */ exec = ff_vk_exec_get(vkctx, &s->e); ff_vk_exec_start(vkctx, exec); @@ -725,11 +754,11 @@ static int compute_flow(AVFilterContext *avctx) RET(ff_vk_exec_add_dep_frame(vkctx, exec, s->f1, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); - /* The grayscale images are a single instance reused every pair, and the - * previous pair's optical flow reads them. Wait for that read to retire - * (sem_flow at the previous generation) before overwriting them. */ - if (s->gen > 1) - RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_flow, s->gen - 1, + /* Wait for the prior occupant of this slot (FRUC_NB_SLOTS pairs ago) to + * finish reading its grayscale images on the flow engine before overwriting + * them; the slot's first uses have no prior occupant. */ + if (s->gen > FRUC_NB_SLOTS) + RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_flow, s->gen - FRUC_NB_SLOTS, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); RET(ff_vk_exec_add_dep_signal_sem(vkctx, exec, s->sem_gray, s->gen, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); @@ -743,9 +772,9 @@ static int compute_flow(AVFilterContext *avctx) f1_views[0], VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, s->sampler); ff_vk_shader_update_img(vkctx, exec, &s->grayscale, 0, 1, 0, - s->gray_view[0], VK_IMAGE_LAYOUT_GENERAL, VK_NULL_HANDLE); + fs->gray_view[0], VK_IMAGE_LAYOUT_GENERAL, VK_NULL_HANDLE); ff_vk_shader_update_img(vkctx, exec, &s->grayscale, 0, 1, 1, - s->gray_view[1], VK_IMAGE_LAYOUT_GENERAL, VK_NULL_HANDLE); + fs->gray_view[1], VK_IMAGE_LAYOUT_GENERAL, VK_NULL_HANDLE); ff_vk_exec_bind_shader(vkctx, exec, &s->grayscale); ff_vk_shader_update_push_const(vkctx, exec, &s->grayscale, @@ -765,10 +794,10 @@ static int compute_flow(AVFilterContext *avctx) VK_ACCESS_SHADER_READ_BIT, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, VK_QUEUE_FAMILY_IGNORED); - of_image_barrier(&img_bar[nb_img_bar++], s->gray_img[0], + of_image_barrier(&img_bar[nb_img_bar++], fs->gray_img[0], VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, 0, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_WRITE_BIT); - of_image_barrier(&img_bar[nb_img_bar++], s->gray_img[1], + of_image_barrier(&img_bar[nb_img_bar++], fs->gray_img[1], VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, 0, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_WRITE_BIT); @@ -794,19 +823,17 @@ static int compute_flow(AVFilterContext *avctx) /* Wait for the grayscale writes, signal once the flow has been written. */ RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_gray, s->gen, VK_PIPELINE_STAGE_2_OPTICAL_FLOW_BIT_NV)); - /* The flow images are a single instance reused every pair. The previous - * pair's interpolations sampled them; wait for the last such read to retire - * (the highest interpolation value signalled so far, which belongs to the - * previous pair since this pair has produced none yet) before overwriting - * them. A value of 0 is the initial state and is satisfied immediately. */ - RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_interp, s->interp_value, + /* Wait for the slot's prior occupant to finish sampling its flow images + * (fs->interp_done) before overwriting them; 0 is the initial state of an + * unused slot and is satisfied immediately. */ + RET(ff_vk_exec_add_dep_wait_sem(vkctx, exec, s->sem_interp, fs->interp_done, VK_PIPELINE_STAGE_2_OPTICAL_FLOW_BIT_NV)); RET(ff_vk_exec_add_dep_signal_sem(vkctx, exec, s->sem_flow, s->gen, VK_PIPELINE_STAGE_2_OPTICAL_FLOW_BIT_NV)); /* The flow images stay in VK_IMAGE_LAYOUT_GENERAL and the semaphores handle * cross-queue visibility, so no image barriers are needed here. */ - vk->CmdOpticalFlowExecuteNV(exec->buf, s->session, + vk->CmdOpticalFlowExecuteNV(exec->buf, fs->session, &(VkOpticalFlowExecuteInfoNV) { .sType = VK_STRUCTURE_TYPE_OPTICAL_FLOW_EXECUTE_INFO_NV, }); @@ -853,6 +880,9 @@ static int interpolate_frame(AVFilterContext *avctx, AVFrame *out, float t) return err; } + /* Same slot compute_flow selected for this pair's generation. */ + FRUCFlowSlot *fs = &s->slots[s->gen % FRUC_NB_SLOTS]; + exec = ff_vk_exec_get(vkctx, &s->e); ff_vk_exec_start(vkctx, exec); @@ -867,6 +897,8 @@ static int interpolate_frame(AVFilterContext *avctx, AVFrame *out, float t) VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); RET(ff_vk_exec_add_dep_signal_sem(vkctx, exec, s->sem_interp, ++s->interp_value, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT)); + /* Record this pair's value so the next occupant of the slot can fence on it. */ + fs->interp_done = s->interp_value; RET(ff_vk_exec_add_dep_frame(vkctx, exec, out, VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, @@ -891,9 +923,9 @@ static int interpolate_frame(AVFilterContext *avctx, AVFrame *out, float t) ff_vk_shader_update_img_array(vkctx, exec, &s->interpolate, out, out_views, 0, 2, VK_IMAGE_LAYOUT_GENERAL, VK_NULL_HANDLE); ff_vk_shader_update_img(vkctx, exec, &s->interpolate, 0, 3, 0, - s->flow_sint_view[0], VK_IMAGE_LAYOUT_GENERAL, s->flow_sampler); + fs->flow_sint_view[0], VK_IMAGE_LAYOUT_GENERAL, s->flow_sampler); ff_vk_shader_update_img(vkctx, exec, &s->interpolate, 0, 4, 0, - s->flow_sint_view[1], VK_IMAGE_LAYOUT_GENERAL, s->flow_sampler); + fs->flow_sint_view[1], VK_IMAGE_LAYOUT_GENERAL, s->flow_sampler); ff_vk_exec_bind_shader(vkctx, exec, &s->interpolate); ff_vk_shader_update_push_const(vkctx, exec, &s->interpolate, @@ -918,10 +950,10 @@ static int interpolate_frame(AVFilterContext *avctx, AVFrame *out, float t) VK_ACCESS_SHADER_READ_BIT, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL, VK_QUEUE_FAMILY_IGNORED); - of_image_barrier(&img_bar[nb_img_bar++], s->flow_img[0], + of_image_barrier(&img_bar[nb_img_bar++], fs->flow_img[0], VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, 0, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_READ_BIT); - of_image_barrier(&img_bar[nb_img_bar++], s->flow_img[1], + of_image_barrier(&img_bar[nb_img_bar++], fs->flow_img[1], VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, 0, VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, VK_ACCESS_2_SHADER_READ_BIT); @@ -1296,24 +1328,27 @@ static av_cold void uninit(AVFilterContext *avctx) vk->DestroySemaphore(vkctx->hwctx->act_dev, s->sem_flow, vkctx->hwctx->alloc); if (s->sem_interp) vk->DestroySemaphore(vkctx->hwctx->act_dev, s->sem_interp, vkctx->hwctx->alloc); - if (s->session) - vk->DestroyOpticalFlowSessionNV(vkctx->hwctx->act_dev, s->session, - vkctx->hwctx->alloc); - for (int i = 0; i < 2; i++) { - if (s->gray_view[i]) - vk->DestroyImageView(vkctx->hwctx->act_dev, s->gray_view[i], vkctx->hwctx->alloc); - if (s->gray_img[i]) - vk->DestroyImage(vkctx->hwctx->act_dev, s->gray_img[i], vkctx->hwctx->alloc); - if (s->gray_mem[i]) - vk->FreeMemory(vkctx->hwctx->act_dev, s->gray_mem[i], vkctx->hwctx->alloc); - if (s->flow_view[i]) - vk->DestroyImageView(vkctx->hwctx->act_dev, s->flow_view[i], vkctx->hwctx->alloc); - if (s->flow_sint_view[i]) - vk->DestroyImageView(vkctx->hwctx->act_dev, s->flow_sint_view[i], vkctx->hwctx->alloc); - if (s->flow_img[i]) - vk->DestroyImage(vkctx->hwctx->act_dev, s->flow_img[i], vkctx->hwctx->alloc); - if (s->flow_mem[i]) - vk->FreeMemory(vkctx->hwctx->act_dev, s->flow_mem[i], vkctx->hwctx->alloc); + for (int slot = 0; slot < FRUC_NB_SLOTS; slot++) { + FRUCFlowSlot *fs = &s->slots[slot]; + if (fs->session) + vk->DestroyOpticalFlowSessionNV(vkctx->hwctx->act_dev, fs->session, + vkctx->hwctx->alloc); + for (int i = 0; i < 2; i++) { + if (fs->gray_view[i]) + vk->DestroyImageView(vkctx->hwctx->act_dev, fs->gray_view[i], vkctx->hwctx->alloc); + if (fs->gray_img[i]) + vk->DestroyImage(vkctx->hwctx->act_dev, fs->gray_img[i], vkctx->hwctx->alloc); + if (fs->gray_mem[i]) + vk->FreeMemory(vkctx->hwctx->act_dev, fs->gray_mem[i], vkctx->hwctx->alloc); + if (fs->flow_view[i]) + vk->DestroyImageView(vkctx->hwctx->act_dev, fs->flow_view[i], vkctx->hwctx->alloc); + if (fs->flow_sint_view[i]) + vk->DestroyImageView(vkctx->hwctx->act_dev, fs->flow_sint_view[i], vkctx->hwctx->alloc); + if (fs->flow_img[i]) + vk->DestroyImage(vkctx->hwctx->act_dev, fs->flow_img[i], vkctx->hwctx->alloc); + if (fs->flow_mem[i]) + vk->FreeMemory(vkctx->hwctx->act_dev, fs->flow_mem[i], vkctx->hwctx->alloc); + } } } -- 2.52.0 From 04cb4ec408999ada5b0b1d0d2b57ea4149b2c1a1 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Sun, 21 Jun 2026 10:45:46 -0700 Subject: [PATCH 14/14] avfilter/fruc_vulkan: suppress spurious flow in featureless regions In textureless regions the optical flow engine has no data to track and its regularizer invents a smooth, often large, and internally self-consistent flow field. Backward warping then traces a background pixel along that invented flow onto a nearby moving object and pulls object pixels into what should be empty background, leaving a ghost outline at a distance from the object where nothing is actually moving. Forward/backward flow consistency does not catch this because the invented flow round-trips cleanly. I have no idea if this is a common limitation in optical flow implementations but it's definitely happening with the nvidia hardware, so this change attempts to introduce a heuristic to work around it. When the luma at a given location is sufficiently identical in both frames, and the optical flow magnitude is sufficiently large, we will now assume that the location is static and the flow is bogus. Genuine motion will usually lead to luma differences, and low magnitude flow will be respected either way. Again, this is all highly empirical, and the constants are justified purely by by fiddling around with specific samples. --- libavfilter/vulkan/fruc_interpolate.comp.glsl | 34 ++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/libavfilter/vulkan/fruc_interpolate.comp.glsl b/libavfilter/vulkan/fruc_interpolate.comp.glsl index fbe7b13d17..76a046558e 100644 --- a/libavfilter/vulkan/fruc_interpolate.comp.glsl +++ b/libavfilter/vulkan/fruc_interpolate.comp.glsl @@ -38,6 +38,20 @@ layout (set = 0, binding = 4) uniform isampler2D flow_bwd; #define FLOW_FIXED_POINT_SCALE (1.0 / 32.0) +// Thresholds for the spurious-flow guard + +/* STATIC_LUMA_SIGMA is the per-pixel luma tolerance for calling a region "static" */ +#define STATIC_LUMA_SIGMA 0.03 +/* WARP_NEAR_PX is the threshold below which a warp is small enough that it won't + ever be considered "long range" */ +#define WARP_NEAR_PX 5.0 +/* WARP_FAR_PX is the threshold above which a warp is large enough that it will + always be considered "long range" */ +#define WARP_FAR_PX 20.0 +/* WARP_REACH_SCALE is the factor by which a warp's reach is scaled before the + * range test. Higher values will trigger the guard for smaller displacements. */ +#define WARP_REACH_SCALE 1.0 + /* These Picard values were established empirically on a couple of different * samples, but one could easily imagine reaching a different conclusion from * different data. */ @@ -81,7 +95,25 @@ void main() vec4 c0 = texture(f0_img[i], s0); vec4 c1 = texture(f1_img[i], s1); + vec4 warped = mix(c0, c1, t); - imageStore(out_img[i], pos, mix(c0, c1, t)); + /* Spurious-flow guard. In textureless regions the flow + * engine invents bogus flows we need to ignore. */ + vec4 z0 = texture(f0_img[i], base); + vec4 z1 = texture(f1_img[i], base); + vec4 stat = mix(z0, z1, t); + + float lz = abs(texture(f0_img[0], base).x - texture(f1_img[0], base).x); + float staticness = exp(-(lz * lz) / (STATIC_LUMA_SIGMA * STATIC_LUMA_SIGMA)); + + float disp = max(length((s0 - base) * luma_size), + length((s1 - base) * luma_size)); + /* smoothstep is used to transition between the two thresholds and avoid + * abrupt changes at the boundary. */ + float reach = smoothstep(WARP_NEAR_PX, WARP_FAR_PX, disp * WARP_REACH_SCALE); + + vec4 result = mix(warped, stat, staticness * reach); + + imageStore(out_img[i], pos, result); } } -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]