[PR] avfilter/af_whisper: Reuse detected language across chunks (PR #23671)

PR #23671 opened by WyattBlue
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23671
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23671.patch

With language=auto, whisper reruns language detection on every queue
chunk, an extra encoder pass that can also drift to a different
language mid-file. Cache the first detected language so subsequent
chunks reuse it.

Only lock once a chunk produces segments, so a silent or musical
opening chunk does set the detected language.

Improves multilingual-model performance by ~33%



>From 0bf16d65d85ca0b330c39ecb3a11f32c885489fe Mon Sep 17 00:00:00 2001
From: WyattBlue <[email protected]>
Date: Wed, 1 Jul 2026 00:33:54 -0400
Subject: [PATCH] avfilter/af_whisper: Reuse detected language across chunks

With language=auto, whisper reruns language detection on every queue
chunk, an extra encoder pass that can also drift to a different
language mid-file. Cache the first detected language so subsequent
chunks reuse it.

Only lock once a chunk produces segments, so a silent or musical
opening chunk does set the detected language.
---
 libavfilter/af_whisper.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/libavfilter/af_whisper.c b/libavfilter/af_whisper.c
index 75103ced0c..77250274b7 100644
--- a/libavfilter/af_whisper.c
+++ b/libavfilter/af_whisper.c
@@ -233,6 +233,19 @@ static void run_transcription(AVFilterContext *ctx, AVFrame *frame, int samples)
     const int n_segments = whisper_full_n_segments(wctx->ctx_wsp);
     char *segments_text = NULL;
 
+    if (n_segments > 0 && !av_strcasecmp(wctx->language, "auto")) {
+        const int lang_id = whisper_full_lang_id(wctx->ctx_wsp);
+        if (lang_id >= 0) {
+            char *detected = av_strdup(whisper_lang_str(lang_id));
+            if (detected) {
+                av_freep(&wctx->language_str);
+                wctx->language_str = detected;
+                wctx->language     = detected;
+                av_log(ctx, AV_LOG_INFO, "Locked auto-detected language: %s\n", detected);
+            }
+        }
+    }
+
     for (int i = 0; i < n_segments; ++i) {
         const char *text = whisper_full_get_segment_text(wctx->ctx_wsp, i);
         if (av_isspace(text[0]))
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]