[PATCH] avcodec/videotoolbox: don't restart session on kVTVideoDecoderBadDataErr

c6214b0d69 stopped restarting the decode session on
kVTVideoDecoderReferenceMissingErr (-17694), which otherwise loops on streams
with CRA but no IDR. The same restart loop occurs with kVTVideoDecoderBadDataErr
(-12909): VideoToolbox reports it for a picture it cannot decode, e.g. transient
corrupt or truncated input data. Restarting the session there discards its
reference frames, so the next inter-coded picture also fails -> another null
frame -> another restart, latching a permanent reconfig loop (one
videotoolbox_stop()/videotoolbox_start() per frame).

Treat -12909 like -17694: drop the picture and recover on subsequent valid
pictures. Genuine reconfiguration is unaffected -- SPS profile/level changes
still set reconfig_needed, and session faults
(kVTVideoDecoderMalfunctionErr / kVTInvalidSessionErr) still set it from the
decode-status check.

Signed-off-by: Derek Cyrus-Chow <[email protected]>
---
 libavcodec/videotoolbox.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
index 8df8a3214c..6dfe6804f7 100644
--- a/libavcodec/videotoolbox.c
+++ b/libavcodec/videotoolbox.c
@@ -730,8 +730,17 @@ static void videotoolbox_decoder_callback(void *opaque,
     }
 
     if (!image_buffer) {
-        // kVTVideoDecoderReferenceMissingErr, defined since the macOS 12 SDKs
-        if (status != -17694)
+        /* Skip the session restart for recoverable per-picture errors. A
+         * restart (videotoolbox_stop()/start()) discards reference frames, so
+         * the next inter-coded picture fails too -> another null frame ->
+         * another restart, latching a reconfig loop (one stop/start per frame).
+         *   -17694 kVTVideoDecoderReferenceMissingErr (named since the macOS 12
+         *          SDKs): missing references, e.g. a CRA without a leading IDR.
+         *   -12909 kVTVideoDecoderBadDataErr: a picture that cannot be decoded,
+         *          e.g. transient corrupt or truncated input; recovers on the
+         *          next valid picture, so dropping it beats tearing down the
+         *          session. */
+        if (status != -17694 && status != -12909)
             vtctx->reconfig_needed = true;
 
         av_log(vtctx->logctx, status ? AV_LOG_WARNING : AV_LOG_DEBUG,
-- 
2.41.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]