[PR] avcodec/cuviddec, avutil/hwcontext_cuda: fix 12-bit 4:2:0/4:2:2 and 10/12-bit 4:4:4 NVDEC/CUVID decode (PR #23572)

PR #23572 opened by Diego de Souza (ddesouza)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23572
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23572.patch

Commit 7fd9780a ("avcodec: Remove FF_API_NVDEC_OLD_PIX_FMTS") completed the switch to the new NVDEC pixel formats, removing the old-format fallbacks. The NVDEC/CUVID decoders now select the new formats for high-bit-depth content:

- 12-bit 4:2:0 → AV_PIX_FMT_P012
- 10/12-bit 4:2:2 → AV_PIX_FMT_P210 / AV_PIX_FMT_P212
- 10/12-bit 4:4:4 → AV_PIX_FMT_YUV444P10MSB / AV_PIX_FMT_YUV444P12MSB

but two places were never updated to handle them, so decoding these streams fails on current master:

1. avutil/hwcontext_cuda — P012 and P212 are missing from the CUDA frames context's supported_formats[], so av_hwframe_ctx_init() rejects them with AVERROR(ENOSYS) ("Pixel format not supported"). This breaks 12-bit decoding to CUDA frames (NVDEC and CUVID) and hwdownload of such frames.
2. avcodec/cuviddec — cuvid_handle_video_sequence() never mapped P012/P210/P212/YUV444P10MSB/YUV444P12MSB to a CUVID surface format, so they hit the default: arm with "Unsupported output format" (AVERROR(EINVAL)). Because every packet then errors without producing a frame, this can also surface as an apparent hang rather than a clean error. The system-memory copy path in cuvid_output_frame() rejected the same formats with AVERROR_BUG.

**Changes**

avutil/hwcontext_cuda: add P012 and P212 to supported formats
- Add AV_PIX_FMT_P012 and AV_PIX_FMT_P212 next to the existing P010/P016 and P210/P216 entries. The per-plane device transfer is derived generically from the pixel descriptor, so no further changes are needed.

avcodec/cuviddec: map new NVDEC 4:2:0/4:2:2/4:4:4 output formats
- Map P012 → cudaVideoSurfaceFormat_P016 (mirroring P010), P210/P212 → cudaVideoSurfaceFormat_P216 (mirroring P216), and YUV444P10MSB/YUV444P12MSB → cudaVideoSurfaceFormat_YUV444_16Bit (mirroring YUV444P16). The hardware surface is physically 16-bit; the pixel format only records the number of valid bits.
- Add the same formats to the system-memory copy path, whose per-plane geometry is already derived from the pixel descriptor.


From 531a6527a25c5721540f38f89eab8bcb7ff8fe7d Mon Sep 17 00:00:00 2001
From: Diego de Souza <[email protected]>
Date: Wed, 24 Jun 2026 00:30:45 +0200
Subject: [PATCH 1/2] avutil/hwcontext_cuda: add P012 and P212 to supported
 formats

NVDEC and CUVID decode 12-bit 4:2:0 content to AV_PIX_FMT_P012 and
12-bit 4:2:2 to AV_PIX_FMT_P212, but these formats were missing from
the CUDA frames context supported format list. As a result
av_hwframe_ctx_init() rejected them ("Pixel format not supported",
AVERROR(ENOSYS)), so decoding 12-bit content to CUDA frames and
hwdownload of such frames both failed.

Add P012 and P212 next to the existing P010/P016 and P210/P216 entries.
The per-plane device transfer is derived generically from the pixel
descriptor, so no other changes are required.

Signed-off-by: Diego de Souza <[email protected]>
---
 libavutil/hwcontext_cuda.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/libavutil/hwcontext_cuda.c b/libavutil/hwcontext_cuda.c
index b0b65b2446..c57ea4bb7a 100644
--- a/libavutil/hwcontext_cuda.c
+++ b/libavutil/hwcontext_cuda.c
@@ -47,8 +47,10 @@ static const enum AVPixelFormat supported_formats[] = {
     AV_PIX_FMT_YUVA420P,
     AV_PIX_FMT_YUV444P,
     AV_PIX_FMT_P010,
+    AV_PIX_FMT_P012,
     AV_PIX_FMT_P016,
     AV_PIX_FMT_P210,
+    AV_PIX_FMT_P212,
     AV_PIX_FMT_P216,
     AV_PIX_FMT_YUV422P,
     AV_PIX_FMT_YUV420P10,
-- 
2.52.0


From 8108c864f7d9fca1369beebefb688fe8346daedd Mon Sep 17 00:00:00 2001
From: Diego de Souza <[email protected]>
Date: Wed, 24 Jun 2026 00:30:45 +0200
Subject: [PATCH 2/2] avcodec/cuviddec: map new NVDEC 4:2:0/4:2:2/4:4:4 output
 formats

The switch to the new NVDEC pixel formats made cuvid_derive_output_format()
and the format probe select AV_PIX_FMT_P012 for 12-bit 4:2:0,
AV_PIX_FMT_P210/P212 for 10/12-bit 4:2:2 and AV_PIX_FMT_YUV444P10MSB/
YUV444P12MSB for 10/12-bit 4:4:4, but cuvid_handle_video_sequence() still
only mapped the older formats to a CUVID surface format. Decoding such
streams therefore failed with "Unsupported output format" (AVERROR(EINVAL));
because every packet then errored without producing a frame, the failure
could also manifest as a hang rather than a clean error exit. The
system-memory copy path in cuvid_output_frame() rejected the same formats
with AVERROR_BUG.

Map P012 to cudaVideoSurfaceFormat_P016 (mirroring P010), P210/P212 to
cudaVideoSurfaceFormat_P216 (mirroring P216) and YUV444P10MSB/YUV444P12MSB
to cudaVideoSurfaceFormat_YUV444_16Bit (mirroring YUV444P16); the hardware
surface is physically 16-bit and the pixel format only records the number
of valid bits. Add the same formats to the system-memory copy path, whose
per-plane geometry is already derived generically from the pixel descriptor.

Signed-off-by: Diego de Souza <[email protected]>
---
 libavcodec/cuviddec.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/libavcodec/cuviddec.c b/libavcodec/cuviddec.c
index d423d2611b..292015d18a 100644
--- a/libavcodec/cuviddec.c
+++ b/libavcodec/cuviddec.c
@@ -333,6 +333,7 @@ static int CUDAAPI cuvid_handle_video_sequence(void *opaque, CUVIDEOFORMAT* form
         cuinfo.OutputFormat = cudaVideoSurfaceFormat_NV12;
         break;
     case AV_PIX_FMT_P010:
+    case AV_PIX_FMT_P012:
     case AV_PIX_FMT_P016:
         cuinfo.OutputFormat = cudaVideoSurfaceFormat_P016;
         break;
@@ -340,6 +341,8 @@ static int CUDAAPI cuvid_handle_video_sequence(void *opaque, CUVIDEOFORMAT* form
     case AV_PIX_FMT_NV16:
         cuinfo.OutputFormat = cudaVideoSurfaceFormat_NV16;
         break;
+    case AV_PIX_FMT_P210:
+    case AV_PIX_FMT_P212:
     case AV_PIX_FMT_P216:
         cuinfo.OutputFormat = cudaVideoSurfaceFormat_P216;
         break;
@@ -347,6 +350,8 @@ static int CUDAAPI cuvid_handle_video_sequence(void *opaque, CUVIDEOFORMAT* form
     case AV_PIX_FMT_YUV444P:
         cuinfo.OutputFormat = cudaVideoSurfaceFormat_YUV444;
         break;
+    case AV_PIX_FMT_YUV444P10MSB:
+    case AV_PIX_FMT_YUV444P12MSB:
     case AV_PIX_FMT_YUV444P16:
         cuinfo.OutputFormat = cudaVideoSurfaceFormat_YUV444_16Bit;
         break;
@@ -632,12 +637,17 @@ static int cuvid_output_frame(AVCodecContext *avctx, AVFrame *frame)
             }
         } else if (avctx->pix_fmt == AV_PIX_FMT_NV12      ||
                    avctx->pix_fmt == AV_PIX_FMT_P010      ||
+                   avctx->pix_fmt == AV_PIX_FMT_P012      ||
                    avctx->pix_fmt == AV_PIX_FMT_P016      ||
 #ifdef NVDEC_HAVE_422_SUPPORT
                    avctx->pix_fmt == AV_PIX_FMT_NV16      ||
+                   avctx->pix_fmt == AV_PIX_FMT_P210      ||
+                   avctx->pix_fmt == AV_PIX_FMT_P212      ||
                    avctx->pix_fmt == AV_PIX_FMT_P216      ||
 #endif
                    avctx->pix_fmt == AV_PIX_FMT_YUV444P   ||
+                   avctx->pix_fmt == AV_PIX_FMT_YUV444P10MSB ||
+                   avctx->pix_fmt == AV_PIX_FMT_YUV444P12MSB ||
                    avctx->pix_fmt == AV_PIX_FMT_YUV444P16) {
             unsigned int offset = 0;
             AVFrame *tmp_frame = av_frame_alloc();
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]