Re: [PATCH 00/18] avcodec/h264dec: H.264 MVC (Annex H) multiview decoding

Did you start from the MVC decoder from a long time ago?

On Tue, 25 Aug 2026, at 18:49, Dom Cobley via ffmpeg-devel wrote:
> This series adds decoding of H.264 MVC streams - Blu-ray 3D and the
> consumer camcorder recordings that use the same coding - to the existing
> h264 decoder, alongside the MV-HEVC support hevcdec already has, and using
> the same view_ids/view_ids_available/view_pos_available option interface so
> that the ffmpeg CLI view specifiers work for both.
>
> Only the base view is decoded by default, so nothing changes for a caller
> that does not ask for more.
>
> This work was significantly helped by Claude Opus 5. I have tried to keep the
> commits as clean and self contained as possible. I will take ownership of the
> changes and help with resolving any future issues.
>
> The series is bisectable - every commit builds and passes fate-h264 - and it has
> had a good deal of testing beyond that, including a public LibreELEC test build:
>
>   https://forum.libreelec.tv/thread/30548-3d-support-builds-for-raspberry-pi
>
> FATE SAMPLE
> ===========
>
> Patch 18 adds three tests that need a new sample, h264/mvc-2view.264
> (18628 bytes, md5 99901e05413ff29aecc6f191c664d8b2):
>
>   
> https://drive.google.com/uc?export=download&id=1y6xxbM8nwFCgtCvZizujChhcQoz8_S4w
>
> I have sent a separate mail to samples-request. I generated the file with
> the JMVC 8.5 reference encoder from synthetic input, so it is freely
> redistributable: nothing in the real world MVC material I tested against
> can be, being either commercial films or camcorder captures of unclear
> origin. Both views decode bit-identically to JMVC's own reconstruction, so
> the tests check conformance rather than only locking in current behaviour.
>
> APPROACH
> ========
>
> Rather than address every view explicitly, as hevcdec does with
> HEVCLayerContext, the decoder keeps the per-view state that actually
> differs - POC and the reference lists - in a small H264ViewContext and
> swaps it in and out around a view change. That avoids touching the ~250
> sites that would otherwise have needed a layer index threaded through them,
> and keeps the diff to the parts of the decoder that genuinely differ per
> view.
>
> Patch 8 converts the decoder to receive_frame(). An MVC access unit
> produces two frames from one packet, which the decode() callback cannot
> express.
>
> Patches 1 and 2 are not multiview at all. Both fix frame_num gap
> concealment on damaged input, and both reproduce on an unmodified tree
> without any of the rest of this series. They come first because the
> multiview paths make the second one about five times easier to reach: on
> one corpus file it deadlocks 1 run in 14 as things stand, and 5 in 14 once
> a view is selected with -map 0:v:0:view:1.
>
> WHAT WORKS
> ==========
>
> Both views, bit-identical to a single threaded decode, under frame
> threading, slice threading and no threading, at every thread count I
> tested, including on a Matroska file whose blocks do not start on an access
> unit boundary - the dependent view slices of one access unit sit at the
> head of the packet that goes on to carry the base view slices of the next.
> That case drove several of the fixes folded into patches 11 and 14.
>
> Field coded multiview works (patch 17).
>
> Hardware acceleration does not, and is refused with a warning: every h264
> hwaccel builds its reference list from short_ref/long_ref and skips
> pictures with reference == 0, which is exactly what an inter-view reference
> looks like. Patch 15 has the detail.
>
> TESTING
> =======
>
> - make fate: 5649/5649 passing.
> - Each of the 18 commits builds and passes fate-h264 standalone, so the
>   series bisects cleanly.
> - A corpus of 43 real MVC and 3D files, 35 GB, decoded end to end with both
>   views mapped: Blu-ray rips, camcorder .264/.MTS/.ssif, and Matroska. Every
>   file that carries two views decodes both of them with no errors and with
>   matching frame counts, the one exception being a Matroska file that is
>   itself missing the dependent view slices of its last access unit (899
>   against 900, which is what the file contains). Frame threading and slice
>   threading give byte-identical framecrc output over the whole corpus. Where
>   I compared against an unpatched build - the files exercising the awkward
>   paths, not all 43 - the base view output was bit-exact.
> - Corrupted input, as doc/developer.texi asks for: 12000 decodes of damaged
>   streams across 16 files, under 10 threading configurations and 13
>   mapping/decoder-option combinations, with the noise bitstream filter,
>   truncation and byte flipping. Clean under ASan and UBSan, and no hangs.
>   Getting there found four deadlocks and a batch of undefined behaviour,
>   all fixed in the patches that introduced them, except the concealment one
>   described above.
>
> Three faults that turned up during that and are NOT this series' - each
> reproduces on an unmodified tree, and I will report them separately:
>
>   - fftools/ffmpeg_sched.c:2168 asserts on some damaged input.
>   - -err_detect +explode with frame threading deadlocks on plain
>     single-view H.264.
>   - Mapping both views in one ffmpeg command collapses the dependent view's
>     timestamps in fftools when the two views share a PTS, which they do by
>     definition. Mapping either view alone is correct, and the decoder emits
>     correct timestamps in both cases. This looks like it would affect
>     MV-HEVC equally.
>
> KNOWN GAPS
> ==========
>
> - mpegts carries the two views on separate PIDs (0x1011 and 0x1012) and
>   presents them as separate streams, so a Blu-ray .ssif or .MTS still
>   decodes only the view that was mapped. Associating the PIDs needs demuxer
>   work that is not in this series.
> - cbs_h264 still refuses NAL 15 and NAL 20.
>
> Dom Cobley (18):
>   avcodec/h264dec: always initialise a concealed frame_num gap picture
>   avcodec/h264dec: do not await a field that was never decoded
>   avcodec/h264_ps: fix swapped MVC profile comments
>   avcodec/h264: parse subset SPS (NAL 15) and its MVC extension
>   avcodec/h2645_parse: parse the H.264 MVC NAL unit header extension
>   avcodec/h264dec: add multiview view_ids options
>   avcodec/h264dec: add a per-view context
>   avcodec/h264dec: switch to receive_frame()
>   avcodec/h264: decode dependent views and inter-view references
>   avcodec/h264_slice: do not reclaim the picture awaiting output
>   avcodec/h264dec: output both views of an access unit together
>   avcodec/h264: implement inter-view reference list modification
>   avcodec/h264dec: pick up the view list wherever the subset SPS appears
>   avcodec/h264dec: support frame and slice threading for multiview
>   avcodec/h264dec: refuse multiview with hardware acceleration
>   avformat/mov: read the mvcC box
>   avcodec/h264dec: support field coded multiview
>   doc/decoders, Changelog, fate: document and test H.264 MVC
>
>  Changelog                          |   1 +
>  doc/decoders.texi                  |  45 +++
>  libavcodec/bsf/extract_extradata.c |   2 +-
>  libavcodec/h2645_parse.c           |  38 ++
>  libavcodec/h2645_parse.h           |  13 +
>  libavcodec/h264_direct.c           |  15 +-
>  libavcodec/h264_parse.c            |   4 +
>  libavcodec/h264_parser.c           |   3 +
>  libavcodec/h264_picture.c          |   2 +
>  libavcodec/h264_ps.c               | 161 +++++++-
>  libavcodec/h264_ps.h               |  57 +++
>  libavcodec/h264_refs.c             | 285 +++++++++++++-
>  libavcodec/h264_slice.c            | 573 +++++++++++++++++++++++++++--
>  libavcodec/h264dec.c               | 307 ++++++++++++++--
>  libavcodec/h264dec.h               | 182 +++++++++
>  libavformat/mov.c                  | 122 ++++++
>  tests/fate/h264.mak                |   9 +
>  tests/ref/fate/flv-demux           |   2 +-
>  tests/ref/fate/h264-mvc-base       |  21 ++
>  tests/ref/fate/h264-mvc-both       |  42 +++
>  tests/ref/fate/h264-mvc-dependent  |  21 ++
>  tests/ref/fate/mov-zombie          |   2 +-
>  tests/ref/fate/ts-small-demux      |   2 +-
>  23 files changed, 1823 insertions(+), 86 deletions(-)
>  create mode 100644 tests/ref/fate/h264-mvc-base
>  create mode 100644 tests/ref/fate/h264-mvc-both
>  create mode 100644 tests/ref/fate/h264-mvc-dependent
>
> -- 
> 2.53.0
>
> _______________________________________________
> ffmpeg-devel mailing list -- [email protected]
> To unsubscribe send an email to [email protected]

-- 
Jean-Baptiste Kempf -  President
+33 672 704 734
https://jbkempf.com/
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]