[PATCH 00/18] avcodec/h264dec: H.264 MVC (Annex H) multiview decoding

This series adds decoding of H.264 MVC streams - Blu-ray 3D and the
consumer camcorder recordings that use the same coding - to the existing
h264 decoder, alongside the MV-HEVC support hevcdec already has, and using
the same view_ids/view_ids_available/view_pos_available option interface so
that the ffmpeg CLI view specifiers work for both.

Only the base view is decoded by default, so nothing changes for a caller
that does not ask for more.

This work was significantly helped by Claude Opus 5. I have tried to keep the
commits as clean and self contained as possible. I will take ownership of the
changes and help with resolving any future issues.

The series is bisectable - every commit builds and passes fate-h264 - and it has
had a good deal of testing beyond that, including a public LibreELEC test build:

  https://forum.libreelec.tv/thread/30548-3d-support-builds-for-raspberry-pi

FATE SAMPLE
===========

Patch 18 adds three tests that need a new sample, h264/mvc-2view.264
(18628 bytes, md5 99901e05413ff29aecc6f191c664d8b2):

  https://drive.google.com/uc?export=download&id=1y6xxbM8nwFCgtCvZizujChhcQoz8_S4w

I have sent a separate mail to samples-request. I generated the file with
the JMVC 8.5 reference encoder from synthetic input, so it is freely
redistributable: nothing in the real world MVC material I tested against
can be, being either commercial films or camcorder captures of unclear
origin. Both views decode bit-identically to JMVC's own reconstruction, so
the tests check conformance rather than only locking in current behaviour.

APPROACH
========

Rather than address every view explicitly, as hevcdec does with
HEVCLayerContext, the decoder keeps the per-view state that actually
differs - POC and the reference lists - in a small H264ViewContext and
swaps it in and out around a view change. That avoids touching the ~250
sites that would otherwise have needed a layer index threaded through them,
and keeps the diff to the parts of the decoder that genuinely differ per
view.

Patch 8 converts the decoder to receive_frame(). An MVC access unit
produces two frames from one packet, which the decode() callback cannot
express.

Patches 1 and 2 are not multiview at all. Both fix frame_num gap
concealment on damaged input, and both reproduce on an unmodified tree
without any of the rest of this series. They come first because the
multiview paths make the second one about five times easier to reach: on
one corpus file it deadlocks 1 run in 14 as things stand, and 5 in 14 once
a view is selected with -map 0:v:0:view:1.

WHAT WORKS
==========

Both views, bit-identical to a single threaded decode, under frame
threading, slice threading and no threading, at every thread count I
tested, including on a Matroska file whose blocks do not start on an access
unit boundary - the dependent view slices of one access unit sit at the
head of the packet that goes on to carry the base view slices of the next.
That case drove several of the fixes folded into patches 11 and 14.

Field coded multiview works (patch 17).

Hardware acceleration does not, and is refused with a warning: every h264
hwaccel builds its reference list from short_ref/long_ref and skips
pictures with reference == 0, which is exactly what an inter-view reference
looks like. Patch 15 has the detail.

TESTING
=======

- make fate: 5649/5649 passing.
- Each of the 18 commits builds and passes fate-h264 standalone, so the
  series bisects cleanly.
- A corpus of 43 real MVC and 3D files, 35 GB, decoded end to end with both
  views mapped: Blu-ray rips, camcorder .264/.MTS/.ssif, and Matroska. Every
  file that carries two views decodes both of them with no errors and with
  matching frame counts, the one exception being a Matroska file that is
  itself missing the dependent view slices of its last access unit (899
  against 900, which is what the file contains). Frame threading and slice
  threading give byte-identical framecrc output over the whole corpus. Where
  I compared against an unpatched build - the files exercising the awkward
  paths, not all 43 - the base view output was bit-exact.
- Corrupted input, as doc/developer.texi asks for: 12000 decodes of damaged
  streams across 16 files, under 10 threading configurations and 13
  mapping/decoder-option combinations, with the noise bitstream filter,
  truncation and byte flipping. Clean under ASan and UBSan, and no hangs.
  Getting there found four deadlocks and a batch of undefined behaviour,
  all fixed in the patches that introduced them, except the concealment one
  described above.

Three faults that turned up during that and are NOT this series' - each
reproduces on an unmodified tree, and I will report them separately:

  - fftools/ffmpeg_sched.c:2168 asserts on some damaged input.
  - -err_detect +explode with frame threading deadlocks on plain
    single-view H.264.
  - Mapping both views in one ffmpeg command collapses the dependent view's
    timestamps in fftools when the two views share a PTS, which they do by
    definition. Mapping either view alone is correct, and the decoder emits
    correct timestamps in both cases. This looks like it would affect
    MV-HEVC equally.

KNOWN GAPS
==========

- mpegts carries the two views on separate PIDs (0x1011 and 0x1012) and
  presents them as separate streams, so a Blu-ray .ssif or .MTS still
  decodes only the view that was mapped. Associating the PIDs needs demuxer
  work that is not in this series.
- cbs_h264 still refuses NAL 15 and NAL 20.

Dom Cobley (18):
  avcodec/h264dec: always initialise a concealed frame_num gap picture
  avcodec/h264dec: do not await a field that was never decoded
  avcodec/h264_ps: fix swapped MVC profile comments
  avcodec/h264: parse subset SPS (NAL 15) and its MVC extension
  avcodec/h2645_parse: parse the H.264 MVC NAL unit header extension
  avcodec/h264dec: add multiview view_ids options
  avcodec/h264dec: add a per-view context
  avcodec/h264dec: switch to receive_frame()
  avcodec/h264: decode dependent views and inter-view references
  avcodec/h264_slice: do not reclaim the picture awaiting output
  avcodec/h264dec: output both views of an access unit together
  avcodec/h264: implement inter-view reference list modification
  avcodec/h264dec: pick up the view list wherever the subset SPS appears
  avcodec/h264dec: support frame and slice threading for multiview
  avcodec/h264dec: refuse multiview with hardware acceleration
  avformat/mov: read the mvcC box
  avcodec/h264dec: support field coded multiview
  doc/decoders, Changelog, fate: document and test H.264 MVC

 Changelog                          |   1 +
 doc/decoders.texi                  |  45 +++
 libavcodec/bsf/extract_extradata.c |   2 +-
 libavcodec/h2645_parse.c           |  38 ++
 libavcodec/h2645_parse.h           |  13 +
 libavcodec/h264_direct.c           |  15 +-
 libavcodec/h264_parse.c            |   4 +
 libavcodec/h264_parser.c           |   3 +
 libavcodec/h264_picture.c          |   2 +
 libavcodec/h264_ps.c               | 161 +++++++-
 libavcodec/h264_ps.h               |  57 +++
 libavcodec/h264_refs.c             | 285 +++++++++++++-
 libavcodec/h264_slice.c            | 573 +++++++++++++++++++++++++++--
 libavcodec/h264dec.c               | 307 ++++++++++++++--
 libavcodec/h264dec.h               | 182 +++++++++
 libavformat/mov.c                  | 122 ++++++
 tests/fate/h264.mak                |   9 +
 tests/ref/fate/flv-demux           |   2 +-
 tests/ref/fate/h264-mvc-base       |  21 ++
 tests/ref/fate/h264-mvc-both       |  42 +++
 tests/ref/fate/h264-mvc-dependent  |  21 ++
 tests/ref/fate/mov-zombie          |   2 +-
 tests/ref/fate/ts-small-demux      |   2 +-
 23 files changed, 1823 insertions(+), 86 deletions(-)
 create mode 100644 tests/ref/fate/h264-mvc-base
 create mode 100644 tests/ref/fate/h264-mvc-both
 create mode 100644 tests/ref/fate/h264-mvc-dependent

-- 
2.53.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]