[PR] Implement JPEG Ultra HDR (PR #24264)
Niklas Haas via ffmpeg-devel <[email protected]>
| Newsgroups | gmane.comp.video.ffmpeg.devel |
|---|---|
| Message-ID | <[email protected]> |
PR #24264 opened by Niklas Haas (haasn) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24264 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24264.patch This consists of a number of components which all need to work together: - An abstract definition for AVGainMapParams (following ISO 21496-1) and corresponding stream groups (defined as another subtype of layered video) - A new video filter to generate gain maps from two variants of an image (`vf_gainmap`) and attach the measure parameters as frame side data - The ability to serialize ISO 21496-1 blobs from side data in `mjpegenc`, via new helper function `ff_gain_map_params_to_iso21496` - A new muxer which takes a JPEG base image and a JPEG gain map (with an ISO 21496-1 blob) and muxes them into a single JPEG Multiple Picture Format file This degree of flexibility permits a number of different use cases, e.g.: - The base image can be an already compressed JPEG file passed verbatim to the muxer - A bitstream filter (not included as part of this PR) could serialize the ISO 21496 blob from parameters specified on the command line - The optimal gain map parameters can be deduced from the image contents, rather than hard-coded or specified, and the mode can be chosen freely by the user This PR does introduce one piece of dead code (`ff_gain_map_params_from_iso21496`), because the corresponding *demuxer* has not yet been written as part of this PR. However, I think it's natural to define it anyways for symmetry with `ff_gain_map_params_to_iso21496`. Since this function has no external linkage, it should be eliminated automatically during compilation. I relied on libultrahdr internals during development, though my code deviates from it in a number of important ways, especially w.r.t. the JPEG MPF muxing. Unlike libultrahdr, we also make an effort to e.g. compute the optimal encoding gamma during gain map generation. The most notable omission from this PR, besides a corresponding decoder/demuxer, is the absence of any XMP metadata. Google's JPEG Ultra HDR spec defines an XMP-based encoding for the gain map parameters, that is orthogonal to (and inferior to) the ISO 21496-1 scheme, and *suggests* files be written with both XMP and ISO metadata. However, this is not a hard requirement, and the XMP format both requires float->string printing (locale-sensitive), and is less flexible (e.g. not permitting differing parameters per-channel). >From 3adb07cfaa90eaa8fe191e170feedea4059866b3 Mon Sep 17 00:00:00 2001 From: Niklas Haas <[email protected]> Date: Tue, 25 Aug 2026 13:54:39 +0200 Subject: [PATCH 1/8] avutil/gain_map: add AVGainMapParams and side data type This commit just adds the public facing struct definitions and side data. The struct follows ISO 21496-1 exactly, with a validation function to check for representable and legal values. Signed-off-by: Niklas Haas <[email protected]> --- doc/APIchanges | 6 +++ libavutil/Makefile | 2 + libavutil/frame.h | 7 +++ libavutil/gain_map.c | 93 +++++++++++++++++++++++++++++++++++ libavutil/gain_map.h | 109 ++++++++++++++++++++++++++++++++++++++++++ libavutil/side_data.c | 1 + libavutil/version.h | 2 +- 7 files changed, 219 insertions(+), 1 deletion(-) create mode 100644 libavutil/gain_map.c create mode 100644 libavutil/gain_map.h diff --git a/doc/APIchanges b/doc/APIchanges index 40b4ba3580..1b49f7183e 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -2,6 +2,12 @@ The last version increases of all libraries were on 2026-06-23. API changes, most recent first: +2026-08-17 - xxxxxxxxxx - lavu 61.7.100 - frame.h gain_map.h + Add AV_FRAME_DATA_GAIN_MAP_PARAMS. + Add AV_ISO21496_VERSION, AV_ISO21496_IDENTIFIER. + Add AVGainMapParams, av_gain_map_params_validate(). + Add av_gain_map_params_alloc(), av_gain_map_params_create_side_data(). + 2026-08-23 - xxxxxxxxxx - lavu 61.6.100 - channel_layout.h Add AV_CH_LAYOUT_5POINT1POINT4 and AV_CHANNEL_LAYOUT_5POINT1POINT4. Add AV_CH_LAYOUT_7POINT1POINT4 and AV_CHANNEL_LAYOUT_7POINT1POINT4. diff --git a/libavutil/Makefile b/libavutil/Makefile index 9cb3108b38..287858eb34 100644 --- a/libavutil/Makefile +++ b/libavutil/Makefile @@ -37,6 +37,7 @@ HEADERS = adler32.h \ file.h \ film_grain_params.h \ frame.h \ + gain_map.h \ hash.h \ hdr_dynamic_metadata.h \ hdr_dynamic_vivid_metadata.h \ @@ -146,6 +147,7 @@ OBJS = adler32.o \ film_grain_params.o \ fixed_dsp.o \ frame.o \ + gain_map.o \ hash.o \ hdr_dynamic_metadata.o \ hdr_dynamic_vivid_metadata.o \ diff --git a/libavutil/frame.h b/libavutil/frame.h index e8cc765e5d..1fd296aa83 100644 --- a/libavutil/frame.h +++ b/libavutil/frame.h @@ -305,6 +305,13 @@ enum AVFrameSideDataType { * The data is the AVDownmixMatrix struct defined in libavutil/downmix_info.h. */ AV_FRAME_DATA_DOWNMIX_MATRIX, + + /** + * Parameters describing how to combine this gain map with its base image + * to form an alternate rendition. The payload is the AVGainMapParams + * struct defined in libavutil/gain_map.h. + */ + AV_FRAME_DATA_GAIN_MAP_PARAMS, }; enum AVActiveFormatDescription { diff --git a/libavutil/gain_map.c b/libavutil/gain_map.c new file mode 100644 index 0000000000..566c5a9df8 --- /dev/null +++ b/libavutil/gain_map.c @@ -0,0 +1,93 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "gain_map.h" +#include "mem.h" + +static void gain_map_params_default(AVGainMapParams *p) +{ + *p = (AVGainMapParams) { + .base_hdr_headroom = { 0, 1 }, + .alternate_hdr_headroom = { 0, 1 }, + .nb_channels = 1, + }; + + /* Pre-fill all three channels as a convenience */ + for (int c = 0; c < 3; c++) { + p->channels[c] = (struct AVGainMapChannel) { + .gain_map_min = { 0, 1 }, + .gain_map_max = { 0, 1 }, + .gamma = { 1, 1 }, + .base_offset = { 1, 64 }, + .alternate_offset = { 1, 64 }, + }; + } +} + +#define CHECK(cond) do { if (!(cond)) return AVERROR_INVALIDDATA; } while (0) +#define CHECK_SIGNED(q) CHECK((q).den > 0); +#define CHECK_UNSIGNED(q) CHECK((q).num >= 0 && (q).den > 0); + +int av_gain_map_params_validate(const AVGainMapParams *p) +{ + CHECK(0 <= p->version && p->version <= AV_ISO21496_VERSION); + CHECK(p->nb_channels == 1 || p->nb_channels == 3); + CHECK_UNSIGNED(p->base_hdr_headroom); + CHECK_UNSIGNED(p->alternate_hdr_headroom); + + for (int c = 0; c < p->nb_channels; c++) { + const struct AVGainMapChannel *const ch = &p->channels[c]; + CHECK_SIGNED(ch->gain_map_min); + CHECK_SIGNED(ch->gain_map_max); + CHECK_UNSIGNED(ch->gamma); + CHECK(ch->gamma.num != 0); + CHECK_SIGNED(ch->base_offset); + CHECK_SIGNED(ch->alternate_offset); + CHECK(av_cmp_q(ch->gain_map_min, ch->gain_map_max) <= 0); + } + + return 0; +} + +AVGainMapParams *av_gain_map_params_alloc(size_t *size) +{ + AVGainMapParams *p = av_malloc(sizeof(AVGainMapParams)); + if (!p) + return NULL; + + gain_map_params_default(p); + + if (size) + *size = sizeof(*p); + + return p; +} + +AVGainMapParams *av_gain_map_params_create_side_data(AVFrameSideData ***fsd, int *nb_sd) +{ + AVFrameSideData *sd; + sd = av_frame_side_data_new(fsd, nb_sd, AV_FRAME_DATA_GAIN_MAP_PARAMS, + sizeof(AVGainMapParams), + AV_FRAME_SIDE_DATA_FLAG_REPLACE); + if (!sd) + return NULL; + + AVGainMapParams *const p = (AVGainMapParams *) sd->data; + gain_map_params_default(p); + return p; +} diff --git a/libavutil/gain_map.h b/libavutil/gain_map.h new file mode 100644 index 0000000000..b05de4cba2 --- /dev/null +++ b/libavutil/gain_map.h @@ -0,0 +1,109 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVUTIL_GAIN_MAP_H +#define AVUTIL_GAIN_MAP_H + +#include <stddef.h> + +#include "rational.h" +#include "frame.h" + +#define AV_ISO21496_VERSION 0 +#define AV_ISO21496_IDENTIFIER "urn:iso:std:iso:ts:21496:-1" + +/** + * Parameters describing how to combine a gain map rendition with its base + * rendition to recover the alternate rendition, as defined by + * ISO 21496-1:2025 "Digital photography - Gain map metadata for image + * conversion". + * + * Note: sizeof(AVGainMapParams) is not part of the ABI. New fields may be + * added at the end of the struct. + */ +typedef struct AVGainMapParams { + /** + * The version of this gain map struct. Reserved to allow future + * extensions of the struct. Must be <= AV_ISO21496_VERSION. + */ + int version; + + /** + * If non-zero, the gain map is applied in the colour space of the base + * rendition; otherwise in that of the alternate rendition. + */ + int use_base_color_space; + + /** + * log2 of the display headroom at which the base rendition is shown + * unmodified, and at which the map is applied in full. + */ + AVRational base_hdr_headroom; + AVRational alternate_hdr_headroom; + + struct AVGainMapChannel { + /** + * log2-domain minimum and maximum gain applied by the map, i.e. the + * values the map's 0.0 and 1.0 endpoints decode to. + */ + AVRational gain_map_min; + AVRational gain_map_max; + + /** + * Encoding gamma of the stored map values. Must be > 0. + */ + AVRational gamma; + + /** + * Small constants added before the log-domain math to avoid a + * singularity at zero. + */ + AVRational base_offset; + AVRational alternate_offset; + } channels[3]; /* R, G, B */ + + /** + * Must be 1 or 3. If 1, all channels share the same set of parameters. + */ + int nb_channels; +} AVGainMapParams; + +/** + * Returns >= 0 if the given AVGainMapParams struct contains valid data; + * or a negative AVERROR otherwise. + */ +int av_gain_map_params_validate(const AVGainMapParams *p); + +/** + * Allocate an AVGainMapParams structure and initialize it to default values. + * The resulting pointer must be freed using av_free(). + * + * @param size if non-NULL, set to sizeof(AVGainMapParams) + * @return the newly allocated struct, or NULL on failure + */ +AVGainMapParams *av_gain_map_params_alloc(size_t *size); + +/** + * Allocate and add an AVGainMapParams structure to an existing AVFrameSideData + * array as AV_FRAME_DATA_GAIN_MAP_PARAMS side data. + * + * @return the newly allocated struct, or NULL on failure + */ +AVGainMapParams *av_gain_map_params_create_side_data(AVFrameSideData ***sd, int *nb_sd); + +#endif /* AVUTIL_GAIN_MAP_H */ diff --git a/libavutil/side_data.c b/libavutil/side_data.c index 0b4869c941..0d0289a9b6 100644 --- a/libavutil/side_data.c +++ b/libavutil/side_data.c @@ -63,6 +63,7 @@ static const AVSideDataDescriptor sd_props[] = { [AV_FRAME_DATA_IAMF_RECON_GAIN_INFO_PARAM] = { "IAMF Recon Gain Info Parameter Data" }, [AV_FRAME_DATA_RAW_COLOR_PARAMS] = { "RAW camera color parameters", AV_SIDE_DATA_PROP_GLOBAL | AV_SIDE_DATA_PROP_COLOR_DEPENDENT }, [AV_FRAME_DATA_DOWNMIX_MATRIX] = { "Downmix Matrix", AV_SIDE_DATA_PROP_CHANNEL_DEPENDENT }, + [AV_FRAME_DATA_GAIN_MAP_PARAMS] = { "Gain map parameters", AV_SIDE_DATA_PROP_GLOBAL | AV_SIDE_DATA_PROP_COLOR_DEPENDENT }, }; const AVSideDataDescriptor *av_frame_side_data_desc(enum AVFrameSideDataType type) diff --git a/libavutil/version.h b/libavutil/version.h index b065c39343..b83c74d755 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -79,7 +79,7 @@ */ #define LIBAVUTIL_VERSION_MAJOR 61 -#define LIBAVUTIL_VERSION_MINOR 6 +#define LIBAVUTIL_VERSION_MINOR 7 #define LIBAVUTIL_VERSION_MICRO 100 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \ -- 2.52.0 >From 188c93c27f49b5f3687e7a7e8a093d67f4d1448d Mon Sep 17 00:00:00 2001 From: Niklas Haas <[email protected]> Date: Tue, 25 Aug 2026 13:55:00 +0200 Subject: [PATCH 2/8] avformat: add AV_STREAM_GROUP_PARAMS_GAIN_MAP To identify ISO 21496-1 gain maps. Signed-off-by: Niklas Haas <[email protected]> --- doc/APIchanges | 3 +++ doc/ffmpeg.texi | 13 +++++++++++++ fftools/ffmpeg_mux_init.c | 3 +++ libavformat/avformat.c | 2 ++ libavformat/avformat.h | 6 ++++-- libavformat/options.c | 3 +++ libavformat/version.h | 2 +- 7 files changed, 29 insertions(+), 3 deletions(-) diff --git a/doc/APIchanges b/doc/APIchanges index 1b49f7183e..2b99b0d386 100644 --- a/doc/APIchanges +++ b/doc/APIchanges @@ -2,6 +2,9 @@ The last version increases of all libraries were on 2026-06-23. API changes, most recent first: +2026-08-xx - xxxxxxxxxx - lavf 63.7.100 - avformat.h + Add AV_STREAM_GROUP_PARAMS_GAIN_MAP. + 2026-08-17 - xxxxxxxxxx - lavu 61.7.100 - frame.h gain_map.h Add AV_FRAME_DATA_GAIN_MAP_PARAMS. Add AV_ISO21496_VERSION, AV_ISO21496_IDENTIFIER. diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi index 1cbb1ccbdd..4fdd375cf1 100644 --- a/doc/ffmpeg.texi +++ b/doc/ffmpeg.texi @@ -1226,6 +1226,19 @@ that specifies the language for the "value" string. "key" must be the same as th all sub-mix element's @var{annotations}s @end table +@item gain_map +Groups together a base @var{stream} with a gain map, which together form a +pair of renditions (e.g. HDR and SDR) as per ISO 21496-1. + +For this group @var{type}, the following options are available + +@table @option +@item el_index +Index of the gain map stream within the group. Defaults to 0. +@item video_size +Size of the final image for presentation. +@end table + @end table E.g. to create an scalable 5.1 IAMF file from several WAV input files diff --git a/fftools/ffmpeg_mux_init.c b/fftools/ffmpeg_mux_init.c index f80b4427b4..bb935fd324 100644 --- a/fftools/ffmpeg_mux_init.c +++ b/fftools/ffmpeg_mux_init.c @@ -2506,6 +2506,7 @@ static int of_map_group(Muxer *mux, AVDictionary **dict, AVBPrint *bp, const cha case AV_STREAM_GROUP_PARAMS_LCEVC: case AV_STREAM_GROUP_PARAMS_TREF: case AV_STREAM_GROUP_PARAMS_DOLBY_VISION: + case AV_STREAM_GROUP_PARAMS_GAIN_MAP: break; default: av_log(mux, AV_LOG_ERROR, "Unsupported mapped group type %d.\n", stg->type); @@ -2533,6 +2534,8 @@ static int of_parse_group_token(Muxer *mux, const char *token, char *ptr) { .i64 = AV_STREAM_GROUP_PARAMS_LCEVC }, .unit = "type" }, { "tref", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = AV_STREAM_GROUP_PARAMS_TREF }, .unit = "type" }, + { "gain_map", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = AV_STREAM_GROUP_PARAMS_GAIN_MAP }, .unit = "type" }, { NULL }, }; const AVClass class = { diff --git a/libavformat/avformat.c b/libavformat/avformat.c index ae12f00975..3e39a7eb65 100644 --- a/libavformat/avformat.c +++ b/libavformat/avformat.c @@ -109,6 +109,7 @@ void ff_free_stream_group(AVStreamGroup **pstg) break; case AV_STREAM_GROUP_PARAMS_LCEVC: case AV_STREAM_GROUP_PARAMS_DOLBY_VISION: + case AV_STREAM_GROUP_PARAMS_GAIN_MAP: av_opt_free(stg->params.layered_video); av_freep(&stg->params.layered_video); break; @@ -275,6 +276,7 @@ const char *avformat_stream_group_name(enum AVStreamGroupParamsType type) case AV_STREAM_GROUP_PARAMS_LCEVC: return "LCEVC (Split video and enhancement)"; case AV_STREAM_GROUP_PARAMS_TREF: return "Track Reference"; case AV_STREAM_GROUP_PARAMS_DOLBY_VISION: return "Dolby Vision (Split base and enhancement layer)"; + case AV_STREAM_GROUP_PARAMS_GAIN_MAP: return "Gain Map (Split base rendition and gain map)"; } return NULL; } diff --git a/libavformat/avformat.h b/libavformat/avformat.h index 9df6459f91..39b76369d7 100644 --- a/libavformat/avformat.h +++ b/libavformat/avformat.h @@ -1087,8 +1087,9 @@ typedef struct AVStreamGroupTileGrid { * AVStreamGroupLayeredVideo is meant to define the relation between a base * layer video stream and a separate enhancement layer stream that together * form a single layered video presentation (for example a video stream and a - * data stream containing LCEVC enhancement layer NALUs, or Dolby Vision - * Profile 7 dual-layer encoding). + * data stream containing LCEVC enhancement layer NALUs, Dolby Vision + * Profile 7 dual-layer encoding, or a base rendition accompanied by an + * ISO 21496-1 gain map). * * The enhancement layer stream is identified by @ref el_index. */ @@ -1153,6 +1154,7 @@ enum AVStreamGroupParamsType { AV_STREAM_GROUP_PARAMS_LCEVC, AV_STREAM_GROUP_PARAMS_TREF, AV_STREAM_GROUP_PARAMS_DOLBY_VISION, + AV_STREAM_GROUP_PARAMS_GAIN_MAP, }; struct AVIAMFAudioElement; diff --git a/libavformat/options.c b/libavformat/options.c index 54dafdf5bf..042fe494cf 100644 --- a/libavformat/options.c +++ b/libavformat/options.c @@ -403,6 +403,7 @@ static void *stream_group_child_next(void *obj, void *prev) return stg->params.tref; case AV_STREAM_GROUP_PARAMS_LCEVC: case AV_STREAM_GROUP_PARAMS_DOLBY_VISION: + case AV_STREAM_GROUP_PARAMS_GAIN_MAP: return stg->params.layered_video; default: break; @@ -436,6 +437,7 @@ static const AVClass *stream_group_child_iterate(void **opaque) break; case AV_STREAM_GROUP_PARAMS_LCEVC: case AV_STREAM_GROUP_PARAMS_DOLBY_VISION: + case AV_STREAM_GROUP_PARAMS_GAIN_MAP: ret = &layered_video_class; break; default: @@ -516,6 +518,7 @@ AVStreamGroup *avformat_stream_group_create(AVFormatContext *s, break; case AV_STREAM_GROUP_PARAMS_LCEVC: case AV_STREAM_GROUP_PARAMS_DOLBY_VISION: + case AV_STREAM_GROUP_PARAMS_GAIN_MAP: stg->params.layered_video = av_mallocz(sizeof(*stg->params.layered_video)); if (!stg->params.layered_video) goto fail; diff --git a/libavformat/version.h b/libavformat/version.h index 4bde82abb4..70c554c19c 100644 --- a/libavformat/version.h +++ b/libavformat/version.h @@ -31,7 +31,7 @@ #include "version_major.h" -#define LIBAVFORMAT_VERSION_MINOR 6 +#define LIBAVFORMAT_VERSION_MINOR 7 #define LIBAVFORMAT_VERSION_MICRO 100 #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \ -- 2.52.0 >From 1657b469b7651186dfb3b59512efb6d285c47a2e Mon Sep 17 00:00:00 2001 From: Niklas Haas <[email protected]> Date: Tue, 18 Aug 2026 15:04:30 +0200 Subject: [PATCH 3/8] avfilter: add AV_FRAME_DATA_GAIN_MAP_PARAMS support To both f_sidedata and vf_showinfo. Useful for debugging. Signed-off-by: Niklas Haas <[email protected]> --- doc/filters.texi | 1 + libavfilter/f_sidedata.c | 1 + libavfilter/vf_showinfo.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 32 insertions(+) diff --git a/doc/filters.texi b/doc/filters.texi index 8425382225..1f8e924415 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -34227,6 +34227,7 @@ Possible values are: @item DYNAMIC_HDR_VIVID @item AMBIENT_VIEWING_ENVIRONMENT @item VIDEO_HINT +@item GAIN_MAP_PARAMS @end table @end table diff --git a/libavfilter/f_sidedata.c b/libavfilter/f_sidedata.c index 6c81c6b6a5..eb0301c5ce 100644 --- a/libavfilter/f_sidedata.c +++ b/libavfilter/f_sidedata.c @@ -80,6 +80,7 @@ static const AVOption filt_name##_options[] = { \ { "DYNAMIC_HDR_VIVID", "", 0, AV_OPT_TYPE_CONST, {.i64 = AV_FRAME_DATA_DYNAMIC_HDR_VIVID }, 0, 0, FLAGS, .unit = "type" }, \ { "AMBIENT_VIEWING_ENVIRONMENT","", 0, AV_OPT_TYPE_CONST, {.i64 = AV_FRAME_DATA_AMBIENT_VIEWING_ENVIRONMENT}, 0, 0, FLAGS, .unit = "type" }, \ { "VIDEO_HINT", "", 0, AV_OPT_TYPE_CONST, {.i64 = AV_FRAME_DATA_VIDEO_HINT }, 0, 0, FLAGS, .unit = "type" }, \ + { "GAIN_MAP_PARAMS", "", 0, AV_OPT_TYPE_CONST, {.i64 = AV_FRAME_DATA_GAIN_MAP_PARAMS }, 0, 0, FLAGS, .unit = "type" }, \ { NULL } \ } diff --git a/libavfilter/vf_showinfo.c b/libavfilter/vf_showinfo.c index 4ac3d45dc0..21cde369f5 100644 --- a/libavfilter/vf_showinfo.c +++ b/libavfilter/vf_showinfo.c @@ -32,6 +32,7 @@ #include "libavutil/imgutils.h" #include "libavutil/internal.h" #include "libavutil/film_grain_params.h" +#include "libavutil/gain_map.h" #include "libavutil/hdr_dynamic_metadata.h" #include "libavutil/hdr_dynamic_vivid_metadata.h" #include "libavutil/opt.h" @@ -655,6 +656,32 @@ static void dump_dovi_metadata(AVFilterContext *ctx, const AVFrameSideData *sd) av_log(ctx, AV_LOG_INFO, "source_diagonal=%"PRIu16"; ", color->source_diagonal); } +static void dump_gain_map_params(AVFilterContext *ctx, const AVFrameSideData *sd) +{ + const AVGainMapParams *p = (const AVGainMapParams *) sd->data; + av_log(ctx, AV_LOG_INFO, "version=%d, nb_channels=%d, use_base_color_space=%d, " + "base_hdr_headroom=%f, alternate_hdr_headroom=%f, channels={ ", + p->version, p->nb_channels, p->use_base_color_space, + av_q2d(p->base_hdr_headroom), av_q2d(p->alternate_hdr_headroom)); + + if (!(p->nb_channels == 1 || p->nb_channels == 3)) { + av_log(ctx, AV_LOG_ERROR, "invalid data }"); + return; + } + + for (int c = 0; c < p->nb_channels; c++) { + const struct AVGainMapChannel *ch = &p->channels[c]; + av_log(ctx, AV_LOG_INFO, + "{ gain_map_min=%f, gain_map_max=%f, gamma=%f, " + "base_offset=%f, alternate_offset=%f } ", + av_q2d(ch->gain_map_min), av_q2d(ch->gain_map_max), + av_q2d(ch->gamma), av_q2d(ch->base_offset), + av_q2d(ch->alternate_offset)); + } + + av_log(ctx, AV_LOG_INFO, "}"); +} + static void dump_ambient_viewing_environment(AVFilterContext *ctx, const AVFrameSideData *sd) { const AVAmbientViewingEnvironment *ambient_viewing_environment = @@ -865,6 +892,9 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame) case AV_FRAME_DATA_AMBIENT_VIEWING_ENVIRONMENT: dump_ambient_viewing_environment(ctx, sd); break; + case AV_FRAME_DATA_GAIN_MAP_PARAMS: + dump_gain_map_params(ctx, sd); + break; case AV_FRAME_DATA_VIEW_ID: av_log(ctx, AV_LOG_INFO, "view id: %d\n", *(int*)sd->data); break; -- 2.52.0 >From 250ec1be8f1e23af2d7d072a8a728e1a71840c71 Mon Sep 17 00:00:00 2001 From: Niklas Haas <[email protected]> Date: Wed, 19 Aug 2026 13:28:47 +0200 Subject: [PATCH 4/8] avfilter/vf_gainmap: add filter to compute a gain map This is a mostly straightforward translation of the logic from ISO 21496. Implementation notes: - Slice threading is used to vastly speed up processing - We quantize the EOTF to a LUT if possible; this doesn't work for HLG becaus HLG embeds a channel-dependent OOTF. It's possible we could implement a custom HLG function here in the future, but for now, simply calling out to the EOTF function is a reasonable fallback. - Similar to libultrahdr, we use a two-pass approach when the gain range is unknown a priori, and fold the quantization step into a single pass if all parameters are specified up-front. - Unlike libultrahdr, we make an effort to also pick an optimal gamma to tune the quantized encoding. Users can always disable this by specifying `:gamma=1.0` explicitly. - This filter is bidirectional and can also compute an HDR->SDR gainmap, so the naming is left deliberately abstract internally. Signed-off-by: Niklas Haas <[email protected]> --- doc/filters.texi | 74 +++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_gainmap.c | 751 +++++++++++++++++++++++++++ tests/fate/filter-video.mak | 8 + tests/ref/fate/filter-gainmap | 6 + tests/ref/fate/filter-gainmap-fixed | 6 + tests/ref/fate/filter-gainmap-luma | 6 + tests/ref/fate/filter-gainmap-maxrgb | 6 + 9 files changed, 859 insertions(+) create mode 100644 libavfilter/vf_gainmap.c create mode 100644 tests/ref/fate/filter-gainmap create mode 100644 tests/ref/fate/filter-gainmap-fixed create mode 100644 tests/ref/fate/filter-gainmap-luma create mode 100644 tests/ref/fate/filter-gainmap-maxrgb diff --git a/doc/filters.texi b/doc/filters.texi index 1f8e924415..ae6248c4b7 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -15165,6 +15165,80 @@ sort -n MAP_FILE ffmpeg -i INPUT -i OUTPUT -filter_complex '[0:v]fsync=file=MAP_FILE[ref];[1:v][ref]ssim' -f null - @end example +@section gainmap + +Compute an HDR gain map from two renditions of the same image, as defined by +ISO 21496-1. + +The first input is the base rendition (typically SDR), and the second input +is the alternate rendition (typically HDR). Both must have the same +dimensions. Outputs a gain map, as either @samp{gbrpf32} or @samp{grayf32} +depending on the chosen @option{mode}. + +@table @option +@item mode +Controls what quantity the gain map is computed against. + +The accepted values are: +@table @samp +@item rgb +One gain map channel per component. + +@item luma +A single gain map channel, computed from the luma. + +@item maxrgb +A single gain map channel, computed from @samp{max(R, G, B)}. +@end table + +Default is @samp{rgb}. + +@item colorspace +Chooses which colorspace to compute the gain map in. + +The accepted values are: +@table @samp +@item base +The base rendition's colorspace. + +@item alternate +The alternate rendition's colorspace. +@end table + +Default is @samp{base}. + +@item min +@item max +Bounds on the encoded gain (log2). If unspecified, these will be +measured from the frame. + +@item gamma +Encoding gamma of the resulting gain map. If unspecified, this will be +optimized for the frame in question. Setting this to @code{1.0} will produce +a linear gain map, which may be more efficient in some applications. + +@item base_offset +@item alt_offset +Offset to prevent numerical instability near zero. Both default to @code{1/64}. + +@item base_nits +@item alt_nits +Override the reference luminance of the input, in nits. If unspecified, uses +the values from the input metadata. +@end table + +The @code{gainmap} filter also supports the @ref{framesync} options. + +@subsection Examples + +@itemize +@item +Compute a gain map between an SDR and an HDR version of the same image: +@example +ffmpeg -i $sdr -i $hdr -filter_complex "[0:v][1:v]gainmap" $out +@end example +@end itemize + @section gblur Apply Gaussian blur filter. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index daa4d552f0..40b5223602 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -332,6 +332,7 @@ OBJS-$(CONFIG_FREEZEFRAMES_FILTER) += vf_freezeframes.o OBJS-$(CONFIG_FREI0R_FILTER) += vf_frei0r.o OBJS-$(CONFIG_FSPP_FILTER) += vf_fspp.o vf_fsppdsp.o qp_table.o OBJS-$(CONFIG_FSYNC_FILTER) += vf_fsync.o +OBJS-$(CONFIG_GAINMAP_FILTER) += vf_gainmap.o colorspace.o framesync.o OBJS-$(CONFIG_GBLUR_FILTER) += vf_gblur.o OBJS-$(CONFIG_GBLUR_VULKAN_FILTER) += vf_gblur_vulkan.o vulkan.o vulkan_filter.o OBJS-$(CONFIG_GEQ_FILTER) += vf_geq.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 4af7a3bbbf..320689cb54 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -306,6 +306,7 @@ extern const FFFilter ff_vf_freezeframes; extern const FFFilter ff_vf_frei0r; extern const FFFilter ff_vf_fspp; extern const FFFilter ff_vf_fsync; +extern const FFFilter ff_vf_gainmap; extern const FFFilter ff_vf_gblur; extern const FFFilter ff_vf_gblur_vulkan; extern const FFFilter ff_vf_geq; diff --git a/libavfilter/vf_gainmap.c b/libavfilter/vf_gainmap.c new file mode 100644 index 0000000000..880c069c6b --- /dev/null +++ b/libavfilter/vf_gainmap.c @@ -0,0 +1,751 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Filter to compute an HDR gain map from a base and alternate rendition. + */ + +#include <float.h> +#include <math.h> + +#include "libavutil/csp.h" +#include "libavutil/gain_map.h" +#include "libavutil/internal.h" +#include "libavutil/mastering_display_metadata.h" +#include "libavutil/mem.h" +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" + +#include "avfilter.h" +#include "colorspace.h" +#include "filters.h" +#include "formats.h" +#include "framesync.h" +#include "video.h" + +#define SDR_DIFFUSE_WHITE 203.0 + +enum GainMapColorspace { + GAINMAP_CSP_BASE, ///< apply the map in the base rendition's space + GAINMAP_CSP_ALT, ///< apply it in the alternate rendition's space + GAINMAP_CSP_NB, +}; + +enum GainMapMode { + GAINMAP_RGB, ///< one channel per component + GAINMAP_LUMA, ///< single channel, from the luminance + GAINMAP_MAXRGB, ///< single channel, from max(R,G,B) + GAINMAP_MODE_NB, +}; + +enum GainMapMeasure { + MEASURE_MIN = 1 << 0, + MEASURE_MAX = 1 << 1, + MEASURE_GAMMA = 1 << 2, + MEASURE_RANGE = MEASURE_MIN | MEASURE_MAX, + MEASURE_ALL = MEASURE_RANGE | MEASURE_GAMMA, +}; + +typedef struct GainMapEOTF { + enum AVColorTransferCharacteristic trc; + av_csp_eotf_function eotf; /* or NULL */ + double Lw; + +#define GAINMAP_LUT_SIZE 1025 + float lut[GAINMAP_LUT_SIZE + 1]; /* extra padding entry */ +} GainMapEOTF; + +typedef struct GainMapContext { + const AVClass *class; + FFFrameSync fs; + + /* Filter options */ + int mode; + int colorspace; + float gain_min; + float gain_max; + float gamma; + float base_offset; + float alt_offset; + float base_nits; + float alt_nits; + + int warned_noop; + int nb_channels; + int nb_threads; /* for slice threading */ + int convert; /* colorspace conversion needed */ + int sign; /* sign(alt_peak - base_peak) */ + + /* Colorspace parameters */ + GainMapEOTF base_eotf; + GainMapEOTF alt_eotf; + float rgb2y[3]; + float rgb2rgb[3][3]; + + /* Quantization parameters, either static or recomputed dynamically */ + float quant_scale[3]; + float quant_offset[3]; + float quant_gamma[3]; + + /* Measured frame statistics */ + float (*slice_min)[3]; /* for MEASURE_MIN */ + float (*slice_max)[3]; /* for MEASURE_MAX */ + double (*slice_sum)[3]; /* for MEASURE_GAMMA */ + enum GainMapMeasure measure; + + /* Generated gain map parameters (recomputed per frame) */ + AVGainMapParams params; +} GainMapContext; + +typedef struct ThreadData { + AVFrame *out; + const AVFrame *base, *alt; +} ThreadData; + +#define OFFSET(x) offsetof(GainMapContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM + +static const AVOption gainmap_options[] = { + { "mode", "quantity the gain is computed over", OFFSET(mode), AV_OPT_TYPE_INT, { .i64 = GAINMAP_RGB }, 0, GAINMAP_MODE_NB - 1, FLAGS, .unit = "mode" }, + { "rgb", "one gain channel per component", 0, AV_OPT_TYPE_CONST, { .i64 = GAINMAP_RGB }, 0, 0, FLAGS, .unit = "mode" }, + { "luma", "single gain channel, from luminance", 0, AV_OPT_TYPE_CONST, { .i64 = GAINMAP_LUMA }, 0, 0, FLAGS, .unit = "mode" }, + { "maxrgb", "single gain channel, from max(R,G,B)", 0, AV_OPT_TYPE_CONST, { .i64 = GAINMAP_MAXRGB }, 0, 0, FLAGS, .unit = "mode" }, + { "colorspace", "rendition whose colour space the map is applied in", OFFSET(colorspace), AV_OPT_TYPE_INT, { .i64 = GAINMAP_CSP_BASE }, 0, GAINMAP_CSP_NB - 1, FLAGS, .unit = "colorspace" }, + { "base", "the base rendition's colour space", 0, AV_OPT_TYPE_CONST, { .i64 = GAINMAP_CSP_BASE }, 0, 0, FLAGS, .unit = "colorspace" }, + { "alternate", "the alternate rendition's colour space", 0, AV_OPT_TYPE_CONST, { .i64 = GAINMAP_CSP_ALT }, 0, 0, FLAGS, .unit = "colorspace" }, + { "min", "override lower bound on the encoded gain, in log2 space", OFFSET(gain_min), AV_OPT_TYPE_FLOAT, { .dbl = NAN }, -32.0, 32.0, FLAGS }, + { "max", "override upper bound on the encoded gain, in log2 space", OFFSET(gain_max), AV_OPT_TYPE_FLOAT, { .dbl = NAN }, -32.0, 32.0, FLAGS }, + { "gamma", "override encoding gamma of the stored map (default: measured)", OFFSET(gamma), AV_OPT_TYPE_FLOAT, { .dbl = NAN }, 0.0001, 100.0, FLAGS }, + { "base_offset", "constant added to the base rendition", OFFSET(base_offset), AV_OPT_TYPE_FLOAT, { .dbl = 1.0 / 64.0 }, 1e-6, 1.0, FLAGS }, + { "alt_offset", "constant added to the alternate rendition", OFFSET(alt_offset), AV_OPT_TYPE_FLOAT, { .dbl = 1.0 / 64.0 }, 1e-6, 1.0, FLAGS }, + { "base_nits", "override input luminance of the base rendition", OFFSET(base_nits), AV_OPT_TYPE_FLOAT, { .dbl = NAN }, 1.0, 10000.0, FLAGS }, + { "alt_nits", "override input luminance of the alternate rendition", OFFSET(alt_nits), AV_OPT_TYPE_FLOAT, { .dbl = NAN }, 1.0, 10000.0, FLAGS }, + { NULL } +}; + +FRAMESYNC_DEFINE_CLASS(gainmap, GainMapContext, fs); + +static void setup_range(AVFilterContext *ctx, int ch, float min, float max) +{ + GainMapContext *s = ctx->priv; + if (max <= min) { + /* Nothing to quantize; flat gain map */ + s->quant_scale[ch] = s->quant_offset[ch] = 0.0f; + max = min; /* sanity */ + } else { + s->quant_scale[ch] = 1.0f / (max - min); + s->quant_offset[ch] = -min * s->quant_scale[ch]; + } + + s->params.channels[ch].gain_map_min = av_d2q(min, INT_MAX); + s->params.channels[ch].gain_map_max = av_d2q(max, INT_MAX); + av_log(ctx, AV_LOG_TRACE, "channel %d: measured min=%g max=%g\n", ch, min, max); +} + +static void setup_gamma(AVFilterContext *ctx, int ch, float gamma) +{ + GainMapContext *s = ctx->priv; + s->quant_gamma[ch] = gamma; + s->params.channels[ch].gamma = av_d2q(gamma, INT_MAX); + av_log(ctx, AV_LOG_TRACE, "channel %d: measured gamma=%g\n", ch, gamma); +} + +static av_cold int init(AVFilterContext *ctx) +{ + GainMapContext *s = ctx->priv; + + if (s->gain_min >= s->gain_max) { + av_log(ctx, AV_LOG_ERROR, "min (%g) must be below max (%g)\n", + s->gain_min, s->gain_max); + return AVERROR(EINVAL); + } + + s->nb_channels = s->mode == GAINMAP_RGB ? 3 : 1; + if (isnan(s->gain_min)) + s->measure |= MEASURE_MIN; + if (isnan(s->gain_max)) + s->measure |= MEASURE_MAX; + if (isnan(s->gamma)) + s->measure |= MEASURE_GAMMA; + if (s->measure) { + av_log(ctx, AV_LOG_VERBOSE, "Using two passes to measure gain map " + "parameters from the input frame.\n"); + } + + s->params = (AVGainMapParams) { + .version = 0, + .nb_channels = s->nb_channels, + .use_base_color_space = s->colorspace == GAINMAP_CSP_BASE, + /* payload metadata recomputed per frame */ + }; + + for (int c = 0; c < s->nb_channels; c++) { + struct AVGainMapChannel *ch = &s->params.channels[c]; + ch->base_offset = av_d2q(s->base_offset, INT_MAX); + ch->alternate_offset = av_d2q(s->alt_offset, INT_MAX); + if (!(s->measure & MEASURE_RANGE)) + setup_range(ctx, c, s->gain_min, s->gain_max); + if (!(s->measure & MEASURE_GAMMA)) + setup_gamma(ctx, c, s->gamma); + } + + return 0; +} + +static int query_formats(const AVFilterContext *ctx, + AVFilterFormatsConfig **cfg_in, + AVFilterFormatsConfig **cfg_out) +{ + const GainMapContext *s = ctx->priv; + enum AVPixelFormat in_fmt, out_fmt; + int ret; + + in_fmt = AV_PIX_FMT_GBRPF32; + out_fmt = s->nb_channels == 1 ? AV_PIX_FMT_GRAYF32 : AV_PIX_FMT_GBRPF32; + + ret = ff_formats_ref(ff_make_formats_list_singleton(out_fmt), &cfg_out[0]->formats); + if (ret < 0) + return ret; + + return ff_set_common_formats2(ctx, cfg_in, cfg_out, + ff_make_formats_list_singleton(in_fmt)); +} + +static double frame_luminance(const AVFrame *frame) +{ + const AVFrameSideData *sd; + sd = av_frame_get_side_data(frame, AV_FRAME_DATA_MASTERING_DISPLAY_METADATA); + if (sd) { + const AVMasteringDisplayMetadata *mdm; + mdm = (const AVMasteringDisplayMetadata *) sd->data; + if (mdm->has_luminance && mdm->max_luminance.num > 0) + return av_q2d(mdm->max_luminance); + } + + switch (frame->color_trc) { + case AVCOL_TRC_SMPTE2084: return 10000.0; + case AVCOL_TRC_ARIB_STD_B67: return 1000.0; + default: return SDR_DIFFUSE_WHITE; + } +} + +static void update_eotf(GainMapEOTF *tf, enum AVColorTransferCharacteristic trc, + av_csp_eotf_function eotf, double Lw) +{ + if (trc == tf->trc && Lw == tf->Lw) + return; /* no change */ + tf->trc = trc; + tf->Lw = Lw; + + switch (trc) { + /* EOTFs that are safe to collapse into a 1D LUT */ + case AVCOL_TRC_BT709: + case AVCOL_TRC_GAMMA22: + case AVCOL_TRC_GAMMA28: + case AVCOL_TRC_SMPTE170M: + case AVCOL_TRC_SMPTE240M: + case AVCOL_TRC_LINEAR: + case AVCOL_TRC_IEC61966_2_4: + case AVCOL_TRC_BT1361_ECG: + case AVCOL_TRC_IEC61966_2_1: + case AVCOL_TRC_BT2020_10: + case AVCOL_TRC_BT2020_12: + case AVCOL_TRC_SMPTE2084: { + for (int i = 0; i < GAINMAP_LUT_SIZE; i++) { + double x[3] = { i / (GAINMAP_LUT_SIZE - 1.0) }; + eotf(Lw, 0.0, x); + tf->lut[i] = x[0] / SDR_DIFFUSE_WHITE; /* normalize */ + } + + tf->lut[GAINMAP_LUT_SIZE] = tf->lut[GAINMAP_LUT_SIZE - 1]; + tf->eotf = NULL; + break; + } + /* EOTFs that use the av_csp_eotf_function fallback */ + case AVCOL_TRC_ARIB_STD_B67: /* nontrivial OOTF */ + case AVCOL_TRC_SMPTE428: /* different normalization per channel */ + default: + tf->eotf = eotf; + break; + } +} + +static const char *unknown_if_null(const char *s) +{ + return s ? s : "unknown"; +} + +/* Run per frame, since trc/primaries are not (yet) link-level properties */ +static int setup_colorspace(AVFilterContext *ctx, const AVFrame *base, const AVFrame *alt) +{ + GainMapContext *const s = ctx->priv; + av_csp_eotf_function base_eotf = av_csp_itu_eotf(base->color_trc); + av_csp_eotf_function alt_eotf = av_csp_itu_eotf(alt->color_trc); + if (!base_eotf || !alt_eotf) { + av_log(ctx, AV_LOG_ERROR, "Unknown transfer: base=%s, alternate=%s\n", + unknown_if_null(av_color_transfer_name(base->color_trc)), + unknown_if_null(av_color_transfer_name(alt->color_trc))); + return AVERROR(EINVAL); + } + + const AVColorPrimariesDesc *base_desc, *alt_desc; + base_desc = av_csp_primaries_desc_from_id(base->color_primaries); + alt_desc = av_csp_primaries_desc_from_id(alt->color_primaries); + if (!base_desc || !alt_desc) { + av_log(ctx, AV_LOG_ERROR, "Unknown primaries: base=%s, alternate=%s\n", + unknown_if_null(av_color_primaries_name(base->color_primaries)), + unknown_if_null(av_color_primaries_name(alt->color_primaries))); + return AVERROR(EINVAL); + } + + double base_lw = isnan(s->base_nits) ? frame_luminance(base) : s->base_nits; + double alt_lw = isnan(s->alt_nits) ? frame_luminance(alt) : s->alt_nits; + double base_headroom = fmax(log2(base_lw / SDR_DIFFUSE_WHITE), 0.0); + double alt_headroom = fmax(log2(alt_lw / SDR_DIFFUSE_WHITE), 0.0); + s->params.base_hdr_headroom = av_d2q(base_headroom, INT_MAX); + s->params.alternate_hdr_headroom = av_d2q(alt_headroom, INT_MAX); + s->sign = FFDIFFSIGN(alt_headroom, base_headroom); + if (!s->sign) { + /* No-op, set up empty gain map */ + for (int i = 0; i < s->nb_channels; i++) { + setup_range(ctx, i, 0.0f, 0.0f); + setup_gamma(ctx, i, 1.0f); + } + + av_log_once(ctx, AV_LOG_WARNING, AV_LOG_VERBOSE, &s->warned_noop, + "Base and alternate renditions have the same peak " + "luminance (%g nits), gain map will be empty\n", base_lw); + return 0; + } else { + av_log(ctx, AV_LOG_DEBUG, "Base peak: %g nits, alternate peak: %g nits\n", + base_lw, alt_lw); + } + + update_eotf(&s->base_eotf, base->color_trc, base_eotf, base_lw); + update_eotf(&s->alt_eotf, alt->color_trc, alt_eotf, alt_lw); + + /* Assume conversion from alt to base */ + if (s->colorspace == GAINMAP_CSP_ALT) + FFSWAP(const AVColorPrimariesDesc *, base_desc, alt_desc); + + double rgb2xyz[3][3]; + ff_fill_rgb2xyz_table(&base_desc->prim, &base_desc->wp, rgb2xyz); + for (int i = 0; i < 3; i++) + s->rgb2y[i] = rgb2xyz[1][i] / av_q2d(base_desc->wp.y); + + s->convert = base->color_primaries != alt->color_primaries; + if (s->convert) { + /* Note: Ignores whitepoint differences (chromatic adapatation) */ + double xyz2rgb[3][3], rgb2rgb[3][3]; + ff_fill_rgb2xyz_table(&base_desc->prim, &base_desc->wp, rgb2xyz); + ff_matrix_invert_3x3(rgb2xyz, xyz2rgb); + ff_fill_rgb2xyz_table(&alt_desc->prim, &alt_desc->wp, rgb2xyz); + ff_matrix_mul_3x3(rgb2rgb, rgb2xyz, xyz2rgb); + for (int i = 0; i < 3; i++) + for (int j = 0; j < 3; j++) + s->rgb2rgb[i][j] = (float) rgb2rgb[i][j]; + } else { + memset(s->rgb2rgb, 0, sizeof(s->rgb2rgb)); + for (int i = 0; i < 3; i++) + s->rgb2rgb[i][i] = 1.0f; + } + + return 0; +} + +static av_always_inline float quantize(float gain, float scale, float offset, float gamma) +{ + const float norm = av_clipf(scale * gain + offset, 0.0f, 1.0f); + return gamma == 1.0f ? norm : powf(norm, gamma); +} + +static av_always_inline void +get_pixel(const GainMapContext *s, float dst[3], + const float *restrict const src[3], int x, int alt) +{ + const GainMapEOTF *const tf = alt ? &s->alt_eotf : &s->base_eotf; + float rgb[3]; + + /* Linearize to normalized RGB */ + if (tf->eotf) { + double rgbd[3] = { src[0][x], src[1][x], src[2][x] }; + tf->eotf(tf->Lw, 0.0, rgbd); + for (int i = 0; i < 3; i++) + rgb[i] = rgbd[i] / SDR_DIFFUSE_WHITE; + } else { + for (int i = 0; i < 3; i++) { + const float fx = av_clipf(src[i][x], 0.0f, 1.0f) * (GAINMAP_LUT_SIZE - 1.0); + const int ix = (int) fx; + const float lo = tf->lut[ix]; + const float hi = tf->lut[ix + 1]; + rgb[i] = lo + (fx - ix) * (hi - lo); + } + } + + /* Convert to correct colorspace if needed */ + if (s->convert && (s->colorspace == GAINMAP_CSP_ALT) != alt) { + const float r = rgb[0], g = rgb[1], b = rgb[2]; + for (int i = 0; i < 3; i++) { + dst[i] = fmaxf(r * s->rgb2rgb[i][0] + + g * s->rgb2rgb[i][1] + + b * s->rgb2rgb[i][2], 0.0f); + } + } else { + for (int i = 0; i < 3; i++) + dst[i] = fmaxf(rgb[i], 0.0f); + } +} + +/* GBRP plane order */ +static const int gbr_order[3] = { 2, 0, 1 }; + +static av_always_inline int +slice_internal(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs, int quant) +{ + GainMapContext *s = ctx->priv; + const ThreadData *td = arg; + const AVFrame *base = td->base, *alt = td->alt, *out = td->out; + + const float *restrict scale = s->quant_scale; + const float *restrict offset = s->quant_offset; + const float *restrict gamma = s->quant_gamma; + + /* Normalize both to the base colorspace */ + const float base_off = s->base_offset; + const float alt_off = s->alt_offset; + const float sign = s->sign; + + const int y_start = ff_slice_pos(out->height, jobnr, nb_jobs); + const int y_end = ff_slice_pos(out->height, jobnr + 1, nb_jobs); + const int width = out->width; + const int mode = s->mode; + const int nb_ch = s->nb_channels; + + float min[3] = { FLT_MAX, FLT_MAX, FLT_MAX }; + float max[3] = { -FLT_MAX, -FLT_MAX, -FLT_MAX }; + double sum[3] = { 0.0, 0.0, 0.0 }; + + for (int y = y_start; y < y_end; y++) { + const float *restrict b_row[3], *restrict a_row[3]; + for (int i = 0; i < 3; i++) { + const int p = gbr_order[i]; + b_row[i] = (const float *) (base->data[p] + y * base->linesize[p]); + a_row[i] = (const float *) ( alt->data[p] + y * alt->linesize[p]); + } + + float *restrict out_row[3]; + for (int i = 0; i < nb_ch; i++) { + const int p = nb_ch == 1 ? 0 : gbr_order[i]; + out_row[i] = (float *) (out->data[p] + y * out->linesize[p]); + } + + for (int x = 0; x < width; x++) { + float b[3], a[3], gain[3]; + get_pixel(s, b, b_row, x, 0); + get_pixel(s, a, a_row, x, 1); + + #define GAIN(a, b) (sign * log2f(((a) + alt_off) / ((b) + base_off))) + switch (mode) { + case GAINMAP_MAXRGB: { + const float max_a = fmaxf(fmaxf(a[0], a[1]), a[2]); + const float max_b = fmaxf(fmaxf(b[0], b[1]), b[2]); + gain[0] = GAIN(max_a, max_b); + break; + } + case GAINMAP_LUMA: { + const float y_a = s->rgb2y[0] * a[0] + s->rgb2y[1] * a[1] + s->rgb2y[2] * a[2]; + const float y_b = s->rgb2y[0] * b[0] + s->rgb2y[1] * b[1] + s->rgb2y[2] * b[2]; + gain[0] = GAIN(y_a, y_b); + break; + } + case GAINMAP_RGB: + gain[0] = GAIN(a[0], b[0]); + gain[1] = GAIN(a[1], b[1]); + gain[2] = GAIN(a[2], b[2]); + break; + } + #undef GAIN + + if (quant) { + for (int i = 0; i < nb_ch; i++) + out_row[i][x] = quantize(gain[i], scale[i], offset[i], gamma[i]); + } else { + for (int i = 0; i < nb_ch; i++) { + out_row[i][x] = gain[i]; + min[i] = fminf(min[i], gain[i]); + max[i] = fmaxf(max[i], gain[i]); + sum[i] += gain[i]; + } + } + + } + } + + for (int i = 0; !quant && i < 3; i++) { + s->slice_min[jobnr][i] = min[i]; + s->slice_max[jobnr][i] = max[i]; + s->slice_sum[jobnr][i] = sum[i]; + } + + return 0; +} + +static int slice_gain(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) +{ + return slice_internal(ctx, arg, jobnr, nb_jobs, 0); +} + +static int slice_gain_quant(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) +{ + return slice_internal(ctx, arg, jobnr, nb_jobs, 1); +} + +static int slice_quant(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) +{ + const ThreadData *td = arg; + const GainMapContext *s = ctx->priv; + const AVFrame *out = td->out; + + const float *restrict scale = s->quant_scale; + const float *restrict offset = s->quant_offset; + const float *restrict gamma = s->quant_gamma; + + const int y_start = ff_slice_pos(out->height, jobnr, nb_jobs); + const int y_end = ff_slice_pos(out->height, jobnr + 1, nb_jobs); + const int width = out->width; + const int nb_ch = s->nb_channels; + + for (int y = y_start; y < y_end; y++) { + float *restrict out_row[3]; + for (int i = 0; i < nb_ch; i++) { + const int p = nb_ch == 1 ? 0 : gbr_order[i]; + out_row[i] = (float *) (out->data[p] + y * out->linesize[p]); + } + + for (int x = 0; x < width; x++) { + for (int i = 0; i < nb_ch; i++) + out_row[i][x] = quantize(out_row[i][x], scale[i], offset[i], gamma[i]); + } + } + + return 0; +} + +static void choose_quant_params(AVFilterContext *ctx, const AVFrame *out, int nb_jobs) +{ + GainMapContext *s = ctx->priv; + float frame_min[3] = { FLT_MAX, FLT_MAX, FLT_MAX }; + float frame_max[3] = { -FLT_MAX, -FLT_MAX, -FLT_MAX }; + double frame_sum[3] = { 0.0, 0.0, 0.0 }; + + for (int j = 0; j < nb_jobs; j++) { + for (int i = 0; i < s->nb_channels; i++) { + frame_min[i] = fminf(frame_min[i], s->slice_min[j][i]); + frame_max[i] = fmaxf(frame_max[i], s->slice_max[j][i]); + frame_sum[i] += s->slice_sum[j][i]; + } + } + + for (int i = 0; i < s->nb_channels; i++) { + float min = (s->measure & MEASURE_MIN) ? frame_min[i] : s->gain_min; + float max = (s->measure & MEASURE_MAX) ? frame_max[i] : s->gain_max; + if (s->measure & MEASURE_RANGE) + setup_range(ctx, i, min, max); + + if (!(s->measure & MEASURE_GAMMA)) + continue; + + const int64_t nb_pixels = (int64_t) out->width * out->height; + const double mean = frame_sum[i] / nb_pixels; + const double mean_quant = s->quant_scale[i] * mean + s->quant_offset[i]; + + /** + * Choose gamma such that mean_quant ^ gamma = 0.5; clamp to a sane + * value range of [1/8, 8] to prevent degenerate encodings. A gamma + * of >8 would collapse half the encoding space into a single 8-bit + * value, at which point we're losing more then gaining, and a value + * of <1/8 limits the amount by which a single high outlier pixel + * can determine the overall encoding gamma. + * + * We also clamp the mean to [0.01, 0.99] to prevent numerical explosion + * in the case that the mean is very close to 0 or 1. This is a looser + * bound than the final gamma clamp, so the exact values chosen do not + * matter as much. + */ + const double gamma = log(0.5) / log(av_clipd(mean_quant, 0.01, 0.99)); + setup_gamma(ctx, i, av_clipd(gamma, 1/8.0, 8.0)); + } +} + +static int process_frame(FFFrameSync *fs) +{ + AVFilterContext *ctx = fs->parent; + GainMapContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + AVFrame *base, *alt, *out; + + int ret = ff_framesync_dualinput_get(fs, &base, &alt); + if (ret < 0) + return ret; + if (!base || !alt) + return AVERROR_BUG; + + ret = setup_colorspace(ctx, base, alt); + if (ret < 0) + return ret; + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) + return AVERROR(ENOMEM); + av_frame_copy_props(out, base); + av_frame_side_data_remove_by_props(&out->side_data, &out->nb_side_data, + AV_SIDE_DATA_PROP_COLOR_DEPENDENT); + + out->color_trc = AVCOL_TRC_UNSPECIFIED; + out->color_primaries = AVCOL_PRI_UNSPECIFIED; + out->colorspace = AVCOL_SPC_UNSPECIFIED; + out->color_range = AVCOL_RANGE_JPEG; + + ThreadData td = { + .out = out, + .base = base, + .alt = alt, + }; + + if (!s->sign) { /* no-op */ + for (int i = 0; i < s->nb_channels; i++) + memset(out->data[i], 0, out->height * out->linesize[i]); + goto skip; + } + + const int nb_jobs = FFMIN(outlink->h, s->nb_threads); + ret = ff_filter_execute(ctx, s->measure ? slice_gain : slice_gain_quant, + &td, NULL, nb_jobs); + if (ret < 0) + goto fail; + + if (s->measure) { + choose_quant_params(ctx, out, nb_jobs); + ret = ff_filter_execute(ctx, slice_quant, &td, NULL, nb_jobs); + if (ret < 0) + goto fail; + } + +skip:; + AVGainMapParams *params; + params = av_gain_map_params_create_side_data(&out->side_data, &out->nb_side_data); + if (!params) { + ret = AVERROR(ENOMEM); + goto fail; + } + + *params = s->params; + if (av_gain_map_params_validate(params) < 0) + return AVERROR_BUG; /* should never happen */ + + return ff_filter_frame(outlink, out); + +fail: + av_frame_free(&out); + return ret; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + GainMapContext *s = ctx->priv; + AVFilterLink *base = ctx->inputs[0]; + AVFilterLink *alt = ctx->inputs[1]; + int ret; + + if (base->w != alt->w || base->h != alt->h) { + av_log(ctx, AV_LOG_ERROR, + "Input dimensions must match (%dx%d != %dx%d)\n", + base->w, base->h, alt->w, alt->h); + return AVERROR(EINVAL); + } + + outlink->w = base->w; + outlink->h = base->h; + outlink->colorspace = AVCOL_SPC_UNSPECIFIED; + outlink->color_range = AVCOL_RANGE_JPEG; + + s->nb_threads = ff_filter_get_nb_threads(ctx); + s->slice_min = av_calloc(s->nb_threads, sizeof(*s->slice_min)); + s->slice_max = av_calloc(s->nb_threads, sizeof(*s->slice_max)); + s->slice_sum = av_calloc(s->nb_threads, sizeof(*s->slice_sum)); + if (!s->slice_min || !s->slice_max || !s->slice_sum) + return AVERROR(ENOMEM); + + ret = ff_framesync_init_dualinput(&s->fs, ctx); + if (ret < 0) + return ret; + + s->fs.on_event = process_frame; + return ff_framesync_configure(&s->fs); +} + +static int activate(AVFilterContext *ctx) +{ + GainMapContext *s = ctx->priv; + return ff_framesync_activate(&s->fs); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + GainMapContext *s = ctx->priv; + ff_framesync_uninit(&s->fs); + av_freep(&s->slice_min); + av_freep(&s->slice_max); + av_freep(&s->slice_sum); +} + +static const AVFilterPad gainmap_inputs[] = { + { + .name = "base", + .type = AVMEDIA_TYPE_VIDEO, + }, + { + .name = "alternate", + .type = AVMEDIA_TYPE_VIDEO, + }, +}; + +static const AVFilterPad gainmap_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, +}; + +const FFFilter ff_vf_gainmap = { + .p.name = "gainmap", + .p.description = NULL_IF_CONFIG_SMALL("Generate a gain map from a base/alternate pair."), + .p.priv_class = &gainmap_class, + .p.flags = AVFILTER_FLAG_SLICE_THREADS, + .preinit = gainmap_framesync_preinit, + .priv_size = sizeof(GainMapContext), + .init = init, + .uninit = uninit, + .activate = activate, + FILTER_INPUTS(gainmap_inputs), + FILTER_OUTPUTS(gainmap_outputs), + FILTER_QUERY_FUNC2(query_formats), +}; diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index e0decc9b78..cd8cf26242 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -221,6 +221,14 @@ $(FATE_FILTER_FRAMEPACK): CMD = framecrc -c:v pgmyuv -i $(TARGET_PATH)/tests/vsy FATE_FILTER_VSYNTH_PGMYUV-$(CONFIG_FRAMEPACK_FILTER) += $(FATE_FILTER_FRAMEPACK) fate-filter-framepack: $(FATE_FILTER_FRAMEPACK) +FATE_FILTER_GAINMAP_SRC = testsrc2=s=64x48:r=1:d=1,scale,format=gbrpf32,split[b][a];[b]setparams=color_trc=iec61966-2-1:color_primaries=bt709[base];[a]setparams=color_trc=smpte2084:color_primaries=bt2020[alt];[base][alt] + +FATE_FILTER-$(call FILTERFRAMECRC, GAINMAP TESTSRC2 FORMAT SETPARAMS SPLIT, SCALE_FILTER) += fate-filter-gainmap fate-filter-gainmap-luma fate-filter-gainmap-maxrgb fate-filter-gainmap-fixed +fate-filter-gainmap: CMD = framecrc -lavfi "$(FATE_FILTER_GAINMAP_SRC)gainmap,scale" -pix_fmt gbrp +fate-filter-gainmap-luma: CMD = framecrc -lavfi "$(FATE_FILTER_GAINMAP_SRC)gainmap=mode=luma,scale" -pix_fmt gray +fate-filter-gainmap-maxrgb: CMD = framecrc -lavfi "$(FATE_FILTER_GAINMAP_SRC)gainmap=mode=maxrgb,scale" -pix_fmt gray +fate-filter-gainmap-fixed: CMD = framecrc -lavfi "$(FATE_FILTER_GAINMAP_SRC)gainmap=min=0:max=2.3:gamma=1:base_nits=203:alt_nits=1000,scale" -pix_fmt gbrp + FATE_FILTER_VSYNTH_PGMYUV-$(CONFIG_GRADFUN_FILTER) += fate-filter-gradfun fate-filter-gradfun: CMD = framecrc -c:v pgmyuv -i $(SRC) -vf gradfun diff --git a/tests/ref/fate/filter-gainmap b/tests/ref/fate/filter-gainmap new file mode 100644 index 0000000000..96d5904207 --- /dev/null +++ b/tests/ref/fate/filter-gainmap @@ -0,0 +1,6 @@ +#tb 0: 1/1 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 64x48 +#sar 0: 1/1 +0, 0, 0, 1, 9216, 0x91dedb2f diff --git a/tests/ref/fate/filter-gainmap-fixed b/tests/ref/fate/filter-gainmap-fixed new file mode 100644 index 0000000000..93b721a20e --- /dev/null +++ b/tests/ref/fate/filter-gainmap-fixed @@ -0,0 +1,6 @@ +#tb 0: 1/1 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 64x48 +#sar 0: 1/1 +0, 0, 0, 1, 9216, 0x28fde0dc diff --git a/tests/ref/fate/filter-gainmap-luma b/tests/ref/fate/filter-gainmap-luma new file mode 100644 index 0000000000..ea6d074487 --- /dev/null +++ b/tests/ref/fate/filter-gainmap-luma @@ -0,0 +1,6 @@ +#tb 0: 1/1 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 64x48 +#sar 0: 1/1 +0, 0, 0, 1, 3072, 0xbfac1585 diff --git a/tests/ref/fate/filter-gainmap-maxrgb b/tests/ref/fate/filter-gainmap-maxrgb new file mode 100644 index 0000000000..1d52102004 --- /dev/null +++ b/tests/ref/fate/filter-gainmap-maxrgb @@ -0,0 +1,6 @@ +#tb 0: 1/1 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 64x48 +#sar 0: 1/1 +0, 0, 0, 1, 3072, 0x2253f63c -- 2.52.0 >From 2015c8a6b3ca290c0f2830b16ce51ee9cb46a317 Mon Sep 17 00:00:00 2001 From: Niklas Haas <[email protected]> Date: Fri, 21 Aug 2026 13:23:56 +0200 Subject: [PATCH 5/8] avcodec/gain_map: add ff_gain_map_params_(to/from)_iso21496 I tested it locally by ensuring it round trips, though I did not yet test the output against the libultrahdr reference blob. A proper FATE test will be included implicitly once we have a corresponding muxer. Signed-off-by: Niklas Haas <[email protected]> --- libavcodec/gain_map.c | 167 ++++++++++++++++++++++++++++++++++++++++++ libavcodec/gain_map.h | 66 +++++++++++++++++ 2 files changed, 233 insertions(+) create mode 100644 libavcodec/gain_map.c create mode 100644 libavcodec/gain_map.h diff --git a/libavcodec/gain_map.c b/libavcodec/gain_map.c new file mode 100644 index 0000000000..cbcaac14b4 --- /dev/null +++ b/libavcodec/gain_map.c @@ -0,0 +1,167 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/avassert.h" +#include "libavutil/error.h" +#include "libavutil/gain_map.h" +#include "libavutil/mem.h" +#include "libavutil/rational.h" + +#include "bytestream.h" +#include "gain_map.h" + +#define ISO21496_HEADER_SIZE 5 + +#define ISO21496_FLAG_MULTICHANNEL (1 << 7) +#define ISO21496_FLAG_USE_BASE_CG (1 << 6) +#define ISO21496_FLAG_RESERVED 0x3f + +static size_t iso21496_payload_size(int nb_channels) +{ + return 4 * 4 + (size_t) nb_channels * 10 * 4; +} + +static int make_q(int64_t num, uint32_t den, AVRational *out) +{ + if (!den) + return AVERROR_INVALIDDATA; + + /* Ensure the result fits in AVRational */ + av_reduce(&out->num, &out->den, num, den, INT32_MAX); + return 0; +} + +static int get_sq(GetByteContext *gb, AVRational *out) +{ + int32_t num = (int32_t) bytestream2_get_be32(gb); + uint32_t den = bytestream2_get_be32u(gb); + return make_q(num, den, out); +} + +static int get_uq(GetByteContext *gb, AVRational *out) +{ + uint32_t num = bytestream2_get_be32u(gb); + uint32_t den = bytestream2_get_be32u(gb); + return make_q(num, den, out); +} + +int ff_gain_map_params_from_iso21496(AVGainMapParams *restrict p, + const uint8_t *data, size_t size) +{ + int ret; + if (!p || !data) + return AVERROR(EINVAL); + + if (size < ISO21496_HEADER_SIZE || size > INT_MAX) + return AVERROR_INVALIDDATA; + + GetByteContext gb; + bytestream2_init(&gb, data, size); + + /* Parse header */ + const uint16_t minimum_version = bytestream2_get_be16(&gb); + const uint16_t writer_version = bytestream2_get_be16(&gb); + if (minimum_version > FF_GAIN_MAP_VERSION || writer_version < minimum_version) + return AVERROR_PATCHWELCOME; + const uint8_t flags = bytestream2_get_byte(&gb); + + /** + * While the spec does not require reserved bits to be 0, older versions + * of the ISO 21496-1 draft specification had extra flags that got removed + * from the final publication, so conservatively error out just in case; + * unless the writer version is >0 in which case these may be genuine + * additions to later revisions of the published spec, the validity of + * ignoring which is already guarded by the `minimum_version` check. + */ + if (writer_version == 0 && (flags & ISO21496_FLAG_RESERVED)) + return AVERROR_INVALIDDATA; + + p->version = FF_GAIN_MAP_VERSION; + p->nb_channels = (flags & ISO21496_FLAG_MULTICHANNEL) ? 3 : 1; + p->use_base_color_space = !!(flags & ISO21496_FLAG_USE_BASE_CG); + + /* Parse payload */ + if (bytestream2_get_bytes_left(&gb) < iso21496_payload_size(p->nb_channels)) + return AVERROR_INVALIDDATA; + + if ((ret = get_uq(&gb, &p->base_hdr_headroom)) < 0 || + (ret = get_uq(&gb, &p->alternate_hdr_headroom)) < 0) + return ret; + + for (int c = 0; c < p->nb_channels; c++) { + struct AVGainMapChannel *const ch = &p->channels[c]; + if ((ret = get_sq(&gb, &ch->gain_map_min)) < 0 || + (ret = get_sq(&gb, &ch->gain_map_max)) < 0 || + (ret = get_uq(&gb, &ch->gamma)) < 0 || + (ret = get_sq(&gb, &ch->base_offset)) < 0 || + (ret = get_sq(&gb, &ch->alternate_offset)) < 0) + return ret; + } + + /* nb_channels == 1 means every channel shares the same parameters */ + for (int c = p->nb_channels; c < 3; c++) + p->channels[c] = p->channels[0]; + + return av_gain_map_params_validate(p); +} + +static void put_q(PutByteContext *pb, AVRational q) +{ + bytestream2_put_be32(pb, (unsigned) q.num); + bytestream2_put_be32(pb, (unsigned) q.den); +} + +int ff_gain_map_params_to_iso21496(const AVGainMapParams *p, uint8_t *buf) +{ + if (!p || !buf) + return AVERROR(EINVAL); + + /* Ensures representability in the bitstream */ + int ret = av_gain_map_params_validate(p); + if (ret < 0) + return ret; + + if (p->version > FF_GAIN_MAP_VERSION) + return AVERROR_PATCHWELCOME; /* e.g. libraries out of sync */ + + PutByteContext pb; + bytestream2_init_writer(&pb, buf, FF_GAIN_MAP_MAX_PAYLOAD_SIZE); + + uint8_t flags = 0; + if (p->nb_channels == 3) + flags |= ISO21496_FLAG_MULTICHANNEL; + if (p->use_base_color_space) + flags |= ISO21496_FLAG_USE_BASE_CG; + + bytestream2_put_be16(&pb, p->version); /* minimum_version */ + bytestream2_put_be16(&pb, p->version); /* writer_version */ + bytestream2_put_byte(&pb, flags); + put_q(&pb, p->base_hdr_headroom); + put_q(&pb, p->alternate_hdr_headroom); + + for (int c = 0; c < p->nb_channels; c++) { + const struct AVGainMapChannel *const ch = &p->channels[c]; + put_q(&pb, ch->gain_map_min); + put_q(&pb, ch->gain_map_max); + put_q(&pb, ch->gamma); + put_q(&pb, ch->base_offset); + put_q(&pb, ch->alternate_offset); + } + + return bytestream2_tell_p(&pb); +} diff --git a/libavcodec/gain_map.h b/libavcodec/gain_map.h new file mode 100644 index 0000000000..17f1b2720c --- /dev/null +++ b/libavcodec/gain_map.h @@ -0,0 +1,66 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Serialization of AVGainMapParams into the metadata blobs that image formats + * carry it in, and back. Kept separate from the codecs so that an encoder + * writing the blob and a bitstream filter splicing it into an existing + * bitstream can share the same code. + */ + +#ifndef AVCODEC_GAIN_MAP_H +#define AVCODEC_GAIN_MAP_H + +#include "libavutil/gain_map.h" + +/* May differ from AV_ISO21496_VERSION if libraries are not in sync */ +#define FF_GAIN_MAP_VERSION 0 +#define FF_GAIN_MAP_MAX_PAYLOAD_SIZE 141 /* = 5 + 2*8 + 3*5*8 */ + +/** + * Parse the payload of an ISO 21496-1 gain map metadata blob. + * + * @param p struct to fill; overwritten on success, untouched on failure + * @param data payload, excluding container-specific identifiers + * @param size size of `data` in bytes + * + * @return >= 0 on success, or a negative AVERROR on failure + */ +int ff_gain_map_params_from_iso21496(AVGainMapParams *p, const uint8_t *data, + size_t size); + +/** + * Serialize an AVGainMapParams as an ISO 21496-1 gain map metadata blob. + * + * The output starts at the minimum_version field; the caller should prepend + * whatever identifier its container requires. + * + * @param p parameters to serialize + * @param buf A pointer to a pointer to a byte buffer to be filled with the + * serialized metadata. Must contain at least + * FF_GAIN_MAP_MAX_PAYLOAD_SIZE bytes. + * + * @return Number of bytes written on success, or a negative AVERROR on failure + * + * @note Always succeeds if av_gain_map_params_validate() returns >= 0, and + * p->version <= AV_ISO21496_VERSION. + */ +int ff_gain_map_params_to_iso21496(const AVGainMapParams *p, uint8_t *buf); + +#endif /* AVCODEC_GAIN_MAP_H */ -- 2.52.0 >From f449876021df409b99c41afee5522e8d79254668 Mon Sep 17 00:00:00 2001 From: Niklas Haas <[email protected]> Date: Fri, 21 Aug 2026 16:29:40 +0200 Subject: [PATCH 6/8] avcodec/mjpegenc: add support for writing ISO 21496-1 gain map metadata This is automatically emitted as an APP2 blob following the ISO 21496-1 specification. To ensure that writing the gain map will succeed, we validate the paramaters when allocating space, and error out if invalid data is encountered. Signed-off-by: Niklas Haas <[email protected]> --- libavcodec/Makefile | 6 ++--- libavcodec/mjpegenc_common.c | 43 ++++++++++++++++++++++++++++++++++++ libavcodec/mjpegenc_common.h | 2 ++ libavcodec/mpegvideo_enc.c | 3 +++ 4 files changed, 51 insertions(+), 3 deletions(-) diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 3cdfa4f383..45f954c453 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -239,7 +239,7 @@ OBJS-$(CONFIG_AMRWB_DECODER) += amrwbdec.o celp_filters.o \ acelp_pitch_delay.o OBJS-$(CONFIG_AMRNB_MEDIACODEC_DECODER) += mediacodecdec.o OBJS-$(CONFIG_AMRWB_MEDIACODEC_DECODER) += mediacodecdec.o -OBJS-$(CONFIG_AMV_ENCODER) += mjpegenc.o mjpegenc_common.o +OBJS-$(CONFIG_AMV_ENCODER) += mjpegenc.o mjpegenc_common.o gain_map.o OBJS-$(CONFIG_ANM_DECODER) += anm.o OBJS-$(CONFIG_ANULL_DECODER) += null.o OBJS-$(CONFIG_ANULL_ENCODER) += null.o @@ -519,7 +519,7 @@ OBJS-$(CONFIG_KGV1_DECODER) += kgv1dec.o OBJS-$(CONFIG_KMVC_DECODER) += kmvc.o OBJS-$(CONFIG_LAGARITH_DECODER) += lagarith.o lagarithrac.o OBJS-$(CONFIG_LEAD_DECODER) += leaddec.o jpegquanttables.o -OBJS-$(CONFIG_LJPEG_ENCODER) += ljpegenc.o mjpegenc_common.o +OBJS-$(CONFIG_LJPEG_ENCODER) += ljpegenc.o mjpegenc_common.o gain_map.o OBJS-$(CONFIG_LOCO_DECODER) += loco.o OBJS-$(CONFIG_LSCR_DECODER) += lscrdec.o png.o pngdec.o pngdsp.o OBJS-$(CONFIG_M101_DECODER) += m101.o @@ -536,7 +536,7 @@ OBJS-$(CONFIG_MISC4_DECODER) += misc4.o OBJS-$(CONFIG_MJPEG_DECODER) += mjpegdec.o mjpegdec_common.o OBJS-$(CONFIG_MJPEG_QSV_DECODER) += qsvdec.o OBJS-$(CONFIG_MJPEG_ENCODER) += mjpegenc.o mjpegenc_common.o \ - mjpegenc_huffman.o + mjpegenc_huffman.o gain_map.o OBJS-$(CONFIG_MJPEGB_DECODER) += mjpegbdec.o OBJS-$(CONFIG_MJPEG_CUVID_DECODER) += cuviddec.o OBJS-$(CONFIG_MJPEG_QSV_ENCODER) += qsvenc_jpeg.o diff --git a/libavcodec/mjpegenc_common.c b/libavcodec/mjpegenc_common.c index 5effbdbc32..de9518de97 100644 --- a/libavcodec/mjpegenc_common.c +++ b/libavcodec/mjpegenc_common.c @@ -31,6 +31,7 @@ #include "jpegtables.h" #include "put_bits.h" #include "mjpegenc.h" +#include "gain_map.h" #include "mjpegenc_common.h" #include "mjpeg.h" #include "version.h" @@ -159,6 +160,29 @@ int ff_mjpeg_add_icc_profile_size(AVCodecContext *avctx, const AVFrame *frame, return 0; } +int ff_mjpeg_add_gain_map_size(AVCodecContext *avctx, const AVFrame *frame, + size_t *max_pkt_size) +{ + if (avctx->codec_id != AV_CODEC_ID_MJPEG) + return 0; + + AVFrameSideData *sd; + sd = av_frame_get_side_data(frame, AV_FRAME_DATA_GAIN_MAP_PARAMS); + if (!sd || sd->size < sizeof(AVGainMapParams)) + return 0; + + int ret = av_gain_map_params_validate((const AVGainMapParams *) sd->data); + if (ret < 0) + return ret; + + size_t app2_size = 4 + sizeof(AV_ISO21496_IDENTIFIER) + FF_GAIN_MAP_MAX_PAYLOAD_SIZE; + size_t new_pkt_size = *max_pkt_size + app2_size; + if (new_pkt_size < *max_pkt_size) /* overflow */ + return AVERROR_INVALIDDATA; + *max_pkt_size = new_pkt_size; + return 0; +} + static void jpeg_put_comments(AVCodecContext *avctx, PutBitContext *p, const AVFrame *frame) { @@ -221,6 +245,25 @@ static void jpeg_put_comments(AVCodecContext *avctx, PutBitContext *p, av_assert1(!remaining); } + /* ISO 21496-1 gain map metadata */ + sd = avctx->codec_id == AV_CODEC_ID_MJPEG ? + av_frame_get_side_data(frame, AV_FRAME_DATA_GAIN_MAP_PARAMS) : NULL; + if (sd && sd->size >= sizeof(AVGainMapParams)) { + put_marker(p, APP2); + flush_put_bits(p); + ptr = put_bits_ptr(p); + put_bits(p, 16, 0); /* patched later */ + ff_put_string(p, AV_ISO21496_IDENTIFIER, 1); + flush_put_bits(p); + + /* pre-validated by ff_mjpeg_add_gain_map_size() */ + const AVGainMapParams *params = (const AVGainMapParams *) sd->data; + size = ff_gain_map_params_to_iso21496(params, put_bits_ptr(p)); + av_assert0(size >= 0); + skip_put_bytes(p, size); + AV_WB16(ptr, 2 + sizeof(AV_ISO21496_IDENTIFIER) + size); + } + /* comment */ if (!(avctx->flags & AV_CODEC_FLAG_BITEXACT)) { put_marker(p, COM); diff --git a/libavcodec/mjpegenc_common.h b/libavcodec/mjpegenc_common.h index 0cf5a72706..4d1e2a5477 100644 --- a/libavcodec/mjpegenc_common.h +++ b/libavcodec/mjpegenc_common.h @@ -30,6 +30,8 @@ struct MJpegContext; int ff_mjpeg_add_icc_profile_size(AVCodecContext *avctx, const AVFrame *frame, size_t *max_pkt_size); +int ff_mjpeg_add_gain_map_size(AVCodecContext *avctx, const AVFrame *frame, + size_t *max_pkt_size); void ff_mjpeg_encode_picture_header(AVCodecContext *avctx, PutBitContext *pb, const AVFrame *frame, const struct MJpegContext *m, const uint8_t intra_matrix_permutation[64], diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c index 7751de8510..62cf2ecbf4 100644 --- a/libavcodec/mpegvideo_enc.c +++ b/libavcodec/mpegvideo_enc.c @@ -1931,6 +1931,9 @@ int ff_mpv_encode_picture(AVCodecContext *avctx, AVPacket *pkt, ret = ff_mjpeg_add_icc_profile_size(avctx, s->new_pic, &pkt_size); if (ret < 0) return ret; + ret = ff_mjpeg_add_gain_map_size(avctx, s->new_pic, &pkt_size); + if (ret < 0) + return ret; } if ((ret = ff_alloc_packet(avctx, pkt, pkt_size)) < 0) return ret; -- 2.52.0 >From 604ad3255565a9e79d899d4de2040079a57387d2 Mon Sep 17 00:00:00 2001 From: Niklas Haas <[email protected]> Date: Fri, 21 Aug 2026 13:41:38 +0200 Subject: [PATCH 7/8] avformat/jpegmpfenc: add JPEG-MPF muxer This can be used to store two (or more, in principle) JPEG images in a single Multiple Picture Format (MPF) file. This is a JPEG extension defined in CIPA DC-007. Currently, we only support gain maps, so the muxer as written could be made a lot simpler by hard-coding that assumption. However, I anticipate a future need to use this to store e.g. large thumbnails, multiple angles or other MPF picture types, so I made some effort to generalize the code to follow the spec's design more faithfully, rather than taking the lazy way out. This muxer assumes each input is an already encoded JPEG stream. It's up to users to determine how to produce that stream - in theory, it could even be an externally encoded JPEG file that is passed through as-is, though the fact that there's no current way to attach externally provided gain map parameters onto an externally encoded JPEG image means that this functionality is, in practice, essentially restricted to the base image encoding, with the gain map having to be encoded using FFmpeg's own mjpeg muxer to ensure the gain map parameters are included as an ISO21496 blob. Since the MPF spec requires hard-coding offsets to later images as part of the MPF header (relative to the start of that header), we use a two-pass approach, where the first pass writes out the image using a null avio to count the number of bytes, and the second pass writes the actual data. This roughly matches the design used by the LCEVC muxer, among others. The files output by this muxer pass `exiftool -validate`, though they are not bit-exact with the output of libultrahdr because libultrahdr deviates from the spec in a number of ways, all of which appear to be bugs in libultrahdr to me: - We tag 0x050000 ("Gain map image") rather than 0x00000 ("Undefined") - We correctly mark image dependencies (libultrahdr leaves away this metadata) - We correctly place the ISO21496 APP2 after the APP0 JFIF, while libultrahdr places it as the first segment after the SOI - We place the MP IFD index after the APP1 Exif attributes, but before any other APPn segments, which is the order shown in the DC-007 spec (Figure 1), while libultrahdr always places it right before the SOS segment. Signed-off-by: Niklas Haas <[email protected]> --- doc/muxers.texi | 21 ++ libavformat/Makefile | 1 + libavformat/allformats.c | 1 + libavformat/jpegmpfenc.c | 527 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 550 insertions(+) create mode 100644 libavformat/jpegmpfenc.c diff --git a/doc/muxers.texi b/doc/muxers.texi index 99584e2b8d..351212f487 100644 --- a/doc/muxers.texi +++ b/doc/muxers.texi @@ -2780,6 +2780,27 @@ passthrough first with @command{tmux set -g allow-passthrough on}, then add ffmpeg -re -i input.mp4 -f iterm2 -display_height 40 -tmux 1 - @end example +@section jpeg_mpf +JPEG Multi-Picture Format muxer. + +Stores multiple JPEG images in a single file using the CIPA DC-007 MP +extensions. The first input stream is always the primary image. Every other +stream must be specified by a stream group describing what it is; see the +@option{-stream_group} option of the @command{ffmpeg} tool. + +@note Currently, the only implemented type of stream group is @samp{gain_map}, +which may be used to store a gain map for JPEG Ultra HDR. + +@subsection Examples + +Encode an SDR tone-mapped image and an HDR gain map computed from it: +@example +ffmpeg -i input-hdr.png -filter_complex +"[0:v] scale=out_transfer=srgb:out_primaries=bt709,split [base][sdr]; +[base][0:v] gainmap [gain]" -map "[sdr]" -map "[gain]" -c:v mjpeg +-stream_group type=gain_map:st=0:st=1:el_index=1 -f jpeg_mpf output.jpg +@end example + @section ivf On2 IVF muxer. diff --git a/libavformat/Makefile b/libavformat/Makefile index 466f5d1894..cc67d1b376 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -345,6 +345,7 @@ OBJS-$(CONFIG_IVF_MUXER) += ivfenc.o OBJS-$(CONFIG_IVR_DEMUXER) += rmdec.o rm.o rmsipr.o OBJS-$(CONFIG_JACOSUB_DEMUXER) += jacosubdec.o subtitles.o OBJS-$(CONFIG_JACOSUB_MUXER) += jacosubenc.o rawenc.o +OBJS-$(CONFIG_JPEG_MPF_MUXER) += jpegmpfenc.o OBJS-$(CONFIG_JPEGXL_ANIM_DEMUXER) += jpegxl_anim_dec.o OBJS-$(CONFIG_JV_DEMUXER) += jvdec.o OBJS-$(CONFIG_KUX_DEMUXER) += flvdec.o diff --git a/libavformat/allformats.c b/libavformat/allformats.c index e121c7441c..27026e9e5c 100644 --- a/libavformat/allformats.c +++ b/libavformat/allformats.c @@ -250,6 +250,7 @@ extern const FFInputFormat ff_ivr_demuxer; extern const FFInputFormat ff_jacosub_demuxer; extern const FFOutputFormat ff_jacosub_muxer; extern const FFInputFormat ff_jv_demuxer; +extern const FFOutputFormat ff_jpeg_mpf_muxer; extern const FFInputFormat ff_jpegxl_anim_demuxer; extern const FFInputFormat ff_kux_demuxer; extern const FFInputFormat ff_kvag_demuxer; diff --git a/libavformat/jpegmpfenc.c b/libavformat/jpegmpfenc.c new file mode 100644 index 0000000000..40d827c537 --- /dev/null +++ b/libavformat/jpegmpfenc.c @@ -0,0 +1,527 @@ +/* + * JPEG Multi-Picture Format muxer + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Muxes several JPEG images into a single JPEG Multi-Picture Format file. + */ + +#include <string.h> +#include <stddef.h> + +#include "libavutil/avassert.h" +#include "libavutil/gain_map.h" +#include "libavutil/internal.h" +#include "libavutil/mem.h" + +#include "libavcodec/bytestream.h" +#include "libavcodec/exif.h" +#include "libavcodec/mjpeg.h" + +#include "avformat.h" +#include "avio_internal.h" +#include "mux.h" + +/* Individual Image Attribute bits, from CIPA DC-007 Figure 8 */ +enum MPFAttribute { + MPF_DEPENDENT_PARENT = 0x80000000, + MPF_DEPENDENT_CHILD = 0x40000000, + MPF_FORMAT_JPEG = 0x00000000, +}; + +/* MP Index IFD tags, from CIPA DC-007 Table 3 */ +enum MPFTag { + MPF_TAG_MPFVersion = 0xB000, + MPF_TAG_NumberOfImages = 0xB001, + MPF_TAG_MPEntry = 0xB002, +}; + +typedef enum MPFRole { + MPF_ROLE_UNKNOWN = 0, + MPF_ROLE_PRIMARY, + MPF_ROLE_GAIN_MAP, + MPF_ROLE_NB, +} MPFRole; + +/* APP2 extension segments */ +typedef enum MPFExt { + MPF_EXT_INDEX, /* MP Index IFD, First Individual Image only */ + MPF_EXT_ISO21496, /* ISO 21496-1 version fields (in the base image) */ + MPF_EXT_NB, +} MPFExt; + +static const struct { + const char *name; + uint32_t type; /* from CIPA DC-007 Table 4 */ +} mpf_roles[MPF_ROLE_NB] = { + [MPF_ROLE_PRIMARY] = { "primary image", 0x030000 }, + [MPF_ROLE_GAIN_MAP] = { "gain map image", 0x050000 }, +}; + +typedef struct MPFImage MPFImage; +struct MPFImage { + AVPacket *packet; + int index; + MPFRole role; + + /* Relative to the packet data (before splicing) */ + int app2; /* offset to role-specific APP2 header */ + int splice_pos; /* where in `packet->data` to splice our segments */ + + /* Relative to the output file (after splicing) */ + int64_t start, end; /* start and end of this image in the output stream */ + int64_t ext_size[MPF_EXT_NB]; /* APP2 payload sizes */ + + /* Image dependency metadata */ + MPFImage *dep; + MPFImage *rev_deps[2]; /* maximum 2 permissible by the spec */ + int nb_rev_deps; +}; + +typedef struct MPFMuxContext { + const AVClass *class; + MPFImage *images; +} MPFMuxContext; + +static int register_dep(AVFormatContext *ctx, MPFImage *img, MPFImage *base) +{ + if (base->nb_rev_deps == FF_ARRAY_ELEMS(base->rev_deps)) { + av_log(ctx, AV_LOG_ERROR, "Stream %d cannot depend on %d, maximum " + "number of dependents reached (%d).\n", img->index, base->index, + (int) FF_ARRAY_ELEMS(base->rev_deps)); + return AVERROR(EINVAL); + } + + base->rev_deps[base->nb_rev_deps++] = img; + img->dep = base; + return 0; +} + +static int init_gain_map(AVFormatContext *ctx, const AVStreamGroup *sg) +{ + MPFMuxContext *s = ctx->priv_data; + const AVStreamGroupLayeredVideo *lv = sg->params.layered_video; + if (sg->nb_streams != 2 || lv->el_index >= sg->nb_streams) + return AVERROR(EINVAL); + + const AVStream *bl = sg->streams[!lv->el_index]; + const AVStream *el = sg->streams[lv->el_index]; + MPFImage *base = &s->images[bl->index]; + MPFImage *gain = &s->images[el->index]; + + if (base->index != 0) { + av_log(ctx, AV_LOG_ERROR, "Stream group %u: the gain map base layer must " + "be the primary image (stream 0), got stream %d\n", + sg->index, base->index); + return AVERROR(EINVAL); + } + + if (gain->role != MPF_ROLE_UNKNOWN) { + av_log(ctx, AV_LOG_ERROR, "Stream group %u: stream %d already used " + "as %s for a different stream group.\n", sg->index, gain->index, + mpf_roles[gain->role].name); + return AVERROR(EINVAL); + } + + for (int i = 0; i < base->nb_rev_deps; i++) { + const MPFImage *other = base->rev_deps[i]; + if (other->role == MPF_ROLE_GAIN_MAP) { + av_log(ctx, AV_LOG_ERROR, "Stream group %u: stream %d already " + "specified as a gain map for stream %d.\n", sg->index, + other->index, base->index); + return AVERROR(EINVAL); + } + } + + gain->role = MPF_ROLE_GAIN_MAP; + return register_dep(ctx, gain, base); +} + +static int mpf_init(AVFormatContext *ctx) +{ + int ret; + + MPFMuxContext *s = ctx->priv_data; + if (ctx->nb_streams < 2) + av_log(ctx, AV_LOG_WARNING, "Specified only a single stream (no-op)\n"); + + s->images = av_calloc(ctx->nb_streams, sizeof(*s->images)); + if (!s->images) + return AVERROR(ENOMEM); + + for (int i = 0; i < ctx->nb_streams; i++) { + MPFImage *img = &s->images[i]; + img->index = i; + img->start = img->end = -1; + for (int n = 0; n < MPF_EXT_NB; n++) + img->ext_size[n] = -1; + img->packet = av_packet_alloc(); + if (!img->packet) + return AVERROR(ENOMEM); + } + + /* Classify streams by their role */ + s->images[0].role = MPF_ROLE_PRIMARY; + + for (unsigned i = 0; i < ctx->nb_stream_groups; i++) { + const AVStreamGroup *sg = ctx->stream_groups[i]; + switch (sg->type) { + case AV_STREAM_GROUP_PARAMS_GAIN_MAP: + if ((ret = init_gain_map(ctx, sg)) < 0) + return ret; + break; + + default: + av_log(ctx, AV_LOG_ERROR, "Stream group %u: unknown type '%s'\n", i, + avformat_stream_group_name(sg->type)); + return AVERROR(EINVAL); + } + } + + for (int i = 0; i < ctx->nb_streams; i++) { + if (s->images[i].role == MPF_ROLE_UNKNOWN) { + av_log(ctx, AV_LOG_ERROR, "Missing stream group for stream %d.\n", i); + return AVERROR(EINVAL); + } + } + + return 0; +} + +static int parse_app2(AVFormatContext *ctx, MPFImage *img, + const GetByteContext *gb, int len) +{ +#define CHECK_IDENT(strc) (len >= sizeof(strc) && !memcmp(gb->buffer, strc, sizeof(strc))) + switch (img->role) { + case MPF_ROLE_GAIN_MAP: + if (CHECK_IDENT(AV_ISO21496_IDENTIFIER)) { + if (img->app2) { + av_log(ctx, AV_LOG_ERROR, "Stream %d: multiple ISO 21496-1 " + "APP2 segments\n", img->index); + return AVERROR_INVALIDDATA; + } + + if (len < sizeof(AV_ISO21496_IDENTIFIER) + sizeof(uint16_t[2])) { + av_log(ctx, AV_LOG_ERROR, "Stream %d: ISO 21496-1 payload too " + "short\n", img->index); + return AVERROR_INVALIDDATA; + } + img->app2 = bytestream2_tell(gb); + } + break; + } +#undef CHECK_IDENT + + return 0; +} + +static int mpf_write_packet(AVFormatContext *ctx, AVPacket *pkt) +{ + int ret; + MPFMuxContext *s = ctx->priv_data; + MPFImage *img = &s->images[pkt->stream_index]; + if (img->packet->size) { + av_log(ctx, AV_LOG_ERROR, "Stream %d: expected a single JPEG image\n", img->index); + return AVERROR(EINVAL); + } + + GetByteContext gb; + bytestream2_init(&gb, pkt->data, pkt->size); + if (bytestream2_get_bytes_left(&gb) < 2 || + bytestream2_get_be16(&gb) != (0xFF00 | SOI)) + { + av_log(ctx, AV_LOG_ERROR, "Stream %d: not a JPEG image (no SOI)\n", img->index); + return AVERROR_INVALIDDATA; + } + + /* Walk the JPEG marker sections up until the start of entropy coded data */ + while (bytestream2_get_bytes_left(&gb) >= 4) { + const int pos = bytestream2_tell(&gb); + if (bytestream2_get_byte(&gb) != 0xFF) { + av_log(ctx, AV_LOG_ERROR, "Stream %d: malformed JPEG marker at 0x%x\n", + img->index, pos); + return AVERROR_INVALIDDATA; + } + + int marker = bytestream2_get_byte(&gb); + if (marker == SOS || marker == TEM || (marker >= RST0 && marker <= EOI)) { + if (!img->splice_pos) + img->splice_pos = pos; + break; /* start of entropy coded data */ + } + + int len = bytestream2_get_be16(&gb) - 2; + if (len < 0 || len > bytestream2_get_bytes_left(&gb)) { + av_log(ctx, AV_LOG_ERROR, "Stream %d: truncated marker segment at " + "0x%x\n", img->index, pos); + return AVERROR_INVALIDDATA; + } + + switch (marker) { + case APP0: + case APP1: + break; /* ignored */ + case APP2: + ret = parse_app2(ctx, img, &gb, len); + if (ret < 0) + return ret; + av_fallthrough; + default: + /** + * Splice position for the MPF metadata shall be the first segment + * position after any SOI, APP0 or APP1 (cf. CIPA DC-007) + */ + if (!img->splice_pos) + img->splice_pos = pos; + break; + } + + bytestream2_skip(&gb, len); + } + + /* Verify completeness of all needed information */ + switch (img->role) { + case MPF_ROLE_PRIMARY: + if (!img->splice_pos) { + av_log(ctx, AV_LOG_ERROR, "Stream %d: truncated JPEG data\n", img->index); + return AVERROR_INVALIDDATA; + } + break; + case MPF_ROLE_GAIN_MAP: + if (!img->app2) { + av_log(ctx, AV_LOG_ERROR, "Stream %d was declared as a gain map " + "but is missing ISO 21496-1 metadata\n", img->index); + return AVERROR_INVALIDDATA; + } + break; + } + + return av_packet_ref(img->packet, pkt); +} + +static void update_offset(AVIOContext *pb, int64_t base, int64_t *offset) +{ + int64_t val = avio_tell(pb) - base; + av_assert0(val >= 0); + av_assert0(*offset < 0 || *offset == val); + *offset = val; +} + +static int mpf_write_index(AVFormatContext *ctx, AVIOContext *pb, int64_t base, + MPFImage *img) +{ + const MPFMuxContext *s = ctx->priv_data; + + avio_w8(pb, 0xFF); + avio_w8(pb, APP2); + const int64_t app2_start = avio_tell(pb) - base; + avio_wb16(pb, img->ext_size[MPF_EXT_INDEX]); + avio_write(pb, "MPF\0", 4); + + /* every offset below is relative to this byte */ + int64_t tiff_base = avio_tell(pb) - base; +#define RELPOS(x) (avio_tell(pb) - base - tiff_base + (x)) + + avio_write(pb, "MM\0*", 4); /* big endian */ + avio_wb32(pb, RELPOS(4)); /* offset to first IFD */ + + /* First (and only) IFD */ + const int nb_tags = 3; /* MPFVersion, NumberOfImages, MPEntry */ + avio_wb16(pb, nb_tags); /* number of IFD tags */ + + /* MPFVersion tag */ + avio_wb16(pb, MPF_TAG_MPFVersion); + avio_wb16(pb, AV_TIFF_UNDEFINED); + avio_wb32(pb, 4); /* count */ + avio_write(pb, "0100", 4); + + /* NumberOfImages tag */ + avio_wb16(pb, MPF_TAG_NumberOfImages); + avio_wb16(pb, AV_TIFF_LONG); + avio_wb32(pb, 1); + avio_wb32(pb, ctx->nb_streams); + + /* MPEntry tag */ + const int64_t entry_size = sizeof(int32_t[3]) + sizeof(int16_t[2]); + avio_wb16(pb, MPF_TAG_MPEntry); + avio_wb16(pb, AV_TIFF_UNDEFINED); + avio_wb32(pb, ctx->nb_streams * entry_size); + avio_wb32(pb, RELPOS(4 + 4)); /* offset to MPEntry data */ + avio_wb32(pb, 0); /* no MP Attribute IFD, per DC-007 6.1.1.1 */ + + for (int i = 0; i < ctx->nb_streams; i++) { + const MPFImage *entry = &s->images[i]; + int64_t size = entry->end - entry->start; + av_assert0(size <= UINT32_MAX); /* would overflow null buffer anyways */ + uint32_t attr = MPF_FORMAT_JPEG | mpf_roles[entry->role].type; + if (entry->nb_rev_deps) + attr |= MPF_DEPENDENT_PARENT; + if (entry->dep) + attr |= MPF_DEPENDENT_CHILD; + + avio_wb32(pb, attr); + avio_wb32(pb, size); + + /* The data offset of the First Individual Image is always zero */ + int64_t offset = i ? entry->start - tiff_base : 0; + av_assert0(offset <= UINT32_MAX); + avio_wb32(pb, offset); + + /* Indices are 1-based; 0 means "no dependent image" */ + for (int j = 0; j < FF_ARRAY_ELEMS(entry->rev_deps); j++) { + const MPFImage *rdep = entry->rev_deps[j]; + avio_wb16(pb, rdep ? rdep->index + 1 : 0); + } + } +#undef RELPOS + + update_offset(pb, base + app2_start, &img->ext_size[MPF_EXT_INDEX]); + return 0; +} + +static void mpf_write_iso21496(AVIOContext *pb, MPFImage *src) +{ + avio_w8(pb, 0xFF); + avio_w8(pb, APP2); + + const int64_t start = avio_tell(pb); + avio_wb16(pb, src->ext_size[MPF_EXT_ISO21496]); + + /* The primary image contains a subset of the ISO 21496-1 struct, carrying + * just the version header (two 16-bit integers), but no payload */ + avio_write(pb, AV_ISO21496_IDENTIFIER, sizeof(AV_ISO21496_IDENTIFIER)); + avio_write(pb, src->packet->data + src->app2 + sizeof(AV_ISO21496_IDENTIFIER), + sizeof(uint16_t[2])); + + update_offset(pb, start, &src->ext_size[MPF_EXT_ISO21496]); +} + +static int mpf_write_extensions(AVFormatContext *ctx, AVIOContext *pb, + int64_t base, MPFImage *img) +{ + int ret; + + switch (img->role) { + case MPF_ROLE_PRIMARY: + if (ctx->nb_streams > 1) { + ret = mpf_write_index(ctx, pb, base, img); + if (ret < 0) + return ret; + } + break; + } + + for (int i = 0; i < img->nb_rev_deps; i++) { + MPFImage *rdep = img->rev_deps[i]; + switch (rdep->role) { + case MPF_ROLE_GAIN_MAP: + mpf_write_iso21496(pb, rdep); + break; + } + } + + return 0; +} + +static int mpf_write(AVFormatContext *ctx, AVIOContext *pb) +{ + MPFMuxContext *s = ctx->priv_data; + const int64_t base = avio_tell(pb); + int ret; + + for (int i = 0; i < ctx->nb_streams; i++) { + MPFImage *img = &s->images[i]; + const AVPacket *pkt = img->packet; + + update_offset(pb, base, &img->start); + avio_write(pb, pkt->data, img->splice_pos); + if ((ret = mpf_write_extensions(ctx, pb, base, img)) < 0) + return ret; + avio_write(pb, pkt->data + img->splice_pos, pkt->size - img->splice_pos); + update_offset(pb, base, &img->end); + } + + return 0; +} + +static int mpf_write_trailer(AVFormatContext *ctx) +{ + MPFMuxContext *s = ctx->priv_data; + int ret; + + for (int i = 0; i < ctx->nb_streams; i++) { + if (!s->images[i].packet->size) { + av_log(ctx, AV_LOG_ERROR, "No image was supplied on stream %d\n", i); + return AVERROR(EINVAL); + } + } + + /* Write the whole file once to a null buffer to settle offsets/sizes */ + AVIOContext *null; + if ((ret = ffio_open_null_buf(&null)) < 0) + return ret; + ret = mpf_write(ctx, null); + int size = ffio_close_null_buf(null); + if (ret < 0) + return ret; + else if (size < 0) + return size; + + for (int i = 0; i < ctx->nb_streams; i++) { + MPFImage *img = &s->images[i]; + for (int j = 0; j < MPF_EXT_NB; j++) { + if (img->ext_size[j] > UINT16_MAX) { + av_log(ctx, AV_LOG_ERROR, "Stream %d: APP2 segment %d is too " + "large (%"PRId64" bytes)\n", img->index, j, img->ext_size[j]); + return AVERROR(EINVAL); + } + } + } + + return mpf_write(ctx, ctx->pb); +} + +static void mpf_uninit(AVFormatContext *ctx) +{ + MPFMuxContext *s = ctx->priv_data; + if (!s->images) + return; + + for (int i = 0; i < ctx->nb_streams; i++) + av_packet_free(&s->images[i].packet); + + av_freep(&s->images); +} + +const FFOutputFormat ff_jpeg_mpf_muxer = { + .p.name = "jpeg_mpf", + .p.long_name = NULL_IF_CONFIG_SMALL("JPEG Multi-Picture Format"), + .p.mime_type = "image/jpeg", + .p.extensions = "jpg,jpeg", + .priv_data_size = sizeof(MPFMuxContext), + .p.flags = AVFMT_NOTIMESTAMPS | AVFMT_NODIMENSIONS, + .p.video_codec = AV_CODEC_ID_MJPEG, + .flags_internal = FF_OFMT_FLAG_ONLY_DEFAULT_CODECS, + .init = mpf_init, + .deinit = mpf_uninit, + .write_packet = mpf_write_packet, + .write_trailer = mpf_write_trailer, + .interleave_packet = ff_interleave_packet_passthrough, +}; -- 2.52.0 >From 3d64ddf45e298909651a832c41a6b401b5420f4b Mon Sep 17 00:00:00 2001 From: Niklas Haas <[email protected]> Date: Tue, 25 Aug 2026 13:52:26 +0200 Subject: [PATCH 8/8] tests/fate/image: add jpeg-mpf-gainmap test I've also pre-emptively grouped this into jpeg-mpf in anticipation of future MPF test cases. Signed-off-by: Niklas Haas <[email protected]> --- tests/fate/image.mak | 16 +++++++++++++++- tests/filtergraphs/jpeg_mpf_gainmap | 4 ++++ 2 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 tests/filtergraphs/jpeg_mpf_gainmap diff --git a/tests/fate/image.mak b/tests/fate/image.mak index e9fe059ead..17f151ddbd 100644 --- a/tests/fate/image.mak +++ b/tests/fate/image.mak @@ -620,6 +620,20 @@ FATE_IMAGE += $(FATE_IMAGE-yes) FATE_IMAGE_PROBE += $(FATE_IMAGE_PROBE-yes) FATE_IMAGE_TRANSCODE += $(FATE_IMAGE_TRANSCODE-yes) +FATE_JPEG_MPF-$(call ALLYES, LAVFI_INDEV TESTSRC2_FILTER SCALE_FILTER FORMAT_FILTER \ + SPLIT_FILTER SETPARAMS_FILTER GAINMAP_FILTER MJPEG_ENCODER JPEG_MPF_MUXER \ + FILE_PROTOCOL) += fate-jpeg-mpf-gainmap +fate-jpeg-mpf-gainmap: tests/data/filtergraphs/jpeg_mpf_gainmap +fate-jpeg-mpf-gainmap: CMD = md5 -bitexact -sws_flags +accurate_rnd+bitexact \ + -/filter_complex $(TARGET_PATH)/tests/data/filtergraphs/jpeg_mpf_gainmap \ + -map "[base]" -map "[gain]" -stream_group "type=gain_map:st=0:st=1:el_index=1" \ + -pix_fmt yuvj420p -c:v mjpeg -f jpeg_mpf +fate-jpeg-mpf-gainmap: CMP = oneline +fate-jpeg-mpf-gainmap: REF = 5d6d68c953693e26d022b4479ea3b08b + +FATE_FFMPEG += $(FATE_JPEG_MPF-yes) +fate-jpeg-mpf: $(FATE_JPEG_MPF-yes) + FATE_IMG2_MUXER-$(call ALLYES, LAVFI_INDEV COLOR_FILTER FORMAT_FILTER \ PGM_ENCODER IMAGE2_MUXER IMAGE2_DEMUXER FILE_PROTOCOL FFPROBE) \ += fate-img2-update-filemtime @@ -632,4 +646,4 @@ FATE_SAMPLES_FFMPEG += $(FATE_IMAGE) FATE_SAMPLES_FFPROBE += $(FATE_IMAGE_PROBE) FATE_SAMPLES_FFMPEG_FFPROBE += $(FATE_IMAGE_TRANSCODE) -fate-image: $(FATE_IMAGE) $(FATE_IMAGE_PROBE) $(FATE_IMAGE_TRANSCODE) $(FATE_IMG2_MUXER-yes) +fate-image: $(FATE_IMAGE) $(FATE_IMAGE_PROBE) $(FATE_IMAGE_TRANSCODE) $(FATE_IMG2_MUXER-yes) $(FATE_JPEG_MPF-yes) diff --git a/tests/filtergraphs/jpeg_mpf_gainmap b/tests/filtergraphs/jpeg_mpf_gainmap new file mode 100644 index 0000000000..8f778501cc --- /dev/null +++ b/tests/filtergraphs/jpeg_mpf_gainmap @@ -0,0 +1,4 @@ +testsrc2=s=64x48:r=1:d=1,setparams=color_trc=smpte2084:color_primaries=bt2020,scale,split [a][hdr]; +[a] setparams=color_trc=iec61966-2-1:color_primaries=bt709,split [b][sdr]; +[b] scale [base]; +[sdr][hdr] gainmap=mode=maxrgb,scale [gain] -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]