[RFC] Premultiplied filter audit

--===============4295538142443366300==
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="mop2RHMJqvaA9k+U"
Content-Disposition: inline


--mop2RHMJqvaA9k+U
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi everyone

I have used claude (Fable 5) for auditing our filters for premultiplied sup=
port
Niklas, has written the Prompt (see below)
and ive used claude to create a table based on it and run it over the filte=
rs starting with a*
ive also run an adverserical agent over it trying to poke holes in the choi=
ces

My main question to you is, should we change something on the prompt or des=
ign?
Should we run this over all the filters ? (that is, will this give us enoug=
h confidence on what to set each filter to)

Prompt:
    Go through the list of filters in libavfilter/ and categorize them base=
d on how they would interact with premultiplied alpha.

    Categorize filters into the following types:
    * Color-agnostic filters that don't touch or read the pixel data at all=
 (e.g. truly metadata-only filters)
    * Metadata-only filters that read, but don't write, pixel values (e.g. =
measuring a histogram). Subcategorize as best you can into filters that wou=
ld prefer premul, prefer independent, or genuinely don't care. (Though note=
 that this is ultimately a subjective judgement call in many cases)
    * Filters modify pixel data in a way that's truly invariant under alpha=
 premultiplication (e.g. applying a linear scaling factor or swapping RGB c=
hannels), and thus permute under (premultiply,filter) -> (filter,premultipl=
y)
    * "Alpha-dependent" filters that inherently preserve the rgb <=3D alpha=
 property, and would thus work with either alpha mode, but may give differe=
nt results under premultiplied vs nonpremultiplied by virtue of some non-li=
nearity that affects how pixel values are processed. These would be filters=
 that *don't* commute under premultiplication. This includes e.g. scaling f=
ilters, crossfades, and other "linear"-type functions of several pixels tha=
t mix channels according to the same relative weights, as well as anything =
that applies a monotonic (non-linear) function to all channels (including A=
). Again, subcaterorize them as best you can into their alpha mode preferen=
ce. (e.g. scaling and crossfading should always be done with premultiplied =
alpha, to avoid e.g. bleeding transparent pink padding pixels into actual p=
ixel data)
    * Premul-incompatible filters that would definitely give broken results=
 (e.g. rgb > alpha) when fed premultiplied pixel values. This would include=
 any filter that does some nontrivial per-channel processing that might bre=
ak the alpha-scaling, e.g. a brightening filter that adds some offset to th=
e RGB values, or applying a nonlinear transfer function to only the R,G,B c=
hannels while leaving the A channel intact.

    For filters that may span multiple types depending on the selected opti=
ons, list them under both headings; and clarify under which condition/optio=
n it belongs to that category. Flag edge cases and filters you're unsure ab=
out explicitly for human review.

Adverserial agent finding:
  1. alphamerge =E2=80=94 wrong mechanism and wrong I/O. The straight-alpha=
 pin is on the output only and does not propagate to the inputs: a premulti=
plied main input is negotiated as premul and then merely relabeled straight=
 in do_alphamerge =E2=80=94 no
  unpremultiply is inserted upstream, contrary to what the row claimed. Als=
o the filter has two video inputs, so I/O is now VV->V, not V->V.
  2. avgblur-family E conditions were proper subsets of the truth. Not just=
 "planes without bit 3": any mask that blurs A and color unequally (e.g. pl=
anes=3D8, alpha blurred under sharp premul RGB) is E; planes=3D0xF is the o=
nly D mask. All three D/E row
  pairs reworded.
  3. avgblur_vulkan "OOB reads return 0" was unsupported. FFmpeg enables no=
 Vulkan robustness feature, so out-of-bounds imageLoad is undefined per spe=
c; zero-return is common driver behavior, not a guarantee. Row now says so =
and keeps a human-review
  flag.
  4. Header convention gap: Alpha pref was undefined for E rows =E2=80=94 n=
ow documented as "the only mode in which the filter is safe".

And the current results
# libavfilter video filters: premultiplied-alpha survey

Every filter from libavfilter/allfilters.c with a video INPUT (prefixes vf,
vaf, vsink), FFmpeg master 88bc27643a, 392 filters. Pure sources and
audio-to-video visualizers are excluded.

The I/O column is derived from the extern symbol prefix; a few multi-media
filters carry more streams than shown (e.g. concat is N media in/out).

Categories (how the filter behaves on premultiplied-alpha input):

- A-agnostic:  does not read or write pixel data at all (truly metadata-onl=
y)
- B-meta:      reads but never writes pixels (measurements); Alpha pref say=
s on
               which representation the measurement is more meaningful
- C-invariant: pixel changes commute with premultiplication
               (premultiply,filter) =3D=3D (filter,premultiply)
- D-alpha-dep: preserves rgb <=3D alpha, works in either mode, but does NOT
               commute (resampling/mixing with shared weights, monotonic
               functions applied to all channels incl. A); Alpha pref says
               which mode gives the better result (e.g. scaling: premul)
- E-incompat:  can produce broken results (rgb > alpha) on premultiplied in=
put
               (per-channel offsets/nonlinearities on RGB with A untouched,
               writing A independently of RGB)
- sink:        consumes frames

Option-dependent filters get one row per category with the condition in Not=
es.

Columns:

- Alpha formats: whether the filter accepts/produces pixel formats with an
  alpha channel at all ("none" =3D alpha never passes through it)
- Alpha pref:  n/a | premul | straight | dont-care | option-dependent
               (for E rows: the only mode in which the filter is safe)
- Confidence:  high | medium | low (how sure the categorization is)
- Checked:     empty until a human or an adversarial agent verified the row
| Filter              | I/O        | Alpha formats                | Categor=
y    | Alpha pref       | Confidence | Checked          | Notes
|---------------------|------------|------------------------------|--------=
-----|------------------|------------|------------------|-------
| addroi              | V->V       | all (no constraint)          | A-agnos=
tic  | n/a              | high       | agent 2026-08-05 | edits ROI side da=
ta only; AVFILTER_FLAG_METADATA_ONLY, never touches frame->data
| alphaextract        | V->V       | all alpha fmts               | C-invar=
iant | dont-care        | high       | agent 2026-08-05 | copies the A plan=
e only, output has no alpha -- premultiplied input changes nothing; graph s=
till pins input straight (constraint shared with extractplanes)
| alphamerge          | VV->V      | YUVA420/22/44,GBRAP,RGBA-fam | E-incom=
pat  | straight         | high       | agent 2026-08-05 | overwrites A verb=
atim from 2nd input luma, RGB untouched -> rgb>alpha on premul; pins alpha_=
mode=3Dstraight on the OUTPUT only -- the pin does not propagate to the inp=
uts, a premul main input is just relabeled straight (no conversion inserted)
| amplify             | V->V       | planar YUVA/GBRAP 8-16,f32   | E-incom=
pat  | straight         | high       | agent 2026-08-05 | temporal push fro=
m mean, threshold-gated per plane; default planes=3D7 leaves A untouched; n=
onlinear per-plane even with planes=3D15 -> still E
| ass                 | V->V       | all alpha fmts (drawutils)   | E-incom=
pat  | straight         | high       | agent 2026-08-05 | default alpha=3D0=
: A plane excluded from blending, subtitle RGB written under transparent pr=
emul px -> rgb>alpha; harmless on opaque video
| ass                 | V->V       | all alpha fmts (drawutils)   | D-alpha=
-dep | option-dependent | high       | agent 2026-08-05 | alpha=3D1: A blen=
ded, src color premultiplied when negotiated mode=3Dpremul; blend carries a=
_c^2 term (non-textbook OVER, exact only for opaque subtitle colors) -- con=
firmed in drawutils blend_pixel/ff_blend_mask
| atadenoise          | V->V       | planar YUVA/GBRAP 8-16       | E-incom=
pat  | straight         | high       | agent 2026-08-05 | default planes=3D=
7: color denoised per-plane (frame subsets differ per channel), A copied ->=
 rgb>alpha reachable; planes=3D15: plane-3 ptrs never filled -> NULL deref,=
 confirmed by 2nd independent code read, never executed -- human test wanted
| avgblur             | V->V       | planar YUVA/GBRAP 8-16       | D-alpha=
-dep | premul           | high       | agent 2026-08-05 | planes=3D0xF (def=
ault) is the ONLY D mask: box blur, uniform weights shared across all plane=
s incl. A; 420/422 chroma footprint differs from A plane
| avgblur             | V->V       | planar YUVA/GBRAP 8-16       | E-incom=
pat  | straight         | high       | agent 2026-08-05 | any mask blurring=
 A and color unequally (e.g. 7: A stays sharp; 8: only A blurred) -> rgb>al=
pha at edges
| avgblur_opencl      | V->V       | hw (device-dependent)        | D-alpha=
-dep | premul           | medium     | agent 2026-08-05 | planes=3D0xF (def=
ault) is the ONLY D mask: uniform box kernel per plane, edge-renormalized; =
packed RGBA is one image, A never blurred separately (bit 0 gates the whole=
 pixel)
| avgblur_opencl      | V->V       | hw (device-dependent)        | E-incom=
pat  | straight         | medium     | agent 2026-08-05 | any mask blurring=
 a planar alpha plane and color unequally (e.g. 7 or 8 on GBRAP) -> rgb>alp=
ha
| avgblur_vulkan      | V->V       | hw (device-dependent)        | D-alpha=
-dep | premul           | medium     | agent 2026-08-05 | planes=3D0xF (def=
ault) is the ONLY D mask: shared filter_norm for all channels; border image=
Load is OOB and FFmpeg enables no Vulkan robustness -- undefined per spec, =
common driver zero-return hits RGB and A equally (invariant kept if so) -- =
human review
| avgblur_vulkan      | V->V       | hw (device-dependent)        | E-incom=
pat  | straight         | medium     | agent 2026-08-05 | any mask blurring=
 a planar alpha plane and color unequally (e.g. 7 or 8 on GBRAP) -> rgb>alp=
ha
| backgroundkey       | V->V       |                              |        =
     |                  |            |                  |
| bbox                | V->V       |                              |        =
     |                  |            |                  |
| bench               | V->V       |                              |        =
     |                  |            |                  |
| bilateral           | V->V       |                              |        =
     |                  |            |                  |
| bilateral_cuda      | V->V       |                              |        =
     |                  |            |                  |




--=20
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Take away the freedom of one citizen and you will be jailed, take away
the freedom of all citizens and you will be congratulated by your peers
in Parliament.

--mop2RHMJqvaA9k+U
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iF0EABEKAB0WIQSf8hKLFH72cwut8TNhHseHBAsPqwUCanOBYQAKCRBhHseHBAsP
q2y4AJ9FcunVUqss/wV4nZJMj1hhDUcpEgCglsvyOmLoAZPCKF9ojpUZwCpOg4M=
=Hvks
-----END PGP SIGNATURE-----

--mop2RHMJqvaA9k+U--

--===============4295538142443366300==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]

--===============4295538142443366300==--