Re: v4l2: No handling of no-show frames in decoder plugin

--000000000000fb80ca062f28a984
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Patrik,

Le mer. 26 f=C3=A9vr. 2025, 06 h 30, Pratik Pachange (QUIC) via gstreamer-d=
evel <
[email protected]> a =C3=A9crit :

>
>
> Hi Team,
>
>
>
> For AV1 codec format, there are certain no-show frames (show_frame flag
> set to 0) given by the video driver/fw as decoded output which needs to
> dropped/discarded by the client.
> On v4l2 side, the only way to identify such buffers is that their
> bytesused is 0 (or filledlen is 0).
>

The only AV1 decoder we have in upstream GStreamer is the stateless
decoder, which does not have this issue. I know some HW vendor added AV1 in
downstream kernel, without ever upstreaming it and getting their design
reviewed. This obviously lead to mistakes like this.

That being said, I'm not against fixing it with your suggestions. Though
some more thought needs to placed into that since that clashes with the
legacy way to signal EOS, and we also need to make sure it aligns with the
rest of V4L specific mainline.

An alternative is to define V4L2 AV1 as TU aligned. Then multiple frame
gets submitted to the decoder, and only one is to be produced. The downside
is that bitstream buffer size estimation will fall appart, since we have no
idea how many frames a bitstream (OUTPUT queue buffer) may endup.

Since this is a well know problem surface, we could also consider adding
signalling in the av1parse element.

>
> Currently in v4l2 plugins, there is no handling to drop such buffers.
> Can we plan to add any handling to drop/discard empty buffers (bytesused
> 0).
>

We do have some code to drop very old GstVideoFrame objects, but it takes
quite some time. It was added to workaround similar issues when with
interlaced video (H.264) mostly. Assuming it works with H.264 interlaced,
and does not go against the spec, I'll be happy to review and merge your
patches. Use the following to see the contribution manual.

https://gstreamer.freedesktop.org/documentation/contribute/index.html

Nicolas

>

--000000000000fb80ca062f28a984
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto"><div>Hi Patrik,<br><br><div class=3D"gmail_quote gmail_qu=
ote_container"><div dir=3D"ltr" class=3D"gmail_attr">Le mer. 26 f=C3=A9vr. =
2025, 06 h 30, Pratik Pachange (QUIC) via gstreamer-devel &lt;<a href=3D"ma=
ilto:[email protected]">[email protected]=
op.org</a>&gt; a =C3=A9crit=C2=A0:<br></div><blockquote class=3D"gmail_quot=
e" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">





<div lang=3D"EN-US" link=3D"#467886" vlink=3D"#96607D" style=3D"word-wrap:b=
reak-word">
<div class=3D"m_-8756859038899525037WordSection1">
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Hi Team,<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">For AV1 codec format, there are certain no-show fram=
es (show_frame=C2=A0flag set to 0) given by the video driver/fw as decoded =
output which needs to dropped/discarded by the client.<br>
On v4l2 side, the only way to identify such buffers is that their bytesused=
 is 0 (or filledlen is 0).</p></div></div></blockquote></div></div><div dir=
=3D"auto"><br></div><div dir=3D"auto">The only AV1 decoder we have in upstr=
eam GStreamer is the stateless decoder, which does not have this issue. I k=
now some HW vendor added AV1 in downstream kernel, without ever upstreaming=
 it and getting their design reviewed. This obviously lead to mistakes like=
 this.</div><div dir=3D"auto"><br></div><div dir=3D"auto">That being said, =
I&#39;m not against fixing it with your suggestions. Though some more thoug=
ht needs to placed into that since that clashes with the legacy way to sign=
al EOS, and we also need to make sure it aligns with the rest of V4L specif=
ic mainline.</div><div dir=3D"auto"><br></div><div dir=3D"auto">An alternat=
ive is to define V4L2 AV1 as TU aligned. Then multiple frame gets submitted=
 to the decoder, and only one is to be produced. The downside is that bitst=
ream buffer size estimation will fall appart, since we have no idea how man=
y frames a bitstream (OUTPUT queue buffer) may endup.</div><div dir=3D"auto=
"><br></div><div dir=3D"auto">Since this is a well know problem surface, we=
 could also consider adding signalling in the av1parse element.</div><div d=
ir=3D"auto"><div class=3D"gmail_quote gmail_quote_container"><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;p=
adding-left:1ex"><div lang=3D"EN-US" link=3D"#467886" vlink=3D"#96607D" sty=
le=3D"word-wrap:break-word"><div class=3D"m_-8756859038899525037WordSection=
1"><p class=3D"MsoNormal"><u></u><u></u></p>
<p class=3D"MsoNormal"><br>
Currently in v4l2 plugins, there is no handling to drop such buffers. <br>
Can we plan to add any handling to drop/discard empty buffers (bytesused 0)=
.</p></div></div></blockquote></div></div><div dir=3D"auto"><br></div><div =
dir=3D"auto">We do have some code to drop very old GstVideoFrame objects, b=
ut it takes quite some time. It was added to workaround similar issues when=
 with interlaced video (H.264) mostly. Assuming it works with H.264 interla=
ced, and does not go against the spec, I&#39;ll be happy to review and merg=
e your patches. Use the following to see the contribution manual.</div><div=
 dir=3D"auto"><br></div><div dir=3D"auto"><a href=3D"https://gstreamer.free=
desktop.org/documentation/contribute/index.html">https://gstreamer.freedesk=
top.org/documentation/contribute/index.html</a></div><div dir=3D"auto"><br>=
</div><div dir=3D"auto">Nicolas</div><div dir=3D"auto"><div class=3D"gmail_=
quote gmail_quote_container"><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang=3D"EN=
-US" link=3D"#467886" vlink=3D"#96607D" style=3D"word-wrap:break-word"><div=
 class=3D"m_-8756859038899525037WordSection1"><p class=3D"MsoNormal">
<u></u><u></u></p>
</div>
</div>

</blockquote></div></div></div>

--000000000000fb80ca062f28a984--