webrtcsink cannot stream h264 [Was:Re: Choppy audio playing a videofile through webrtcsink]

Hi,

To avoid filing a duplicate, I added info and log to this existing issue :
https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/411

The minimal test case is:
gst-launch-1.0 uridecodebin uri=``
https://download.blender.org/durian/trailer/sintel_trailer-480p.mp4 !
videoconvert ! webrtcsink
which throws an error while :
gst-launch-1.0 uridecodebin uri=``
https://download.blender.org/durian/trailer/sintel_trailer-480p.ogv !
videoconvert ! webrtcsink
(replacing .mp4 by .ogv)
works...

Kind regards,

Vincent

On Mon, Nov 20, 2023 at 3:07 PM Mathieu Duponchelle <[email protected]>
wrote:

> webrtcsink should be able to handle pre-encoded H264, please file an
> issue if you have a minimal test case for reproducing your issue :)
>
> On Sun, 2023-11-19 at 03:10 +0100, Vincent Deconinck via gstreamer-
> devel wrote:
> > Hi,
> >
> > I'm still stuck with the inability to stream h264 video to
> > webrtcsink, but I've done a bit of research.
> >
> > The issue is similar to this one albeit in slightly different
> > context:
> > https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/411
> > "livekitwebrtcsink: failure to negotiate h264"
> > Starting from gstreamer's most obvious error log "There is no codec
> > present that can handle the stream's type.", we get to webrtcsink's
> > error "Error running discovery: No caps found for stream video_0"
> > triggered by start_stream_discovery_if_needed (in webrtcsink\imp.rs)
> > Increasing the log level, this error is preceeded by 4 times the
> > following warning:
> > net\webrtc\src\webrtcsink\imp.rs:3226:gstrswebrtc::webrtcsink::imp::B
> > aseWebRTCSink::lookup_caps::{{closure}}:<wsink> Codec discovery
> > pipeline failed: Linking encoding elements
> > The message "Codec discovery pipeline failed" is logged from
> > lookup_caps(), and if I understand this code correctly, there are two
> > cases:
> > - either "Stream is already encoded with [a supported] codec", and it
> > works out of the box without trying to look up caps (that's what
> > happens when input is vp8)
> > - or the incoming stream uses another codec (h264 or mpeg2video in my
> > tests), and the code tries to enumerate the caps, which fails.
> > Does this make sense ?
> >
> > I'm willing to help (I have zero Rust experience but always eager to
> > learn) and I was able to rebuild the plugin from the sources, but I
> > don't know how to tell gstreamer to use my home-brewed webrtcsink
> > instead of the one in the gstreamer distribution.
> > Is it statically linked in the gstreamer build ?
> >
> > Any help would be greatly appreciated.
> >
> > KR,
> >
> > Vincent
> >
> >
> > On Fri, Nov 17, 2023 at 1:13 AM Vincent Deconinck
> > <[email protected]> wrote:
> > > Hi,
> > >
> > > Thanks for all the insightful answers. The log suggestion and the
> > > audio frame timing issue triggered the idea of a few more tests.
> > >
> > > Here are the tests I did today:
> > > 1. (set logging with GST_DEBUG=5). Ran the gstreamer server and the
> > > browser with "choppy audio". The log is 350+MB and file writing
> > > clearly impacts the playback (audio choppiness is increased and
> > > video becomes choppy too). AFAICT however, the log shows no ERROR
> > > until the end of the video ("pause task, reason: Flushing"). The
> > > zipped log is available on https://we.tl/t-n3kLKluEXE
> > > 2. (still logging with GST_DEBUG=5) Changed the pipeline to only
> > > use the audio from the webm trailer video file, and used a
> > > videotestsrc as the video source => audio is still very choppy
> > > 3. (still logging with GST_DEBUG=5) Extracted the audio to a plain
> > > uncompressed wav file (obtained with "ffmpeg -i sintel_trailer-
> > > 480p.webm sintel_trailer-480p.wav") and used this as the audio
> > > source and the videotestsrc as the video source => audio is still
> > > very choppy
> > > 4. (no logging from here on) Same test: WAV file as the audio
> > > source and the videotestsrc as the video source => audio is perfect
> > > !
> > > 5. Back to using the webm file as the audio source and the
> > > videotestsrc as the video source => audio is choppy
> > > 6. Converted the source file (vp8+vorbis) to mpeg4 (h264+aac) with
> > > "ffmpeg -i sintel_trailer-480p.webm sintel_trailer-480p.mp4" and
> > > used this as the audio source and the videotestsrc as the video
> > > source => audio is "almost" perfect (I can still hear a "drop" in
> > > the second drum at 00:16, which is not present when playing the mp4
> > > file locally with VLC).
> > > 7. Used the mp4 as source for both video and audio => Fails with
> > > "Error received from element wsink: There is no codec present that
> > > can handle the stream's type.". Seems h264 video is not recognized
> > > ?
> > > 8. Converted the source file (vp8+vorbis) to a ts (mpeg2video+ac3)
> > > with "ffmpeg -i sintel_trailer-480p.webm -acodec ac3 -vcodec
> > > mpeg2video sintel_trailer-480p.ts" and used it for both video and
> > > audio => Fails with "There is no codec present that can handle the
> > > stream's type". Seems mpeg2video is also not recognized ?
> > > 9. Baked an alternate webm file using opus instead of vorbis with
> > > "ffmpeg -i sintel_trailer-480p.webm -strict -2 -acodec opus -vcodec
> > > copy sintel_trailer-480p_opus.webm" => audio/video is perfect !
> > >
> > > OK. What did I learn ?
> > > - a lack of resources (e.g. heavy logging) causes choppiness. My PC
> > > is quite good though (core i9 3.6GHz 32GB w/ GeForce RTX 2060) but
> > > maybe logging chokes gsteamer.
> > > Regarding audio codecs :
> > > - Vorbis causes atrocious choppiness.
> > > - AAC is almost OK
> > > - Wav and Opus are perfect
> > > Regarding video codecs:
> > > - It seems this pipeline only accepts vp8 as input
> > >
> > > Questions:
> > > My source files being either mpeg4 (with aac or ac3 audio) or even
> > > MXF (and conversion beforehand not being an option),
> > > - How can I convert the videos on the fly ?
> > > - Isn't vconvert meant to perform the conversion if required ?
> > >
> > > Sorry again for the newbie questions...
> > >
> > > Kind regards,
> > >
> > > Vincent
> > >
> > > On Thu, Nov 16, 2023 at 7:07 PM cfd new <[email protected]> wrote:
> > > >
> > > > Thank you so much for your detailed mail. Very nice!
> > > >
> > > >     Joe
> > > >
> > > >
> > > >
> > > >
> > > >  On Thursday, November 16, 2023, 01:00:44 p.m. EST, cfd new via
> > > > gstreamer-devel <[email protected]> wrote:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Hi, Philipp,
> > > >
> > > >    very interested in your code. I got the following message when
> > > > I stream a live video to youtube. Maybe it is related to your
> > > > case.
> > > >
> > > >    Joe
> > > > 0:03:35.140304346 21323 0x7f1fa004c240 WARN
> > > > audioresample
> > > > gstaudioresample.c:732:gst_audio_resample_check_discont:<audiores
> > > > ample1> encountered timestamp discontinuity of 639 samples =
> > > > 0:00:00.039937500
> > > > 0:03:35.141001930 21323 0x7f1fa004c240 WARN
> > > > audioresample
> > > > gstaudioresample.c:732:gst_audio_resample_check_discont:<audiores
> > > > ample1> encountered timestamp discontinuity of 639 samples =
> > > > 0:00:00.039937500
> > > > 0:03:35.141314636 21323 0x7f1fa004c240 WARN
> > > > audioresample
> > > > gstaudioresample.c:732:gst_audio_resample_check_discont:<audiores
> > > > ample1> encountered timestamp discontinuity of 639 samples =
> > > > 0:00:00.039937500
> > > >
> > > >
> > > >
> > > >
> > > >  On Thursday, November 16, 2023, 06:35:55 a.m. EST, Philipp B via
> > > > gstreamer-devel <[email protected]> wrote:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > > [dislaimer: I didnt work with gstreamer for some time, my
> > > > terminology
> > > > might be a bit rusty]
> > > >
> > > > I also had audio issues when streaming video files over webrtc,
> > > > and
> > > > judging by your video, it could be the same issue you face.
> > > >
> > > > For me it turned out to be small rounding errors in the audio
> > > > frame
> > > > timestamps. Due to the realtime nature, browsers playing webrtc
> > > > audio
> > > > are not aiming for a perfect smooth playback, but rather for
> > > > perfect
> > > > realtime. Small deviations in timestamps instantly lead to the
> > > > browser
> > > > inserting small pieces of silence, or dropping a few ms of audio.
> > > >
> > > > The most obvious effect was the volume being way lower than it
> > > > should
> > > > be. This was caused by the player/browser constantly fading
> > > > segments
> > > > in and out, to overlap them with the next one, or with silence.
> > > > Also,
> > > > I had similar distortion artifacts than you have.
> > > >
> > > > To be clear, my issue seemed to be related to transmission of
> > > > non-live, pre-encoded video only. WebRTC is made for live
> > > > transmissions, where audio frame timestamps are directly related
> > > > to
> > > > the wall clock time at which a segment has been recorded. In
> > > > contrast,
> > > > a non-live video file typically has a stream of audio segments,
> > > > which
> > > > are expected to be played back without gaps, even when there are
> > > > small
> > > > gaps according to the time stamps.
> > > >
> > > > What happens if you increase the audio frame size? In my case,
> > > > this
> > > > was improving the situation dramatically, but not fully fixing
> > > > it.
> > > >
> > > > In the end, I got this solved by "smoothing" audio time stamps. I
> > > > wrote my own gstreamer element, that kept an estimation of the
> > > > next
> > > > frames timestamp, based on the last frame. In case, there is a
> > > > small
> > > > deviation (e.g. below 5 ms), the frames timestamp will be changed
> > > > to
> > > > that of the estimation. I can provide the code of that element in
> > > > case
> > > > your interested.
> > > >
> > > > Philipp
> > > >
> > > > Am Mi., 15. Nov. 2023 um 23:39 Uhr schrieb Vincent Deconinck via
> > > > gstreamer-devel <[email protected]>:
> > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I made some progress in my quest to stream video files to a
> > > > > browser using webrtcssink, but audio is not OK. I built the
> > > > > following pipeline:
> > > > > +--------------------------------------------------------------
> > > > > ---------------+
> > > > > | uridecodebin.audio ->- audioconvert ->- audioresample ->-
> > > > > webrtcssink.audio |
> > > > > | uridecodebin.video ----->----- videoconvert ------>------
> > > > > webrtcssink.video |
> > > > > +--------------------------------------------------------------
> > > > > ---------------+
> > > > >
> > > > > The webrtcsink pads are statically linked, and the uridecodebin
> > > > > pads are dynamically linked upon "pad_added" signal (as in
> > > > >
> https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html
> > > > > ).
> > > > >
> > > > > Problem: The video looks OK in the browser, but the audio is
> > > > > choppy. A quick and dirty (sorry) video tells a thousand words:
> > > > > https://youtu.be/1EbPHLZvVLc?t=27
> > > > >
> > > > > (using signalling server, JS API and gstreamer built 2 days ago
> > > > > from git HEAD)
> > > > >
> > > > > I was first using the http URI to the source video but I also
> > > > > tried using a local copy of the file, to no avail.
> > > > >
> > > > > Any idea what could be wrong ? Or an idea how I could track
> > > > > down that issue ?
> > > > >
> > > > > Kind regards,
> > > > >
> > > > > Vincent
> > > >
> > > >
> > > >
> > > >
>