Re: Problem with NDISRC, WHIPSINK, and audio

Tarun Tej K via gstreamer-devel <[email protected]> Sat, 15 Jul 2023 17:06:24 +0530
Newsgroups	gmane.comp.video.gstreamer.devel
Message-ID	<CAMvyGd_FrNerwpZoGfvTZZWGvSqrtwAtTxuAYLRUeeP9gW2gLw@mail.gmail.com>
Ok. I am not familiar with ndisrcdemux, does this mean the buffers are
being dropped at the demuxer?
Also what is the GST_DEBUG value set in the environment and gstreamer
version you are using?

On Sat, Jul 15, 2023 at 4:45 PM GST Developer <[email protected]> wrote:
>
> besides the full log (second one) that i shared,   http://www.gallery.co.uk/gstlog2.txt.zip
>
>  TL:DR this appears to be the crux of the issue:
>
> Pipeline is live and does not need PREROLL ...
> 0:00:00.036747546 11562 0x558781cb5400 FIXME                default gstutils.c:4025:gst_pad_create_stream_id_internal:<ndisrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
> Pipeline is PREROLLED ...
> Setting pipeline to PLAYING ...
> New clock: GstSystemClock
> Redistribute latency...
> Redistribute latency...
> Redistribute latency...
> 0:00:01.220267971 11562 0x7fe028007330 WARN             ndireceiver net/ndi/src/ndisrc/receiver.rs:850:gstndi::ndisrc::receiver::Receiver::receive_thread:<ndisrc0> Dropping old buffer -- queue has 11 items
> 0:00:01.232472921 11562 0x7fe028007330 WARN             ndireceiver net/ndi/src/ndisrc/receiver.rs:850:gstndi::ndisrc::receiver::Receiver::receive_thread:<ndisrc0> Dropping old buffer -- queue has 11 items
> 0:00:01.239202906 11562 0x7fe028007330 WARN             ndireceiver net/ndi/src/ndisrc/receiver.rs:850:gstndi::ndisrc::receiver::Receiver::receive_thread:<ndisrc0> Dropping old buffer -- queue has 11 items
>
>
>
>
> On 15 Jul 2023, at 11:44, Tarun Tej K <[email protected]> wrote:
>
> This log snippet does not give much. information. Could you please share the full log?
>
> On Sat, 15 Jul, 2023, 14:00 GST Developer, <[email protected]> wrote:
>>
>> Thanks for your reply.
>>
>> Sadly, this doesn't change the behaviour - when attempting to send video *and* audio, it stalls after “Redistribute latency…”
>>
>> So, Just video works:
>> gst-launch-1.0 ndisrc ndi-name="NDIPE8 (SIGGEN)" !  ndisrcdemux name=demux demux.video ! queue ! videoconvert ! x264enc ! video/x-h264,format=byte-stream,profile=baseline ! rtph264pay !   'application/x-rtp,media=video,encoding-name=H264,payload=97,clock-rate=90000' !   whip.  whipsink name=whip   auth-token=“…redacted….."  whip-endpoint="https://director.millicast.com/api/whip/myStreamName”
>>
>>
>> and If I try *just* audio:
>> gst-launch-1.0 ndisrc ndi-name="NDIPE8 (SIGGEN)" !  ndisrcdemux name=demux demux.audio ! queue ! audioconvert ! opusenc ! rtpopuspay ! 'application/x-rtp,media=audio,encoding-name=OPUS,payload=96,clock-rate=48000,encoding-params=(string)2'  !  whip.  whipsink name=whip   auth-token=“...redacted..."  whip-endpoint="https://director.millicast.com/api/whip/myStreamName”
>> This also works !
>>
>> but when I try to combine them :
>>
>> gst-launch-1.0 ndisrc ndi-name="NDIPE8 (SIGGEN)" !  ndisrcdemux name=demux demux.video ! queue ! videoconvert ! x264enc ! video/x-h264,format=byte-stream,profile=baseline ! rtph264pay !   'application/x-rtp,media=video,encoding-name=H264,payload=97,clock-rate=90000' !  whip.  demux.audio ! queue ! audioconvert ! opusenc ! rtpopuspay ! 'application/x-rtp,media=audio,encoding-name=OPUS,payload=96,clock-rate=48000,encoding-params=(string)2'  !  whipsink name=whip   auth-token="…redacted….."  whip-endpoint="https://director.millicast.com/api/whip/myStreamName”
>>
>> it stalls after “Redistribute latency…”
>>
>> This does feel like some sort of demux deadlock between the ndisrcdemux and the whipsink.  I have pasted the debug output for the stall loop below, which repeats over and over:
>>
>> 0:00:19.425157202  9470 0x557d29708a00 TRACE              structure gststructure.c:292:gst_structure_new_id_empty_with_size: created structure 0x557d298c1aa0
>> 0:00:19.425181034  9470 0x557d29708a00 DEBUG              structure gststructure.c:2258:gst_structure_parse_field: trying field name 'long-name'
>> 0:00:19.433174494  9470 0x557d29708a00 DEBUG                default gstvalue.c:2776:_priv_gst_value_parse_value: trying type name 'string'
>> 0:00:19.433194340  9470 0x557d29708a00 DEBUG              structure gststructure.c:2258:gst_structure_parse_field: trying field name 'klass'
>> 0:00:19.433204782  9470 0x557d29708a00 DEBUG                default gstvalue.c:2776:_priv_gst_value_parse_value: trying type name 'string'
>> 0:00:19.433216613  9470 0x557d29708a00 DEBUG              structure gststructure.c:2258:gst_structure_parse_field: trying field name 'description'
>> 0:00:19.433223154  9470 0x557d29708a00 DEBUG                default gstvalue.c:2776:_priv_gst_value_parse_value: trying type name 'string'
>> 0:00:19.433232379  9470 0x557d29708a00 DEBUG              structure gststructure.c:2258:gst_structure_parse_field: trying field name 'author'
>> 0:00:19.433246113  9470 0x557d29708a00 DEBUG                default gstvalue.c:2776:_priv_gst_value_parse_value: trying type name 'string'
>> 0:00:19.433254115  9470 0x557d29708a00 DEBUG           GST_REGISTRY gstregistrychunks.c:627:gst_registry_chunks_load_feature: Element factory : npadtemplates=2
>> 0:00:19.433261379  9470 0x557d29708a00 DEBUG           GST_REGISTRY gstregistrychunks.c:526:gst_registry_chunks_load_pad_template: Reading/casting for GstRegistryChunkPadTemplate at address 0x7f3208f72f30
>> 0:00:19.433271543  9470 0x557d29708a00 DEBUG           GST_REGISTRY gstregistrychunks.c:540:gst_registry_chunks_load_pad_template: Added pad_template src
>> 0:00:19.433279923  9470 0x557d29708a00 DEBUG           GST_REGISTRY gstregistrychunks.c:526:gst_registry_chunks_load_pad_template: Reading/casting for GstRegistryChunkPadTemplate at address 0x7f3208f72fb0
>> 0:00:19.433289253  9470 0x557d29708a00 DEBUG           GST_REGISTRY gstregistrychunks.c:540:gst_registry_chunks_load_pad_template: Added pad_template sink
>> 0:00:19.433302373  9470 0x557d29708a00 DEBUG           GST_REGISTRY gstregistry.c:592:gst_registry_add_feature:<registry0> adding feature 0x557d298bf6d0 (avdec_wmavoice)
>> 0:00:19.433311993  9470 0x557d29708a00 DEBUG        GST_REFCOUNTING gstobject.c:708:gst_object_set_parent:<avdec_wmavoice> set parent (ref and sink)
>> 0:00:19.433322596  9470 0x557d29708a00 TRACE        GST_REFCOUNTING gstobject.c:292:gst_object_ref_sink:<avdec_wmavoice> 0x557d298bf6d0 ref_sink 1->2
>> 0:00:19.433330585  9470 0x557d29708a00 LOG             GST_REGISTRY gstregistry.c:610:gst_registry_add_feature:<registry0> emitting feature-added for avdec_wmavoice
>> 0:00:19.433341108  9470 0x557d29708a00 DEBUG           GST_REGISTRY gstregistrychunks.c:741:gst_registry_chunks_load_feature: Added feature avdec_wmavoice, plugin 0x557d298724d0 libav
>> 0:00:19.433350373  9470 0x557d29708a00 DEBUG           GST_REGISTRY gstregistrychunks.c:583:gst_registry_chunks_load_feature: Plugin 'libav' feature 'avdec_ws_snd1' typename : 'GstElementFactory'
>> 0:00:19.433362464  9470 0x557d29708a00 TRACE        GST_REFCOUNTING gstobject.c:208:gst_object_init:<GstObject@0x557d298c2050> 0x557d298c2050 new
>> 0:00:19.433374983  9470 0x557d29708a00 LOG             GST_REGISTRY gstregistrychunks.c:610:gst_registry_chunks_load_feature: Reading/casting for GstRegistryChunkElementFactory at address 0x7f3208f72ff0
>> 0:00:19.433385348  9470 0x557d29708a00 TRACE              structure gststructure.c:292:gst_structure_new_id_empty_with_size: created structure 0x557d298c1e40
>> 0:00:19.433394095  9470 0x557d29708a00 DEBUG              structure gststructure.c:2258:gst_structure_parse_field: trying field name 'long-name'
>> 0:00:19.433402420  9470 0x557d29708a00 DEBUG                default gstvalue.c:2776:_priv_gst_value_parse_value: trying type name 'string'
>> 0:00:19.433411420  9470 0x557d29708a00 DEBUG              structure gststructure.c:2258:gst_structure_parse_field: trying field name 'klass'
>> 0:00:19.433419783  9470 0x557d29708a00 DEBUG                default gstvalue.c:2776:_priv_gst_value_parse_value: trying type name 'string'
>>
>>
>>
>>
>> On 15 Jul 2023, at 01:17, Tarun Tej K <[email protected]> wrote:
>>
>> Hi,
>>
>> What is the idea behind fixing the sink pad number i.e., sink_0 to video and sink_1 to audio ? The pads would be linked dynamically anyway.
>>
>> Can you try the below modified pipeline. I have also added a queue in audio branch after demux.
>>
>> gst-launch-1.0 ndisrc ndi-name="NDIPE8 (SIGGEN)" !  ndisrcdemux name=demux demux.video ! queue ! videoconvert ! x264enc ! video/x-h264,format=byte-stream,profile=baseline ! rtph264pay !   'application/x-rtp,media=video,encoding-name=H264,payload=97,clock-rate=90000' !  whip. demux.audio ! queue ! audioconvert ! opusenc ! rtpopuspay ! 'application/x-rtp,media=audio,encoding-name=OPUS,payload=96,clock-rate=48000,encoding-params=(string)2'  !  whipsink name=whip   auth-token="…redacted….."  whip-endpoint="https://director.millicast.com/api/whip/myStreamName”
>>
>>
>> On Sat, 15 Jul, 2023, 03:30 GST Developer via gstreamer-devel, <[email protected]> wrote:
>>>
>>> Hi Folks.
>>>
>>> I am attempting use gstreamer to send content from an NDI Source to a WHIP end point at Dolby.io.
>>>
>>> If I just send VIDEO, its working fine:
>>>
>>> gst-launch-1.0 ndisrc ndi-name="NDIPE8 (SIGGEN)" !  ndisrcdemux name=demux demux.video ! queue ! videoconvert ! x264enc ! video/x-h264,format=byte-stream,profile=baseline ! rtph264pay !   'application/x-rtp,media=video,encoding-name=H264,payload=97,clock-rate=90000' !   whip.sink_0 whipsink name=whip   auth-token=“…redacted….."  whip-endpoint="https://director.millicast.com/api/whip/myStreamName”
>>>
>>> This works FINE !! we get the signal to the WHIP server and the end to end latency is about 2 seconds.
>>>
>>> Now I want to add AUDIO to that pipeline, and I am trying:
>>>
>>> gst-launch-1.0 ndisrc ndi-name="NDIPE8 (SIGGEN)" !  ndisrcdemux name=demux demux.video ! queue ! videoconvert ! x264enc ! video/x-h264,format=byte-stream,profile=baseline ! rtph264pay !   'application/x-rtp,media=video,encoding-name=H264,payload=97,clock-rate=90000' !  whip.sink_0 demux.audio ! audioconvert ! opusenc ! rtpopuspay ! 'application/x-rtp,media=audio,encoding-name=OPUS,payload=96,clock-rate=48000,encoding-params=(string)2'  ! whip.sink_1  whipsink name=whip   auth-token="…redacted….."  whip-endpoint="https://director.millicast.com/api/whip/myStreamName”
>>>
>>> Now, it starts up with:
>>>
>>> Setting pipeline to PAUSED ...
>>> Pipeline is live and does not need PREROLL ...
>>> Pipeline is PREROLLED ...
>>> Setting pipeline to PLAYING ...
>>> New clock: GstSystemClock
>>> Redistribute latency...
>>> Redistribute latency...
>>> Redistribute latency…
>>>
>>> But it stalls right there, and never gets to counting time, and we dont see anything arrive at the WHIP server.
>>>
>>> Might anyone know what I am doing wrong ?
>>>
>>> Many thanks !!!
>>>
>>>
>>>
>>
>