Re: GStreamer playback of MP4 files with and without audio
Gareth Alldread via gstreamer-devel <[email protected]> Thu, 3 Oct 2024 09:24:33 +0100
| Newsgroups | gmane.comp.video.gstreamer.devel |
|---|---|
| Message-ID | <CAENg58osT5ChaTZQSkym3Ubxxwsjdc5Jiug_7Rj8-ME+6Em6pA@mail.gmail.com> |
--0000000000006d86bc06238e49a8 Content-Type: multipart/alternative; boundary="0000000000006d86bb06238e49a6" --0000000000006d86bb06238e49a6 Content-Type: text/plain; charset="UTF-8" I haven't got experience with filesrc - but I attached my example doing something similar with uridecodebin (which will work with an video file btw). I have stripped this out of my app - so it is encapsulated inside a class - but I'm sure you get the idea. On Wed, 2 Oct 2024 at 17:46, Terry Barnaby via gstreamer-devel < [email protected]> wrote: > I have tried using g_signal_connect(m_sourceElement, "pad-added", ...) and > within that adding a sub pipeline to the main pipeline, but I am getting > something wrong when I add the sub pipeline. > > My simple program attached, sees the signal callback, but when I add the > sub pipeline and link it to the new pad, the displayed video just freezes. > I've looked at the "Basic tutorial 3: Dynamic pipelines" and it looks like > I am doing the right thing and no errors are seen. > > Any ideas on what I am ding wrong ? > > > On 02/10/2024 08:26, Gareth Alldread via gstreamer-devel wrote: > > I would add a handler for the "pad-added" event on the filesrc element and > only create/add the audio part of the pipeline if you get an audio pad > created. That is what I have done to handle rtspsrc streams that may or > may not have audio. Reply if you want more details. > > g_signal_connect(m_sourceElement, "pad-added", > G_CALLBACK(&SynxRTSPPipeline::onPadAdded), this); > > On Wed, 2 Oct 2024 at 07:30, Terry Barnaby via gstreamer-devel < > [email protected]> wrote: > >> I am developing a C++ video inspection program that creates MP4/H264/MP3 >> files and needs to play them back with the video stream processed by >> various gstreamer elements and audio separately. >> >> In general this has been working fine when just video was being >> recorded/played back, but I am just adding audio to the mix. Some MP4 files >> will contain only a video stream and some will contain both video and audio >> streams. >> >> I need some way to handle the playback of these MP4 files that may or may >> not have MP3 audio streams. >> >> As a simple idea if I use something like (The real C++ code constructs >> the gstreamer piple line and tees the video stream to various gstreamer sub >> pipelines): >> >> gst-launch-1.0 -v filesrc location=temp.mp4 ! qtdemux name=demux >> >> demux.video_0 ! queue ! h264parse ! openh264dec ! glimagesink >> >> demux.audio_0 ! queue ! decodebin ! audioconvert ! pulsesink >> >> This plays back the video and audio streams fine (not sure how well >> synchronised?) from an MP4 with video and audio streams, but hangs if the >> MP4 only has a video stream. >> >> So I think I need to: >> >> 1. Maybe there is some gstreamer element or attribute that can ignore the >> audio stream if not present somehow ? >> >> 2. Check if the MP4 file has an audio stream before creating the >> gstreamer pipeline in C++. >> >> 3. Create the basic C++ pipleline in C++ and interrogate pads or >> something somehow and add the "demux.audio_0 ! queue ! decodebin ! >> audioconvert ! pulsesink" sub pipeline if it is seen an audio stream is >> present. >> Any ideas on the simplest/most CPU efficient way of doing this with >> gstreamer ? >> >> >> > --0000000000006d86bb06238e49a6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>I haven't got experience with filesrc - but I att= ached my example doing something similar with uridecodebin (which will work= with an video file btw).=C2=A0 I have stripped this out of my app - so it = is encapsulated inside a class - but I'm sure you get the idea.<br></di= v><div><br></div><div><br></div></div><br><div class=3D"gmail_quote"><div d= ir=3D"ltr" class=3D"gmail_attr">On Wed, 2 Oct 2024 at 17:46, Terry Barnaby = via gstreamer-devel <<a href=3D"mailto:[email protected]= .org">[email protected]</a>> wrote:<br></div><blockq= uote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1p= x solid rgb(204,204,204);padding-left:1ex"><u></u> =20 =20 =20 <div> <div>I have tried using g_signal_connect(m_sourceElement, "pad-added", ...) and wit= hin that adding a sub pipeline to the main pipeline, but I am getting something wrong when I add the sub pipeline.</div> <div><br> </div> <div>My simple program attached, sees the signal callback, but when I add the sub pipeline and link it to the new pad, the displayed video just freezes.</div> <div>I've looked at the "Basic tutorial 3: Dynamic pipelines" and it looks like I am doing the right thing and no errors are seen.</div> <div><br> </div> <div>Any ideas on what I am ding wrong ?<br> </div> <div><br> </div> <div><br> </div> <div>On 02/10/2024 08:26, Gareth Alldread via gstreamer-devel wrote:<br> </div> <blockquote type=3D"cite"> =20 <div dir=3D"ltr"> <div>I would add a handler for the "pad-added" event on t= he filesrc element and only create/add the audio part of the pipeline if you get an audio pad created.=C2=A0 That is what I ha= ve done to handle rtspsrc streams that may or may not have audio.=C2=A0 Reply if you want more details.</div> <div><br> </div> <div> g_signal_connect(m_sourceElement, "pad-added", G_CALLBACK(&SynxRTSPPipeline::onPadAdded), this);<br> </div> </div> <br> <div class=3D"gmail_quote"> <div dir=3D"ltr" class=3D"gmail_attr">On Wed, 2 Oct 2024 at 07:30, Terry Barnaby via gstreamer-devel <<a href=3D"mailto:gstreamer= [email protected]" target=3D"_blank">[email protected]= esktop.org</a>> wrote:<br> </div> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex= ;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div> <p>I am developing a C++ video inspection program that creates MP4/H264/MP3 files and needs to play them back with the video stream processed by various gstreamer elements and audio separately.</p> <p>In general this has been working fine when just video was being recorded/played back, but I am just adding audio to the mix. Some MP4 files will contain only a video stream and some will contain both video and audio streams.</p> <p>I need some way to handle the playback of these MP4 files that may or may not have MP3 audio streams.<br> </p> <p>As a simple idea if I use something like (The real C++ code constructs the gstreamer piple line and tees the video stream to various gstreamer sub pipelines):<br> </p> <pre>gst-launch-1.0 -v filesrc location=3Dtemp.mp4 ! qtdemux na= me=3Ddemux</pre> <pre>=C2=A0demux.video_0 ! queue ! h264parse ! openh264dec ! gl= imagesink</pre> <pre>=C2=A0demux.audio_0 ! queue ! decodebin ! audioconvert ! p= ulsesink</pre> <p>This plays back the video and audio streams fine (not sure how well synchronised?) from an MP4 with video and audio streams, but hangs if the MP4 only has a video stream.</p> <p>So I think I need to:</p> <p>1. Maybe there is some gstreamer element or attribute that can ignore the audio stream if not present somehow ?</p> <p>2. Check if the MP4 file has an audio stream before creating the gstreamer pipeline in C++.</p> <p>3. Create the basic C++ pipleline in C++ and interrogate pads or something somehow and add the "demux.audio_0 ! queue ! decodebin ! audioconvert ! pulsesink" sub pipeli= ne if it is seen an audio stream is present.<br> </p> Any ideas on the simplest/most CPU efficient way of doing this with gstreamer ?<br> <p><br> </p> </div> </blockquote> </div> </blockquote> <p><br> </p> </div> </blockquote></div> --0000000000006d86bb06238e49a6-- --0000000000006d86bc06238e49a8 Content-Type: text/plain; charset="US-ASCII"; name="uri-decode-example.cpp" Content-Disposition: attachment; filename="uri-decode-example.cpp" Content-Transfer-Encoding: base64 Content-ID: <f_m1t160030> X-Attachment-Id: f_m1t160030 CVVSSVBpcGVsaW5lOjpVUklQaXBlbGluZShndWludHB0ciByZW5kZXJXaW5kb3dIYW5kbGUpDQoJ ew0KCQltX3ZpZGVvU2lua0VsZW1lbnQgPSBnc3RfZWxlbWVudF9mYWN0b3J5X21ha2UoImQzZDEx dmlkZW9zaW5rIiwgImxpdmV2c2luayIpOw0KDQoJCW1fc291cmNlRWxlbWVudCA9IGdzdF9lbGVt ZW50X2ZhY3RvcnlfbWFrZSgidXJpZGVjb2RlYmluIiwgImxpdmVzb3VyY2UiKTsNCg0KCQkvLyBO T1RFIC0gb25seSBhbGxvY2F0ZSB0aGVzZSBpZiB3ZSBhcmUgZ29pbmcgdG8gdXNlIHRoZW0NCgkJ Ly8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8v DQoJCW1fYXVkaW9Db252ZXJ0RWxlbWVudCA9IG51bGxwdHI7DQoJCW1fYXVkaW9SZXNhbXBsZUVs ZW1lbnQgPSBudWxscHRyOw0KCQltX2F1ZGlvU2lua0VsZW1lbnQgPSBudWxscHRyOw0KDQoNCgkJ bV92aWRlb0NvbnZlcnRFbGVtZW50ID0gZ3N0X2VsZW1lbnRfZmFjdG9yeV9tYWtlKCJkM2QxMWNv bnZlcnQiLCAidmlkZW9jb252ZXJ0Iik7DQoJCW1fb3ZlcmxheUVsZW1lbnQgPSBnc3RfZWxlbWVu dF9mYWN0b3J5X21ha2UoInN5bm92ZXJsYXkiLCAib3ZsIik7DQoNCgkJbV9waXBlbGluZSA9IGdz dF9waXBlbGluZV9uZXcoImxpdmUtcGlwZWxpbmUiKTsNCg0KCQlnc3RfYmluX2FkZF9tYW55KEdT VF9CSU4obV9waXBlbGluZSksIG1fc291cmNlRWxlbWVudCwgbV92aWRlb0NvbnZlcnRFbGVtZW50 LCAgbV9vdmVybGF5RWxlbWVudCwgbV92aWRlb1NpbmtFbGVtZW50LCBOVUxMKTsNCg0KCQlpZiAo IWdzdF9lbGVtZW50X2xpbmtfbWFueShtX3ZpZGVvQ29udmVydEVsZW1lbnQsICBtX292ZXJsYXlF bGVtZW50LCBtX3ZpZGVvU2lua0VsZW1lbnQsIE5VTEwpKQ0KCQl7DQoJCX0NCgkJZ3N0X3ZpZGVv X292ZXJsYXlfc2V0X3dpbmRvd19oYW5kbGUoR1NUX1ZJREVPX09WRVJMQVkobV92aWRlb1NpbmtF bGVtZW50KSwgcmVuZGVyV2luZG93SGFuZGxlKTsNCg0KCQlnX3NpZ25hbF9jb25uZWN0KG1fc291 cmNlRWxlbWVudCwgInBhZC1hZGRlZCIsIEdfQ0FMTEJBQ0soJlVSSVBpcGVsaW5lOjpvblBhZEFk ZGVkKSwgdGhpcyk7DQoNCgkJZ19vYmplY3Rfc2V0KG1fc291cmNlRWxlbWVudCwgInVyaSIsICJm aWxlOi8vL21wNC5tcDQiLCBOVUxMKTsNCg0KCX0NCg0KDQoJZ2Jvb2xlYW4gVVJJUGlwZWxpbmU6 Om9uUGFkQWRkZWQoR3N0RWxlbWVudCogc3JjLCBHc3RQYWQqIG5ld19wYWQsIFVSSVBpcGVsaW5l KiBvYmopDQoJew0KCQlHc3RQYWQqIGFzaW5rX3BhZCA9IG51bGxwdHI7DQoJCUdzdFBhZCogdnNp bmtfcGFkID0gZ3N0X2VsZW1lbnRfZ2V0X3N0YXRpY19wYWQob2JqLT5tX3ZpZGVvQ29udmVydEVs ZW1lbnQsICJzaW5rIik7DQoJCUdzdFBhZExpbmtSZXR1cm4gcmV0Ow0KCQlHc3RDYXBzKiBuZXdf cGFkX2NhcHMgPSBOVUxMOw0KCQlHc3RTdHJ1Y3R1cmUqIG5ld19wYWRfc3RydWN0ID0gTlVMTDsN CgkJY29uc3QgZ2NoYXIqIG5ld19wYWRfdHlwZSA9IE5VTEw7DQoNCgkJLyogQ2hlY2sgdGhlIG5l dyBwYWQncyB0eXBlICovDQoJCW5ld19wYWRfY2FwcyA9IGdzdF9wYWRfZ2V0X2N1cnJlbnRfY2Fw cyhuZXdfcGFkKTsNCgkJbmV3X3BhZF9zdHJ1Y3QgPSBnc3RfY2Fwc19nZXRfc3RydWN0dXJlKG5l d19wYWRfY2FwcywgMCk7DQoJCW5ld19wYWRfdHlwZSA9IGdzdF9zdHJ1Y3R1cmVfZ2V0X25hbWUo bmV3X3BhZF9zdHJ1Y3QpOw0KDQoJCWlmIChnX3N0cl9oYXNfcHJlZml4KG5ld19wYWRfdHlwZSwg ImF1ZGlvL3gtcmF3IikpDQoJCXsNCgkJCS8vIElmIG91ciBjb252ZXJ0ZXIgaXMgYWxyZWFkeSBj cmVhdGVkIGFuZCBsaW5rZWQsIHdlIGhhdmUgbm90aGluZyB0byBkbyBoZXJlDQoJCQlpZiAob2Jq LT5tX2F1ZGlvQ29udmVydEVsZW1lbnQgIT0gbnVsbHB0cikNCgkJCXsNCgkJCQlnX3ByaW50KCJX ZSBhcmUgYWxyZWFkeSBsaW5rZWQuIElnbm9yaW5nLlxuIik7DQoJCQkJZ290byBleGl0Ow0KCQkJ fQ0KDQoJCQlvYmotPm1fYXVkaW9Db252ZXJ0RWxlbWVudCA9IGdzdF9lbGVtZW50X2ZhY3Rvcnlf bWFrZSgiYXVkaW9jb252ZXJ0IiwgImF1ZGlvY29udmVydCIpOw0KCQkJb2JqLT5tX2F1ZGlvUmVz YW1wbGVFbGVtZW50ID0gZ3N0X2VsZW1lbnRfZmFjdG9yeV9tYWtlKCJhdWRpb3Jlc2FtcGxlIiwg ImF1ZGlvcmVzYW1wbGUiKTsNCgkJCW9iai0+bV9hdWRpb1NpbmtFbGVtZW50ID0gZ3N0X2VsZW1l bnRfZmFjdG9yeV9tYWtlKCJhdXRvYXVkaW9zaW5rIiwgImF1ZGlvc2luayIpOw0KDQoJCQkvLyBw dXQgdGhlbSBpbiB0aGUgcGlwZWxpbmUNCgkJCWdzdF9iaW5fYWRkX21hbnkoR1NUX0JJTihvYmot Pm1fcGlwZWxpbmUpLCBvYmotPm1fYXVkaW9Db252ZXJ0RWxlbWVudCwgb2JqLT5tX2F1ZGlvUmVz YW1wbGVFbGVtZW50LCBvYmotPm1fYXVkaW9TaW5rRWxlbWVudCwgTlVMTCk7DQoNCgkJCS8vIGxp bmsgdGhlbQ0KCQkJaWYgKCFnc3RfZWxlbWVudF9saW5rX21hbnkob2JqLT5tX2F1ZGlvQ29udmVy dEVsZW1lbnQsIG9iai0+bV9hdWRpb1Jlc2FtcGxlRWxlbWVudCwgb2JqLT5tX2F1ZGlvU2lua0Vs ZW1lbnQsIE5VTEwpKQ0KCQkJew0KCQkJfQ0KDQoJCQkvLyByZXN5bmMgc3RhdGUNCgkJCWdzdF9l bGVtZW50X3N5bmNfc3RhdGVfd2l0aF9wYXJlbnQob2JqLT5tX2F1ZGlvQ29udmVydEVsZW1lbnQp Ow0KCQkJZ3N0X2VsZW1lbnRfc3luY19zdGF0ZV93aXRoX3BhcmVudChvYmotPm1fYXVkaW9SZXNh bXBsZUVsZW1lbnQpOw0KCQkJZ3N0X2VsZW1lbnRfc3luY19zdGF0ZV93aXRoX3BhcmVudChvYmot Pm1fYXVkaW9TaW5rRWxlbWVudCk7DQoNCgkJCS8vIGxpbmsgdGhlIG5ldyBwYWQNCgkJCWFzaW5r X3BhZCA9IGdzdF9lbGVtZW50X2dldF9zdGF0aWNfcGFkKG9iai0+bV9hdWRpb0NvbnZlcnRFbGVt ZW50LCAic2luayIpOw0KDQoJCQkvLy8qIEF0dGVtcHQgdGhlIGxpbmsgKi8NCgkJCXJldCA9IGdz dF9wYWRfbGluayhuZXdfcGFkLCBhc2lua19wYWQpOw0KCQkJaWYgKEdTVF9QQURfTElOS19GQUlM RUQocmV0KSkgew0KCQkJCWdfcHJpbnQoIlR5cGUgaXMgJyVzJyBidXQgbGluayBmYWlsZWQuXG4i LCBuZXdfcGFkX3R5cGUpOw0KCQkJfQ0KCQkJZWxzZSB7DQoJCQkJZ19wcmludCgiTGluayBzdWNj ZWVkZWQgKHR5cGUgJyVzJykuXG4iLCBuZXdfcGFkX3R5cGUpOw0KCQkJfQ0KDQoJCX0NCgkJZWxz ZSBpZiAoZ19zdHJfaGFzX3ByZWZpeChuZXdfcGFkX3R5cGUsICJ2aWRlby94LXJhdyIpKQ0KCQl7 DQoNCgkJCS8qIElmIG91ciBjb252ZXJ0ZXIgaXMgYWxyZWFkeSBsaW5rZWQsIHdlIGhhdmUgbm90 aGluZyB0byBkbyBoZXJlICovDQoJCQlpZiAoZ3N0X3BhZF9pc19saW5rZWQodnNpbmtfcGFkKSkg ew0KCQkJCWdfcHJpbnQoIldlIGFyZSBhbHJlYWR5IGxpbmtlZC4gSWdub3JpbmcuXG4iKTsNCgkJ CQlnb3RvIGV4aXQ7DQoJCQl9DQoNCgkJCS8qIEF0dGVtcHQgdGhlIGxpbmsgKi8NCgkJCXJldCA9 IGdzdF9wYWRfbGluayhuZXdfcGFkLCB2c2lua19wYWQpOw0KCQkJaWYgKEdTVF9QQURfTElOS19G QUlMRUQocmV0KSkgew0KCQkJCWdfcHJpbnQoIlR5cGUgaXMgJyVzJyBidXQgbGluayBmYWlsZWQu XG4iLCBuZXdfcGFkX3R5cGUpOw0KCQkJfQ0KCQkJZWxzZSB7DQoJCQkJZ19wcmludCgiTGluayBz dWNjZWVkZWQgKHR5cGUgJyVzJykuXG4iLCBuZXdfcGFkX3R5cGUpOw0KCQkJfQ0KDQoJCX0NCgkJ ZWxzZQ0KCQl7DQoJCQlnX3ByaW50KCJJdCBoYXMgdHlwZSAnJXMnIHdoaWNoIGlzIG5vdCByYXcg YXVkaW8uIElnbm9yaW5nLlxuIiwNCgkJCQluZXdfcGFkX3R5cGUpOw0KCQkJZ290byBleGl0Ow0K DQoJCX0NCg0KCQkvKnZpZGVvL3gtcmF3Ki8NCg0KCWV4aXQ6DQoJCS8qIFVucmVmZXJlbmNlIHRo ZSBuZXcgcGFkJ3MgY2FwcywgaWYgd2UgZ290IHRoZW0gKi8NCgkJaWYgKG5ld19wYWRfY2FwcyAh PSBOVUxMKQ0KCQkJZ3N0X2NhcHNfdW5yZWYobmV3X3BhZF9jYXBzKTsNCg0KCQkvKiBVbnJlZmVy ZW5jZSB0aGUgc2luayBwYWQgKi8NCgkJaWYgKHZzaW5rX3BhZCkNCgkJew0KCQkJZ3N0X29iamVj dF91bnJlZih2c2lua19wYWQpOw0KCQl9DQoJCWlmIChhc2lua19wYWQpDQoJCXsNCgkJCWdzdF9v YmplY3RfdW5yZWYoYXNpbmtfcGFkKTsNCgkJfQ0KDQoJCXJldHVybiBUUlVFOw0KCX0NCn0NCg== --0000000000006d86bc06238e49a8--