Re: GStreamer playback of MP4 files with and without audio
Terry Barnaby via gstreamer-devel <[email protected]> Wed, 2 Oct 2024 17:46:28 +0100
| Newsgroups | gmane.comp.video.gstreamer.devel |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --------------G0i7g1TP0Tb95doqoZOATVrB Content-Type: multipart/alternative; boundary="------------GCXGV1rH33oWPrSANdRD5drf" --------------GCXGV1rH33oWPrSANdRD5drf Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit I have tried using g_signal_connect(m_sourceElement, "pad-added", ...) and within that adding a sub pipeline to the main pipeline, but I am getting something wrong when I add the sub pipeline. My simple program attached, sees the signal callback, but when I add the sub pipeline and link it to the new pad, the displayed video just freezes. I've looked at the "Basic tutorial 3: Dynamic pipelines" and it looks like I am doing the right thing and no errors are seen. Any ideas on what I am ding wrong ? On 02/10/2024 08:26, Gareth Alldread via gstreamer-devel wrote: > I would add a handler for the "pad-added" event on the filesrc element > and only create/add the audio part of the pipeline if you get an audio > pad created. That is what I have done to handle rtspsrc streams that > may or may not have audio. Reply if you want more details. > > g_signal_connect(m_sourceElement, "pad-added", > G_CALLBACK(&SynxRTSPPipeline::onPadAdded), this); > > On Wed, 2 Oct 2024 at 07:30, Terry Barnaby via gstreamer-devel > <[email protected]> wrote: > > I am developing a C++ video inspection program that creates > MP4/H264/MP3 files and needs to play them back with the video > stream processed by various gstreamer elements and audio separately. > > In general this has been working fine when just video was being > recorded/played back, but I am just adding audio to the mix. Some > MP4 files will contain only a video stream and some will contain > both video and audio streams. > > I need some way to handle the playback of these MP4 files that may > or may not have MP3 audio streams. > > As a simple idea if I use something like (The real C++ code > constructs the gstreamer piple line and tees the video stream to > various gstreamer sub pipelines): > > gst-launch-1.0 -v filesrc location=temp.mp4 ! qtdemux name=demux > > demux.video_0 ! queue ! h264parse ! openh264dec ! glimagesink > > demux.audio_0 ! queue ! decodebin ! audioconvert ! pulsesink > > This plays back the video and audio streams fine (not sure how > well synchronised?) from an MP4 with video and audio streams, but > hangs if the MP4 only has a video stream. > > So I think I need to: > > 1. Maybe there is some gstreamer element or attribute that can > ignore the audio stream if not present somehow ? > > 2. Check if the MP4 file has an audio stream before creating the > gstreamer pipeline in C++. > > 3. Create the basic C++ pipleline in C++ and interrogate pads or > something somehow and add the "demux.audio_0 ! queue ! decodebin ! > audioconvert ! pulsesink" sub pipeline if it is seen an audio > stream is present. > > Any ideas on the simplest/most CPU efficient way of doing this > with gstreamer ? > > --------------GCXGV1rH33oWPrSANdRD5drf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <div class="moz-cite-prefix">I have tried using g_signal_connect(m_sourceElement, "pad-added", ...) and within that adding a sub pipeline to the main pipeline, but I am getting something wrong when I add the sub pipeline.</div> <div class="moz-cite-prefix"><br> </div> <div class="moz-cite-prefix">My simple program attached, sees the signal callback, but when I add the sub pipeline and link it to the new pad, the displayed video just freezes.</div> <div class="moz-cite-prefix">I've looked at the "Basic tutorial 3: Dynamic pipelines" and it looks like I am doing the right thing and no errors are seen.</div> <div class="moz-cite-prefix"><br> </div> <div class="moz-cite-prefix">Any ideas on what I am ding wrong ?<br> </div> <div class="moz-cite-prefix"><br> </div> <div class="moz-cite-prefix"><br> </div> <div class="moz-cite-prefix">On 02/10/2024 08:26, Gareth Alldread via gstreamer-devel wrote:<br> </div> <blockquote type="cite" cite="mid:CAENg58qyeVDbzdJwNwdQ2pcxxajL2QN=U9KabvDqBV17R6ZhfQ@mail.gmail.com"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <div dir="ltr"> <div>I would add a handler for the "pad-added" event on the filesrc element and only create/add the audio part of the pipeline if you get an audio pad created. That is what I have done to handle rtspsrc streams that may or may not have audio. Reply if you want more details.</div> <div><br> </div> <div> g_signal_connect(m_sourceElement, "pad-added", G_CALLBACK(&SynxRTSPPipeline::onPadAdded), this);<br> </div> </div> <br> <div class="gmail_quote"> <div dir="ltr" class="gmail_attr">On Wed, 2 Oct 2024 at 07:30, Terry Barnaby via gstreamer-devel <<a href="mailto:[email protected]" moz-do-not-send="true" class="moz-txt-link-freetext">[email protected]</a>> wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div> <p>I am developing a C++ video inspection program that creates MP4/H264/MP3 files and needs to play them back with the video stream processed by various gstreamer elements and audio separately.</p> <p>In general this has been working fine when just video was being recorded/played back, but I am just adding audio to the mix. Some MP4 files will contain only a video stream and some will contain both video and audio streams.</p> <p>I need some way to handle the playback of these MP4 files that may or may not have MP3 audio streams.<br> </p> <p>As a simple idea if I use something like (The real C++ code constructs the gstreamer piple line and tees the video stream to various gstreamer sub pipelines):<br> </p> <pre>gst-launch-1.0 -v filesrc location=temp.mp4 ! qtdemux name=demux</pre> <pre> demux.video_0 ! queue ! h264parse ! openh264dec ! glimagesink</pre> <pre> demux.audio_0 ! queue ! decodebin ! audioconvert ! pulsesink</pre> <p>This plays back the video and audio streams fine (not sure how well synchronised?) from an MP4 with video and audio streams, but hangs if the MP4 only has a video stream.</p> <p>So I think I need to:</p> <p>1. Maybe there is some gstreamer element or attribute that can ignore the audio stream if not present somehow ?</p> <p>2. Check if the MP4 file has an audio stream before creating the gstreamer pipeline in C++.</p> <p>3. Create the basic C++ pipleline in C++ and interrogate pads or something somehow and add the "demux.audio_0 ! queue ! decodebin ! audioconvert ! pulsesink" sub pipeline if it is seen an audio stream is present.<br> </p> Any ideas on the simplest/most CPU efficient way of doing this with gstreamer ?<br> <p><br> </p> </div> </blockquote> </div> </blockquote> <p><br> </p> </body> </html> --------------GCXGV1rH33oWPrSANdRD5drf-- --------------G0i7g1TP0Tb95doqoZOATVrB Content-Type: text/x-c++src; charset=UTF-8; name="test1.cpp" Content-Disposition: attachment; filename="test1.cpp" Content-Transfer-Encoding: base64 LyoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKgogKgl0ZXN0MS5jcHAJU2ltcGxlIEMrKyBn c3RyZWFtZXIgZXhhbXBsZQogKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqCiAqCiAqIGh0 dHBzOi8vZ3N0cmVhbWVyLmZyZWVkZXNrdG9wLm9yZy9kb2N1bWVudGF0aW9uL3R1dG9yaWFs cwogKgogKi8KI2luY2x1ZGUgPGdzdC9nc3QuaD4KI2luY2x1ZGUgPGdzdC9hcHAvZ3N0YXBw c3JjLmg+CiNpbmNsdWRlIDxnc3QvcGJ1dGlscy9nc3RkaXNjb3ZlcmVyLmg+CiNpbmNsdWRl IDxnbGliLmg+CiNpbmNsdWRlIDxzdGRpby5oPgojaW5jbHVkZSA8dW5pc3RkLmg+CiNpbmNs dWRlIDxzaWduYWwuaD4KI2luY2x1ZGUgPHN0ZGludC5oPgoKR3N0RWxlbWVudCoJcGlwZWxp bmU7CkdzdENsb2NrVGltZQljYW1lcmFUaW1lOwpHc3RDbG9ja1RpbWUJb3ZlcmxheVRpbWU7 CgpzdGF0aWMgZ2Jvb2xlYW4gYnVzX2NhbGwoR3N0QnVzKiBidXMsIEdzdE1lc3NhZ2UqIG1z ZywgZ3BvaW50ZXIgZGF0YSl7CglHTWFpbkxvb3AqCWxvb3AgPSAoR01haW5Mb29wKilkYXRh OwoKCS8vcHJpbnRmKCJidXNfY2FsbDogJWRcbiIsIEdTVF9NRVNTQUdFX1RZUEUobXNnKSk7 CgkKCXN3aXRjaChHU1RfTUVTU0FHRV9UWVBFKG1zZykpewoKCWNhc2UgR1NUX01FU1NBR0Vf RU9TOgoJCXByaW50ZigiRW5kIG9mIHN0cmVhbVxuIik7CgkJLy9nX21haW5fbG9vcF9xdWl0 KGxvb3ApOwoJCWJyZWFrOwoKCWNhc2UgR1NUX01FU1NBR0VfRVJST1I6ewoJCWdjaGFyICAq ZGVidWc7CgkJR0Vycm9yICplcnJvcjsKCgkJZ3N0X21lc3NhZ2VfcGFyc2VfZXJyb3IgKG1z ZywgJmVycm9yLCAmZGVidWcpOwoJCWdfZnJlZSAoZGVidWcpOwoKCQlwcmludGYoIkVycm9y OiAlc1xuIiwgZXJyb3ItPm1lc3NhZ2UpOwoJCWdfZXJyb3JfZnJlZShlcnJvcik7CgoJCWdf bWFpbl9sb29wX3F1aXQobG9vcCk7CgkJYnJlYWs7CgkJfQoKCWRlZmF1bHQ6CgkJLy9wcmlu dGYoImJ1c19jYWxsOiBtZXNzYWdlOiAlZFxuIiwgR1NUX01FU1NBR0VfVFlQRShtc2cpKTsK CQlicmVhazsKCX0KCglyZXR1cm4gR1NUX0JVU19QQVNTOwp9Cgp2b2lkICBnc3RBdWRpb0Fk ZFBhZENhbGxiYWNrMShHc3RFbGVtZW50KiBlbGVtZW50LCBHc3RQYWQqIHBhZCwgZ3BvaW50 ZXIgdXNlcl9kYXRhKXsKCUdzdEVsZW1lbnQqCXBpcGVsaW5lID0gKEdzdEVsZW1lbnQqKXVz ZXJfZGF0YTsKCUdFcnJvcioJCWVycm9yID0gTlVMTDsKCWdjaGFyKgkJbmFtZTsKCUdzdEVs ZW1lbnQqCWUxOwoJR3N0RWxlbWVudCoJZTI7CglHc3RQYWQqCQlwYWQyOwoKCXByaW50Zigi Z3N0QXVkaW9BZGRQYWRDYWxsYmFjazFcbiIpOwoKCW5hbWUgPSBnc3RfcGFkX2dldF9uYW1l KHBhZCk7CglnX3ByaW50ICgiQSBuZXcgcGFkICVzIHdhcyBjcmVhdGVkXG4iLCBuYW1lKTsK CWlmKCFzdHJjbXAobmFtZSwgImF1ZGlvXzAiKSl7CgkJR3N0RWxlbWVudCoJZTE7CgkJR3N0 RWxlbWVudCoJZTI7CgkJR3N0UGFkKgkJcGFkMjsKCgkJZTEgPSBnc3RfcGFyc2VfYmluX2Zy b21fZGVzY3JpcHRpb24oInF1ZXVlIG5hbWU9YXVkaW9RdWV1ZSAhIGRlY29kZWJpbiAhIGF1 ZGlvY29udmVydCAhIHB1bHNlc2luayIsIHRydWUsICZlcnJvcik7CgkJZ3N0X2Jpbl9hZGQo R1NUX0JJTihwaXBlbGluZSksIGUxKTsKCQlnc3RfZWxlbWVudF9zeW5jX3N0YXRlX3dpdGhf cGFyZW50KGUxKTsKCgkJZTIgPSBnc3RfYmluX2dldF9ieV9uYW1lKEdTVF9CSU4oZTEpLCAi YXVkaW9RdWV1ZSIpOwoJCXBhZDIgPSBnc3RfZWxlbWVudF9nZXRfc3RhdGljX3BhZChlMiwg InNpbmsiKTsKCQlwcmludGYoIkxpbmtQYWRzOiAlcCAlcFxuIiwgcGFkLCBwYWQyKTsKCQln c3RfcGFkX2xpbmsocGFkLCBwYWQyKTsKCQlnc3Rfb2JqZWN0X3VucmVmKHBhZDIpOwoJfQoJ Z19mcmVlIChuYW1lKTsKfQoKdm9pZCAgZ3N0QXVkaW9BZGRQYWRDYWxsYmFjazIoR3N0RWxl bWVudCogZWxlbWVudCwgR3N0UGFkKiBwYWQsIGdwb2ludGVyIHVzZXJfZGF0YSl7CglHc3RF bGVtZW50KglwaXBlbGluZSA9IChHc3RFbGVtZW50Kil1c2VyX2RhdGE7CglHRXJyb3IqCQll cnJvciA9IE5VTEw7CglnY2hhcioJCW5hbWU7CglHc3RFbGVtZW50KgllMTsKCUdzdEVsZW1l bnQqCWUyOwoJR3N0RWxlbWVudCoJZTM7CglHc3RFbGVtZW50KgllNDsKCUdzdFBhZCoJCXBh ZDI7CgoJcHJpbnRmKCJnc3RBdWRpb0FkZFBhZENhbGxiYWNrMlxuIik7CgoJbmFtZSA9IGdz dF9wYWRfZ2V0X25hbWUocGFkKTsKCWdfcHJpbnQgKCJBIG5ldyBwYWQgJXMgd2FzIGNyZWF0 ZWRcbiIsIG5hbWUpOwoJaWYoIXN0cmNtcChuYW1lLCAiYXVkaW9fMCIpKXsKCQlHc3RFbGVt ZW50KgllMTsKCQlHc3RFbGVtZW50KgllMjsKCQlHc3RFbGVtZW50KgllMzsKCQlHc3RFbGVt ZW50KgllNDsKCQlHc3RQYWQqCQlwYWQyOwoJCQoJCQoJCWUxID0gZ3N0X2VsZW1lbnRfZmFj dG9yeV9tYWtlKCJxdWV1ZSIsIE5VTEwpOwoJCWUyID0gZ3N0X2VsZW1lbnRfZmFjdG9yeV9t YWtlKCJkZWNvZGViaW4iLCBOVUxMKTsKCQllMyA9IGdzdF9lbGVtZW50X2ZhY3RvcnlfbWFr ZSgiYXVkaW9jb252ZXJ0IiwgTlVMTCk7CgkJZTQgPSBnc3RfZWxlbWVudF9mYWN0b3J5X21h a2UoInB1bHNlc2luayIsIE5VTEwpOwoJCWdzdF9iaW5fYWRkX21hbnkoR1NUX0JJTihwaXBl bGluZSksIGUxLCBlMiwgZTMsIGU0LCAwKTsKCQlnc3RfZWxlbWVudF9saW5rX21hbnkoZTEs IGUyLCBlMywgZTQsIDApOwoJCQoJCXBhZDIgPSBnc3RfZWxlbWVudF9nZXRfc3RhdGljX3Bh ZChlMSwgInNpbmsiKTsKCQlwcmludGYoIkxpbmtQYWRzOiAlcCAlcFxuIiwgcGFkLCBwYWQy KTsKCQlnc3RfcGFkX2xpbmsocGFkLCBwYWQyKTsKCQlnc3Rfb2JqZWN0X3VucmVmKHBhZDIp OwoJfQoJZ19mcmVlIChuYW1lKTsKfQoKaW50IG1haW4gKGludCBhcmdjLCBjaGFyKiogYXJn dil7CglHTWFpbkxvb3AqCWxvb3A7CglHRXJyb3IqCQllcnJvciA9IE5VTEw7CglHc3RCdXMq CQlidXM7CglndWludAkJYnVzX3dhdGNoX2lkOwoJcHRocmVhZF90CXRocmVhZDE7CglwdGhy ZWFkX3QJdGhyZWFkMjsKCWNoYXIJCWNtZFs0MDk2XTsKCUdzdEVsZW1lbnQqCWRlbXV4OwoJ R3N0UGFkKgkJcGFkOwoKCS8vIEluaXRpYWxpc2F0aW9uCglnc3RfaW5pdCgmYXJnYywgJmFy Z3YpOwoKCWxvb3AgPSBnX21haW5fbG9vcF9uZXcoTlVMTCwgRkFMU0UpOwoKCWNtZFswXSA9 IDA7CgoJLy9zdHJjYXQoY21kLCAiZmlsZXNyYyBsb2NhdGlvbj10ZW1wMS5tcDQgISBxdGRl bXV4IG5hbWU9ZGVtdXggIGRlbXV4LnZpZGVvXzAgISBxdWV1ZSAhIGgyNjRwYXJzZSAhIG9w ZW5oMjY0ZGVjICEgZ2xpbWFnZXNpbmsiKTsKCXN0cmNhdChjbWQsICJmaWxlc3JjIGxvY2F0 aW9uPXRlbXAyLm1wNCAhIHF0ZGVtdXggbmFtZT1kZW11eCAgZGVtdXgudmlkZW9fMCAhIHF1 ZXVlICEgaDI2NHBhcnNlICEgb3BlbmgyNjRkZWMgISBnbGltYWdlc2luayIpOwoKCS8vc3Ry Y2F0KGNtZCwgIiBkZW11eC5hdWRpb18wICEgcXVldWUgbmFtZT1hdWRpb1F1ZXVlICEgZGVj b2RlYmluICEgYXVkaW9jb252ZXJ0ICEgcHVsc2VzaW5rIik7CgkvL3N0cmNhdChjbWQsICIg cXVldWUgbmFtZT1hdWRpb1F1ZXVlICEgZGVjb2RlYmluICEgYXVkaW9jb252ZXJ0ICEgcHVs c2VzaW5rIik7CgoJCQoJcHJpbnRmKCJSdW46ICVzXG4iLCBjbWQpOwoJcGlwZWxpbmUgPSBn c3RfcGFyc2VfbGF1bmNoKGNtZCwgJmVycm9yKTsKCglkZW11eCA9IGdzdF9iaW5fZ2V0X2J5 X25hbWUoR1NUX0JJTihwaXBlbGluZSksICJkZW11eCIpOwoJZ19zaWduYWxfY29ubmVjdChk ZW11eCwgInBhZC1hZGRlZCIsIEdfQ0FMTEJBQ0soZ3N0QXVkaW9BZGRQYWRDYWxsYmFjazEp LCBwaXBlbGluZSk7CgkvL2dfc2lnbmFsX2Nvbm5lY3QoZGVtdXgsICJwYWQtYWRkZWQiLCBH X0NBTExCQUNLKGdzdEF1ZGlvQWRkUGFkQ2FsbGJhY2syKSwgcGlwZWxpbmUpOwoKCS8vIFdl IGFkZCBhIG1lc3NhZ2UgaGFuZGxlcgoJYnVzID0gZ3N0X3BpcGVsaW5lX2dldF9idXMoR1NU X1BJUEVMSU5FIChwaXBlbGluZSkpOwoJYnVzX3dhdGNoX2lkID0gZ3N0X2J1c19hZGRfd2F0 Y2goYnVzLCBidXNfY2FsbCwgbG9vcCk7Cglnc3Rfb2JqZWN0X3VucmVmKGJ1cyk7CgoJcHJp bnRmKCJOb3cgc2V0IHBpcGVsaW5lIGluIHN0YXRlIHBsYXlpbmdcbiIpOwoJZ3N0X2VsZW1l bnRfc2V0X3N0YXRlKHBpcGVsaW5lLCBHU1RfU1RBVEVfUExBWUlORyk7CgoJLy8gSXRlcmF0 ZQoJcHJpbnRmKCJSdW5uaW5nLi4uXG4iKTsKCWdfbWFpbl9sb29wX3J1bihsb29wKTsKCgkv LyBPdXQgb2YgdGhlIG1haW4gbG9vcCwgY2xlYW4gdXAgbmljZWx5CglwcmludGYoIlJldHVy bmVkLCBzdG9wcGluZyBwbGF5YmFja1xuIik7Cglnc3RfZWxlbWVudF9zZXRfc3RhdGUocGlw ZWxpbmUsIEdTVF9TVEFURV9OVUxMKTsKCglwcmludGYoIkRlbGV0aW5nIHBpcGVsaW5lXG4i KTsKCWdzdF9vYmplY3RfdW5yZWYoR1NUX09CSkVDVCAocGlwZWxpbmUpKTsKCWdfc291cmNl X3JlbW92ZShidXNfd2F0Y2hfaWQpOwoJZ19tYWluX2xvb3BfdW5yZWYobG9vcCk7CgoJcmV0 dXJuIDA7Cn0K --------------G0i7g1TP0Tb95doqoZOATVrB--