Re: Issue with decoding H264 frames using appsrc in GStreamer

That was the thing!! removing the caps specification!!

If anyone is wondering how to do this pipeline. This is the final code to
create it
(Finally I add a fakesink to avoid the generation of big raw files)

m_pipeline = gst_pipeline_new("mypipeline");
    m_appsrc = gst_element_factory_make("appsrc", "mysource");
    GstElement *h264_parser = gst_element_factory_make("h264parse",
"myparser");
    GstElement *decoder = gst_element_factory_make("omxh264dec", "mydec");
    GstElement *fakesink = gst_element_factory_make("fakesink", "mysink");

    /* g_object_set(G_OBJECT (filesink), "location", "ball_inside.raw",
            NULL); */

    // Check if all elements were created
    if (!m_pipeline && m_appsrc && h264_parser && decoder && fakesink)
    {
        g_print("Could not gst_element_factory_make, terminating\n");
        return;
    }

    // Create gstreamer loop
    loop = g_main_loop_new(NULL, FALSE);

    // add a message handler
    bus = gst_pipeline_get_bus(GST_PIPELINE(m_pipeline));
    bus_watch_id = gst_bus_add_watch(bus, bus_call, loop);
    gst_object_unref(bus);

    // add all elements into pipeline
    gst_bin_add_many(GST_BIN(m_pipeline), m_appsrc, h264_parser, decoder,
fakesink, NULL);

    // link elements into pipe: appsrc -> jpegenc -> appsink
    gst_element_link(m_appsrc, h264_parser);
    gst_element_link(h264_parser, decoder);
    gst_element_link(decoder, fakesink);

    GstPad *encoder_pad;

    encoder_pad = gst_element_get_static_pad(decoder, "src");
    gst_pad_add_probe(encoder_pad, GST_PAD_PROBE_TYPE_BUFFER,
(GstPadProbeCallback)probe_source_callback, NULL, NULL);

    encoder_pad = gst_element_get_static_pad(decoder, "sink");
    gst_pad_add_probe(encoder_pad, GST_PAD_PROBE_TYPE_BUFFER,
(GstPadProbeCallback)probe_sink_callback, NULL, NULL);

    gst_object_unref (encoder_pad);

    g_print("Setting g_main_loop_run to GST_STATE_PLAYING\n");
    // Start pipeline so it could process incoming data
    gst_element_set_state(m_pipeline, GST_STATE_PLAYING);

I also add those Pads to check that I have buffers on both sides of the
decoder.

Anyone know why that was the reason to fix this? (Removing the caps)

Thank you

On Mon, Jul 10, 2023 at 10:41 PM Michael Gruner <[email protected]>
wrote:

> Nevermind, I just re-read that you had a parser already. Try removing the
> caps specification and letting the parser come with its own set of caps.
>
> On 10 Jul 2023, at 14:37, Michael Gruner <[email protected]>
> wrote:
>
> You should be able to use the decoder, yes. Try placing a h264parse right
> after the appsrc. That will read the H264 bitstream, parse it, accumulate
> it to form full NAL units and perform stream format conversions if they are
> needed by the decoder.
>
> -Michael
>
> On 10 Jul 2023, at 09:04, Israel Díaz <[email protected]> wrote:
>
> Ohh. Thank you but it seems I don't have access to gst_buffer_new_memdup
> since I am using 1.18
>
> The other option to not copy I guess is not suitable for me because
> encoded_frame will be freed by another component (the encoded data is part
> of a bigger object that is free when the object call its destructor)
>
> So finally I opted for this
>
> buffer = gst_buffer_new_allocate (NULL, frame_size, NULL);
>     gst_buffer_map (buffer, &map, GST_MAP_WRITE);
>     memcpy((guchar *)map.data, encoded_frame,  gst_buffer_get_size(
> buffer ) );
>     gst_buffer_unmap(buffer, &map);
>
>
> Now I don't see those *broken/invalid nal Type: 0*  messages but
> currently I still don't see that the raw file is receiving bytes.
> Do you think it is possible to use the decoder with this configuration?
> Maybe I need to follow the approach of need-data to feed the decoder or How
> Can I check if the decoder is actually processing everything?
>
> On Fri, Jul 7, 2023 at 4:50 PM Michael Gruner <[email protected]>
> wrote:
>
>> Hi Israel
>>
>> This assignment is invalid:
>>
>> gst_buffer_map(buffer, &map, GST_MAP_WRITE);
>> * map.data = encoded_frame;*
>>   gst_buffer_unmap(buffer, &map);
>>
>> You need to copy the data using a plain memcpy or
>> by creating the new buffer directly as a copy using gst_buffer_new_memdup.
>>
>> If you would like to avoid the copy, then you can use
>> gst_buffer_new_wrapped or gst_buffer_new_wrapped_full but you need to be
>> careful that: 1. The pointer remains valid the while the buffer is alive
>> and 2. The pointer is freed when the buffer is unreffed.
>>
>> Michael
>>
>> On 7 Jul 2023, at 04:02, Israel Díaz via gstreamer-devel <
>> [email protected]> wrote:
>>
>> 
>> Dear GStreamer community,
>>
>> I am currently facing an issue while attempting to decode H264 frames
>> using the appsrc element in GStreamer. I have constructed the following
>> pipeline within my application:
>>
>> appsrc name=mysource ! video/x-h264, alignment=au,
>> stream-format=byte-stream ! h264parse ! video/x-h264, alignment=au,
>> stream-format=byte-stream ! omxh264dec ! video/x-raw, format=NV16_10LE32,
>> width=1920, height=1080, framerate=30/1 ! filesink location=ball.raw
>>
>> To feed the appsrc with frames, I am reading an H264 file frame by frame
>> since I store each frame in a structure along with its size. Then, I push
>> the frames into the appsrc using the following function:
>>
>> void
>> VDRE2GStreamerDecodingDisplaySinkTask::PushBufferToGStreamer(uint8_t*
>> encoded_frame, uint32_t frame_size)
>> {
>>     GstFlowReturn ret;
>>     GstMapInfo map;
>>     GstBuffer *buffer = gst_buffer_new_and_alloc(frame_size);
>>     gst_buffer_map(buffer, &map, GST_MAP_WRITE);
>>     map.data = encoded_frame;
>>     gst_buffer_unmap(buffer, &map);
>>
>>     /* Push the buffer into the appsrc */
>>     g_signal_emit_by_name(m_appsrc, "push-buffer", buffer, &ret);
>>
>>     /* Free the buffer now that we are done with it */
>>     gst_buffer_unref(buffer);
>>
>>     if (ret != GST_FLOW_OK) {
>>         g_print("PushBufferToGStreamer: Error!\n");
>>     }
>> }
>>
>> However, when I execute the pipeline and set GST_DEBUG=4, I encounter the
>> following error messages:
>>
>> Decode Thread: Frame to Process : 1 :s 246
>> Decode Thread: Frame to Process : 2 :s 232
>> Decode Thread: Frame to Process : 3 :s 854
>> Decode Thread: Frame to Process : 4 :s 952
>> Decode Thread: Frame to Process : 5 :s 350
>> Decode Thread: Frame to Process : 6 :s 1038
>> Decode Thread: Frame to Process : 7 :s 1447
>> Decode Thread: Frame to Process : 8 :s 1316
>> Decode Thread: Frame to Process : 9 :s 1297
>> Decode Thread: Frame to Process : 10 :s 1205
>> 0:00:02.370465675  6548 0xffff8c01ff60 WARN               h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 29 will be dropped
>> 0:00:02.370549896  6548 0xffff8c01ff60 WARN               h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 1140 will be dropped
>> Decode Thread: Frame to Process : 11 :s 1222
>> Decode Thread: Frame to Process : 12 :s 1189
>> Decode Thread: Frame to Process : 13 :s 1094
>> Decode Thread: Frame to Process : 14 :s 930
>> Decode Thread: Frame to Process : 15 :s 1000
>> Decode Thread: Frame to Process : 16 :s 888
>> Decode Thread: Frame to Process : 17 :s 901
>> Decode Thread: Frame to Process : 18 :s 788
>> Decode Thread: Frame to Process : 19 :s 833
>> Decode Thread: Frame to Process : 20 :s 764
>> Decode Thread: Frame to Process : 21 :s 727
>> Decode Thread: Frame to Process : 22 :s 616
>> Decode Thread: Frame to Process : 23 :s 728
>> Decode Thread: Frame to Process : 24 :s 622
>> Decode Thread: Frame to Process : 25 :s 609
>> Decode Thread: Frame to Process : 26 :s 657
>> Decode Thread: Frame to Process : 27 :s 651
>> Decode Thread: Frame to Process : 28 :s 621
>> Decode Thread: Frame to Process : 29 :s 534
>> Decode Thread: Frame to Process : 30 :s 2989
>> 0:00:03.033182378  6548 0xffff8c01ff60 WARN               h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 29 will be dropped
>> 0:00:03.033258709  6548 0xffff8c01ff60 WARN               h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 117 will be dropped
>> 0:00:03.033309439  6548 0xffff8c01ff60 WARN               h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 29 will be dropped
>> 0:00:03.033369800  6548 0xffff8c01ff60 WARN               h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 1468 will be dropped
>> Decode Thread: Frame to Process : 31 :s 411
>> Decode Thread: Frame to Process : 32 :s 465
>> Decode Thread: Frame to Process : 33 :s 514
>> Decode Thread: Frame to Process : 34 :s 520
>> Decode Thread: Frame to Process : 35 :s 519
>> Decode Thread: Frame to Process : 36 :s 476
>> Decode Thread: Frame to Process : 37 :s 448
>> Decode Thread: Frame to Process : 38 :s 544
>> Decode Thread: Frame to Process : 39 :s 523
>> Decode Thread: Frame to Process : 40 :s 454
>> Decode Thread: Frame to Process : 41 :s 561
>> Decode Thread: Frame to Process : 42 :s 409
>> Decode Thread: Frame to Process : 43 :s 506
>> Decode Thread: Frame to Process : 44 :s 330
>> Decode Thread: Frame to Process : 45 :s 2180
>> 0:00:03.530095934  6548 0xffff8c01ff60 WARN               h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 29 will be dropped
>> 0:00:03.530192075  6548 0xffff8c01ff60 WARN               h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 117 will be dropped
>> 0:00:03.530251666  654
>>
>>
>> So then I analyzed the encoded_data to check if there are valid NAL units:
>>
>> have analyzed the encoded_frame data to determine if it contains valid
>> NAL units. Here's a summary of your findings for the first five data sets:
>>
>> Encoded data values in hexadecimal: 0, size: 3241
>> 00 00 00 01 27 7A 00 28 B6 D0 0C E8 07 80 22 7E 27 01 6A 02 04 04 80 00
>> 00 03 00 80 00 00 1E 19 A8 00 3D 09 00 01 6E 37 FF FE 05 00 00 00 01 28 EE
>> 3C
>>
>> Encoded data values in hexadecimal: 1, size: 246
>> 00 00 00 01 21 E2 01 08 42 DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00
>> 03 00 00 03 00 00 03 00 00 09 36 B5 84 05 B0 C9 8C DA C9 6F 9D F0 00
>>
>> Encoded data values in hexadecimal: 2, size: 232
>> 00 00 00 01 21 E4 02 08 42 DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00
>> 03 00 00 03 00 00 03 00 00 03 00 16 D0 00 00 01 21 00 1F E3 90 08 21
>>
>> Encoded data values in hexadecimal: 3, size: 854
>> 00 00 00 01 21 E6 03 08 8B 7F 00 00 03 00 00 03 00 00 03 00 00 03 00 00
>> 03 00 00 03 00 00 03 00 00 03 00 06 2C 00 00 01 21 00 1F E3 98 0C 22
>>
>> Encoded data values in hexadecimal: 4, size: 952
>> 00 00 00 01 21 E8 04 0E DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03
>> 00 00 03 00 00 03 00 00 03 00 00 97 80 00 00 01 21 00 1F E3 A0 10 3B
>>
>> It seems that the analyzed data contains valid NAL units, as expected. I
>> have attempted a similar test using a file source and launching the
>> pipeline with gst-launch as follows:
>>
>> gst-launch-1.0 -e filesrc location=ball2.h264 ! video/x-h264,
>> alignment=au, width=1920, height=1080, framerate=30/1 ! h264parse !
>> omxh264dec ! video/x-raw, format=NV16_10LE32, width=1920, height=1080,
>> framerate=30/1 ! filesink location=ball_gst.raw
>>
>> During this test, I did not encounter any issues or error messages with
>> GST_DEBUG=4. Additionally, the raw file was populated with bytes, unlike
>> the appsrc version where the content always appears empty (presumably
>> because it is not being decoded at any point).
>>
>> I even tried with a identity dump=true element between filesrc and
>> h264parse and I also have the same bytes I am injecting to appsrc So I
>> don't know what can be the difference
>>
>> Thank you in advance.
>> Best regards.
>> Israel.
>>
>>
>
>