RE: Help with vaapisink

Hi Victor,
Thanks for the quick reply.
I've tried to install Weston and the basic pipeline works correctly.
The issue now is when I actually try to run a test sample of our application (basically a grid that displays multiple video sources). 
When I've run the test software without the window manger directly with fbdevsink I was able to display a 3x3 view, but with Weston and vaapisink even a 2x2 view is displayed with very poor fps.

Is it due to the compositor component? I saw that in gstva there is an additional vacompositor but unfortunately it is available only from gstreamer 1.22 (Ubuntu 23.04), our distribution is not yet ready for it.

Core of the test application hereafter for reference:

/* Create gstreamer elements */
  pipeline = gst_pipeline_new ("vaapi-sample");

  conv_rgb  = gst_element_factory_make ("vaapipostproc",  "RGBAconverter0");
  sink      = gst_element_factory_make ("vaapisink", "video-output");

  /* we add a message handler */
  bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
  bus_watch_id = gst_bus_add_watch (bus, bus_call, loop);
  gst_object_unref (bus);

  GstElement *compositor = gst_element_factory_make("compositor", "compositor");
  g_object_set(G_OBJECT(compositor), "background", 1, NULL);

  /* we add all elements into the pipeline */
  gst_bin_add_many (GST_BIN (pipeline),
            compositor, conv_rgb, sink, NULL);
  gst_element_link_many(compositor, conv_rgb, sink, NULL);
  for (int i=0; i< grid_rows*grid_columns; i++ )
  {
        static int curr_row;
        static int curr_col;

        if (i==0)
        {
                curr_row = 0;
                curr_col = 0;
        }
        else
        {
                curr_row=i/grid_rows;
                curr_col=i%grid_rows;
        }



        std::string ele_name_compositor_padsink = "sink_" + std::to_string(i);

        GstPadTemplate* compositor_sink_pad_template = gst_element_class_get_pad_template(GST_ELEMENT_GET_CLASS(compositor), "sink_%u");
        GstPad *sink_pad = gst_element_request_pad(compositor, compositor_sink_pad_template, ele_name_compositor_padsink.c_str(), NULL);

        g_object_set((sink_pad), "xpos", curr_col*(display_w/grid_columns), NULL);
        g_object_set((sink_pad), "ypos", curr_row*(display_h/grid_rows), NULL);
        g_object_set((sink_pad), "width", display_w/grid_columns, NULL);
        g_object_set((sink_pad), "height", display_h/grid_rows, NULL);
        gst_pad_set_active(sink_pad, TRUE);
        if (target_pad == NULL)
                target_pad = sink_pad;

        file_n = i+1;
        if (file_n >= argc) file_n = 1;

        //FILE SRC
        sprintf(Tmp, "filesrc_%d", i);
        source[i] = gst_element_factory_make("filesrc", Tmp);
        g_object_set(G_OBJECT(source[i]), "location", argv[file_n], NULL);

        sprintf(Tmp, "qtdemux_%d", i);
        demux[i] = gst_element_factory_make("qtdemux", Tmp);
        sprintf(Tmp, "queue_%d", i);
        queue[i] = gst_element_factory_make("queue", Tmp);
        g_object_set (G_OBJECT (queue[i]), "flush-on-eos", 1 , NULL);

        obj[i].idx = i;
        obj[i].queue = queue[i];

        if (demux[i] && queue[i]) {
                g_signal_connect(demux[i], "pad-added", G_CALLBACK(pad_added_handler), (gpointer) &obj[i]);
        }


        sprintf(Tmp, "parser_%d", i);
        parser[i] = gst_element_factory_make("h264parse", Tmp);
        sprintf(Tmp, "dec_%d", i);
        dec[i] = gst_element_factory_make("vaapih264dec", Tmp); //only h.264

        gst_bin_add_many (GST_BIN (pipeline),
            source[i], demux[i], queue[i], parser[i], dec[i], NULL);

        gst_element_link_many(source[i], demux[i], NULL);
        gst_element_link_many(queue[i], parser[i], dec[i], NULL);
        gst_element_link_pads(dec[i], "src", compositor, ele_name_compositor_padsink.c_str());
  }

  g_print ("Now playing\n");
  gst_element_set_state (pipeline, GST_STATE_PLAYING);

  

-----Original Message-----
From: gstreamer-devel <[email protected]> On Behalf Of Victor Jaquez via gstreamer-devel
Sent: giovedì 8 giugno 2023 11:21
To: [email protected]
Cc: Victor Jaquez <[email protected]>
Subject: Re: Help with vaapisink


Hi Davide,

vaapisink doesn't draw anything with its DRM backend, its drawing method is a no-op, while, either X11 or Wayland, they draw.

vaapisink is thought to be used in X11 or Wayland only. If you are willing to draw directly with DRM you can try kmssink, though zero copy might not work.

Another comment, for new code gstva, in gst-plugins-bad, is preferred over gstreamer-vaapi (check with gst-inspect-1.0 va)

vmjl

On Wed, 07 Jun 2023 at 12:43, Davide Molteni via gstreamer-devel wrote:
> Hello,
> I'm trying to test a simple pipeline using VA-API components, I'm running it on an custom system without any window manager (Yocto distribution), the goal is to use DRM, the pipeline is the following:
>
> gst-launch-1.0 filesrc location=$1 ! qtdemux ! queue ! h264parse ! 
> vaapih264dec ! vaapipostproc ! 
> 'video/x-raw(memory:VASurface),format=BGRx'! vaapisink display=4 
> display-name=/dev/dri/renderD128
>
> The pipeline works correctly and reaches the PLAYING state but I don't see anything on the HDMI monitor, if I use fbdevsink instead I see the decoded video (but obviously it consumes a huge amount of CPU).
>
> Hereafter some additional information:
>
> root@mn-el00d08919cf41:~# vainfo
> error: can't connect to X server!
> libva info: VA-API version 1.14.0
> libva info: User environment variable requested driver 'i965'
> libva info: Trying to open /usr/lib/dri/i965_drv_video.so libva info: 
> Found init function __vaDriverInit_1_14 libva info: va_openDriver() 
> returns 0
> vainfo: VA-API version: 1.14 (libva 2.14.0)
> vainfo: Driver version: Intel i965 driver for Intel(R) Broxton - 2.4.1
> vainfo: Supported profile and entrypoints
>       VAProfileMPEG2Simple            : VAEntrypointVLD
>       VAProfileMPEG2Main              : VAEntrypointVLD
>       VAProfileH264ConstrainedBaseline: VAEntrypointVLD
>       VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
>       VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
>       VAProfileH264Main               : VAEntrypointVLD
>       VAProfileH264Main               : VAEntrypointEncSlice
>       VAProfileH264Main               : VAEntrypointEncSliceLP
>       VAProfileH264High               : VAEntrypointVLD
>       VAProfileH264High               : VAEntrypointEncSlice
>       VAProfileH264High               : VAEntrypointEncSliceLP
>       VAProfileH264MultiviewHigh      : VAEntrypointVLD
>       VAProfileH264MultiviewHigh      : VAEntrypointEncSlice
>       VAProfileH264StereoHigh         : VAEntrypointVLD
>       VAProfileH264StereoHigh         : VAEntrypointEncSlice
>       VAProfileVC1Simple              : VAEntrypointVLD
>       VAProfileVC1Main                : VAEntrypointVLD
>       VAProfileVC1Advanced            : VAEntrypointVLD
>       VAProfileNone                   : VAEntrypointVideoProc
>       VAProfileJPEGBaseline           : VAEntrypointVLD
>       VAProfileJPEGBaseline           : VAEntrypointEncPicture
>       VAProfileVP8Version0_3          : VAEntrypointVLD
>       VAProfileVP8Version0_3          : VAEntrypointEncSlice
>       VAProfileHEVCMain               : VAEntrypointVLD
>       VAProfileHEVCMain               : VAEntrypointEncSlice
>       VAProfileHEVCMain10             : VAEntrypointVLD
>       VAProfileVP9Profile0            : VAEntrypointVLD
>
> root@mn-el00d08919cf41:~# lshw -disable mmc -c video
>   *-display
>        description: VGA compatible controller
>        product: HD Graphics 500
>        vendor: Intel Corporation
>        physical id: 2
>        bus info: pci@0000:00:02.0
>        version: 0b
>        width: 64 bits
>        clock: 33MHz
>        capabilities: pciexpress msi pm vga_controller bus_master cap_list rom
>        configuration: driver=i915 latency=0
>        resources: irq:129 memory:90000000-90ffffff 
> memory:80000000-8fffffff ioport:3000(size=64) memory:c0000-dffff
>
>
> Am I missing something?
>
> Thanks,
> Davide