Re: Python access to video frames

HI Karl,

I would probably do this with FFMPEG and OpenCV.

Step 1: Use FFMPEG to strip the mp4 sound track.
Step 2: Use OpenCV to convert the video into sequential frames
Step 3: Read the Frames into a Mat object in OpenCV, detect squares, and
then crop over the previous frame.
Step 4: Use OpenCV to stitch the frames back into an MP4 file.
Step 5: Use FFMPEG to add back the sound track.

Gstreamer for video portion you would have two pipelines. Filesrc ->
jpegenc -> appsink. You then hookup a callback to appsink store the frames
into memory, process them with OpenCV, then feed them into second pipeline
appsrc -> splitmuxsink.

You would probably need a tee to bring in the audio in the second pipeline,
but others who are more familiar with this can comment.

Best regards.

On Thu, Oct 12, 2023 at 8:04 PM Carl Karsten via gstreamer-devel <
[email protected]> wrote:

> I need to alter video frames, and Python is my happy place.
> my gst skills need some help.
>
> I think I need these 2 things:
>
> 1. pipeline:
> read a file,
> decode the video,
> hook? to read/write pixels,
> encode (keep the original audio)
> save to disk.
>
> 2. that hook code...
> I have done something before: use the wavelevel element to get audio
> data and generate static png images.
> https://github.com/CarlFK/veyepar/blob/master/dj/lib/gslevels.py#L72
>
> So I am assuming some element / message will give me a frame buffer
> thing of data, and somehow i can detect black pixels.
>
>
> End goal: fix some video;s I made of a live event where my setup
> introduced black squares - I want to replace the black with the pixels
> from the previous frame.
> example of what needs fixing:
> https://youtu.be/N5Anbq8vYNk?si=Gl89X3Uk5_2CrMu6&t=148
>
> The slide is mostly a static image, so the previous frame will be fine.
>
> --
> Carl K
>


-- 
Sincerely,

Dwight Kulkarni