Re: yuv4mpeg API audio extensions

On Mon, 2007-10-22 at 18:39 +0200, Bernhard Praschinger wrote:
> > I'm willing to work on some patches to the mjpegtools yuv4mpeg api to
> > add support for audio muxing/demuxing.
> How would that change the audio applications (lav2wav and mp2enc)

We'd have to look into that and avoid breaking things.

> From a point of view you need to the the change y4m_write_frame, 
> y4m_read_frame in a way that the audio data can be present.
> So you would need to extend the stream info (y4m_stream_info_t).

Sounds good.

> Would it be possible to pass the Audio data along when you just use the 
> y4m_read/write_frame. Well on a 2nd thought that does not seem possible.
> So that means to me you would have to change every program that it 
> passed the audio along. And understands the new data.
> 
> Or you just extend the y4m* functions with a 
> y4m_read/write_frame_with_audio to get the audio. So no other program 
> needs to be changed but does not pass audio through.

The latter approach sounds better.

> > Does this idea find some resonance among the mjpegtools developer
> > community, or does it horrify, or something in between?

> No problem at all.
> I would be happy if you could provide some more information why you want 
> to do it. And how, basically I have not problem with extending things.

I want to do it is because of my 'y4m' framework (a versatile framework
for high-level processing of yuv4mpeg format video (and any format which
ffmpeg supports), and which is easily extensible in C and python, refer
to www.freenet.org.nz/y4m).

As this framework grows, gaining more effects and python classes, the
video-only limitation becomes harsher.

For example, y4m makes it easy to cut/paste pieces of existing streams
into whole new streams. It has an abstract 'layer' class, which does
away with the traditional 'timeline' concept and replaces it with an
'edit tree'. Also, keyframed effects can be applied at any node in the
tree.

This is great, but it makes life difficult when it comes time to re-mux
the audio data.

At present, to re-mux audio, I have to save the y4m-edited video in a
lossless or low-loss format, open it up in Cinelerra, and use Cinelerra
to manually cut/paste/sync the audio tracks. This can get very labour
intensive and error prone.

So I want to now extend this framework to be able to traffick the audio
together with the video, or optionally replace it with other audio.

Note too that 'y4m' makes it extremely easy to write new stdio-based
yuv4mpeg stream processing utilities (similar to yuvfps, yuvdeinterlace,
yuvmerge etc).

Also, note that y4m Stream objects can work with any video file or
source that ffmpeg can handle - whenever constructing a Stream that is
not in yuv4mpeg format, y4m starts up a background ffmpeg process which
converts it on the fly to a yuv4mpeg stream.

Two approaches come to mind for accommodating audio in y4m:

1) Sell the mjpegtools community on the idea of extending the yuv4mpeg
   format so that it now includes audio, or

2) Have an additional ffmpeg process (to the existing one which converts
   arbitrary video to/from yuv4mpeg) - a second ffmpeg process for
   converting arbitrary media files to raw audio streams. Then, add
   support for raw audio into y4m, so that in a sense, audio data
   becomes another component of the abstract Stream (in addition to
   the existing Y, Cb and Cr component planes).

On writing this reply, it's feeling more and more that any attempt to
extend yuv4mpeg with audio functionality might be opening a can of
worms, and that I might be best to add audio support just in the higher
level y4m classes.

Thoughts?

Cheers
David


> 
> auf hoffentlich bald,
> 
> Berni the Chaos of Woodquarter
> 
> Email: [email protected]
> www: http://www.lysator.liu.se/~gz/bernhard
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Mjpeg-developer mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/mjpeg-developer


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/