Re: yuv4mpeg API audio extensions

 >From: David McNab <[email protected]>
 >Date: Tue, 23 Oct 2007 08:17:00 +1300
 ...
 >On Mon, 2007-10-22 at 18:39 +0200, Bernhard Praschinger wrote:
 >> > I'm willing to work on some patches to the mjpegtools yuv4mpeg api to
 >> > add support for audio muxing/demuxing.
 >> How would that change the audio applications (lav2wav and mp2enc)
 >
 >We'd have to look into that and avoid breaking things.

I've gotta agree with most of the other sentiments expressed:  you're
 going to have a hard time with this.  To do it remotely correctly,
 you'd probably extend the YUV4MPEG2 format to add "audio planes"...
 and then there are issues, such as:

   o what do audio-unaware filters do?  (particularly filters
      which operate on multiple input or output streams)

   o what do filters which modify the video timebase do with audio?

   o need to handle "unlocked audio" --- formats where the
      A/V clocks are not tightly coupled, or which have sampling
      rates are not simple integer-multiples of one another

   o need to deal with YUV4MPEG2 "frames" of varying duration
      (e.g., depending on the field-repetition flags)

   o specification of audio data format

Bleah --- you're better off putting your energy and research ('cause
 you have a lot of research to do now) into adding some sort of
 stream algebra layer to your python library which understands
 timecodes and timebases, and keeps track of the accounting to
 modifications on streams, so that same/similar transformations
 can be carried over from one stream to another (which is the
 functionality that you are basically wishing for).

[More thoughts at the bottom...]

 >> From a point of view you need to the the change y4m_write_frame, 
 >> y4m_read_frame in a way that the audio data can be present.
 >> So you would need to extend the stream info (y4m_stream_info_t).
 >
 >Sounds good.
 >
 >> Would it be possible to pass the Audio data along when you just use the 
 >> y4m_read/write_frame. Well on a 2nd thought that does not seem possible.
 >> So that means to me you would have to change every program that it 
 >> passed the audio along. And understands the new data.
 >> 
 >> Or you just extend the y4m* functions with a 
 >> y4m_read/write_frame_with_audio to get the audio. So no other program 
 >> needs to be changed but does not pass audio through.
 >
 >The latter approach sounds better.
 >
 >> > Does this idea find some resonance among the mjpegtools developer
 >> > community, or does it horrify, or something in between?
 >
 >> No problem at all.
 >> I would be happy if you could provide some more information why you want 
 >> to do it. And how, basically I have not problem with extending things.
 >
 >I want to do it is because of my 'y4m' framework (a versatile framework
 >for high-level processing of yuv4mpeg format video (and any format which
 >ffmpeg supports), and which is easily extensible in C and python, refer
 >to www.freenet.org.nz/y4m).
 >
 >As this framework grows, gaining more effects and python classes, the
 >video-only limitation becomes harsher.
 >
 >For example, y4m makes it easy to cut/paste pieces of existing streams
 >into whole new streams. It has an abstract 'layer' class, which does
 >away with the traditional 'timeline' concept and replaces it with an
 >'edit tree'. Also, keyframed effects can be applied at any node in the
 >tree.
 >
 >This is great, but it makes life difficult when it comes time to re-mux
 >the audio data.
 >
 >At present, to re-mux audio, I have to save the y4m-edited video in a
 >lossless or low-loss format, open it up in Cinelerra, and use Cinelerra
 >to manually cut/paste/sync the audio tracks. This can get very labour
 >intensive and error prone.
 >
 >So I want to now extend this framework to be able to traffick the audio
 >together with the video, or optionally replace it with other audio.
 >
 >Note too that 'y4m' makes it extremely easy to write new stdio-based
 >yuv4mpeg stream processing utilities (similar to yuvfps, yuvdeinterlace,
 >yuvmerge etc).
 >
 >Also, note that y4m Stream objects can work with any video file or
 >source that ffmpeg can handle - whenever constructing a Stream that is
 >not in yuv4mpeg format, y4m starts up a background ffmpeg process which
 >converts it on the fly to a yuv4mpeg stream.
 >
 >Two approaches come to mind for accommodating audio in y4m:
 >
 >1) Sell the mjpegtools community on the idea of extending the yuv4mpeg
 >   format so that it now includes audio, or
 >
 >2) Have an additional ffmpeg process (to the existing one which converts
 >   arbitrary video to/from yuv4mpeg) - a second ffmpeg process for
 >   converting arbitrary media files to raw audio streams. Then, add
 >   support for raw audio into y4m, so that in a sense, audio data
 >   becomes another component of the abstract Stream (in addition to
 >   the existing Y, Cb and Cr component planes).
 >
 >On writing this reply, it's feeling more and more that any attempt to
 >extend yuv4mpeg with audio functionality might be opening a can of
 >worms, and that I might be best to add audio support just in the higher
 >level y4m classes.
 >
 >Thoughts?

Yeah --- if anything, do (2) "add an audio stream layer to the python lib".
 But go and study the design (and flaws) of something like Quicktime first;
 there are a lot of subtleties that are easy to get wrong, and plenty of
 prior and existing failures to avoid repeating.

-matt m.
(Unfortunately too busy these days to do anything productive,
 but still willing to wave arms frantically when someone wanders
 into a minefield.)


 >
 >Cheers
 >David
 >
 >
 >> 
 >> auf hoffentlich bald,
 >> 
 >> Berni the Chaos of Woodquarter
 >> 
 >> Email: [email protected]
 >> www: http://www.lysator.liu.se/~gz/bernhard
 >> 
 >> -------------------------------------------------------------------------
 >> This SF.net email is sponsored by: Splunk Inc.
 >> Still grepping through log files to find problems?  Stop.
 >> Now Search log events and configuration files using AJAX and a browser.
 >> Download your FREE copy of Splunk now >> http://get.splunk.com/
 >> _______________________________________________
 >> Mjpeg-developer mailing list
 >> [email protected]
 >> https://lists.sourceforge.net/lists/listinfo/mjpeg-developer
 >
 >
 >-------------------------------------------------------------------------
 >This SF.net email is sponsored by: Splunk Inc.
 >Still grepping through log files to find problems?  Stop.
 >Now Search log events and configuration files using AJAX and a browser.
 >Download your FREE copy of Splunk now >> http://get.splunk.com/
 >_______________________________________________
 >Mjpeg-developer mailing list
 >[email protected]
 >https://lists.sourceforge.net/lists/listinfo/mjpeg-developer
 >

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/