Re: Design ideas for the editing system.

----- Original Message -----
From: Steve Lhomme <[email protected]>
Date: Wed, 28 Jan 2004 11:35:32 +0100
To: "The attempt to create a new,  open standard codec API" <[email protected]>,Discussion about the current and future development of Matroska <[email protected]>,General talk about Matroska and other products <[email protected]>
Subject: Re: [Media-api] Design ideas for the editing system.

> Toby Hudon wrote:
> 
> > Ok, now that we're supposedly working on the file format, I've been thinking about what we really need to put in there.
> > 
> > Basicly my main idea is that we have something that looks like this:
> > 1 or more major media types (audio/video/text), specifics like codec or colorspace are unimportant.
> 
> I agree if the architecture can be extended later. Any hacker will be 
> able to add it's own format then. As long as we can handle the most 
> tricky cases.
> 

I agree, extensibility should be a major goal here. What's considered the tricky cases at this point?

> One feature we need is the easy transcoding of AVI files into MKV/MKA 
> files. With an option to keep VfW/DShow/ACM compatibility or convert to 
> the native Matroska format.
> 

I think our best best is the photoshop-like approach. Force everything into a single multitrack MKV for editing when we load it as a source, then perform conversion back to other formats like AVI at final render time if the user specifies that in options. Since editing assumes we're going to be making changes, there's no point in preserving an existing container structure since we'll need to completely tear it down to edit and rebuild it for final output anyway. For transcoding I suggest we just completely rip the existing streams out of the source file with appropriate container demuxers and re-mux them into the project MKV as direct stream copy, so there's no decode/encode. That should be possible, right?

> > For each media type, there exists ONE control stream, and one or more source streams. All have timecodes.
> > 
> > The control stream basicly is the master timecode system, and the timecodes in each control stream (not source streams) should be linked so they always sync, this way we can make sure our video and audio line up.
> 
> Why do you need a 3rd stream to have 2 streams in sync ?
> 

You don't, I'm saying let's keep the video and audio control streams seperate. Why? What if we have filters with the same names but one's for audio and one's for video? They'll have unique ID codes of some sort but still why have the confusion? Also, what if I decide I like how my video was edited but my audio is crap and I want to scrap it and redo the track, say changing my background song for a scene. Might be convenient to be able to take the audio info out seperately without having to parse the entire editing script for audio data and selectively delete it while keeping audio.

> I suggest we have a look at the possibilities of SMIL instead of 
> reinventing the wheel.
> 

Well you know me, I've been coding in a cave, haven't seen that many wheels around yet. If you know of suitable ones by all means let's use em.

> > By putting everything in the control stream, this means we can take source tracks from existing files (direct stream copy) without modification other than just parsing them into the container, regardless of codec or attributes. This means adding a new video source track to an existing edit project should be as simple as selecting a new file to read source from, and doing a data copy into a new track on the project file. Granted there's probably going to be some sort of fun involving interleaving the new data etc, but that should be transparent to this process and handled by the container for the editing file format.
> 
> Yes, the Direct Stream Copy is something really important.
> 

Definately. It probably needs to be the first thing that actually works. Hard to edit without any data. :P

How hard is it to just link up say an avi demuxer and plug the outputs into an mkv muxer? Does that actually work as a direct stream copy?

> > Rendering a project to a final output file is as simple as reading the control stream in order, displaying the relative parts of the source streams as needed, and calling various rendering plugins to handle transitions and effects at specified times. All we need to do is take the desired output samplerate/framerate of the video we're creating, and generate the result at each point in time we need a frame. This may require some interpolation but that's things that should probably be handled "transparently" by the editor for ease of use. I realize this will be much harder than it sounds but at least it should only need to be written once if done right.
> 
> IMO the pull model might not be a good idea. But why not. We need to 
> make sure it works fine with VFR content too.
> 

Ok VFR is exactly why I want a pull model. Say our source file has source tracks at different framerates, some of which are varaiable. Say our output file has its own designated framerate, possibly vfr, possibly cfr, totally unrelated to the source framerates. If we want to render correctly, which frames do we NEED to generate? The frames in the output file. We don't care what the input was, as long as the output looks correct. So if our output is 24fps CFR we need to produce a frame every 1/24sec regardless of the input source rate or status. To do this we should request the current data at each 1/24 second, which may or may not fall on a discrete frame in any particular source track. That's when we need to worry about are we interpolating by mixing other frames, are we just copying the data from the last discrete frame (particularly with VFR sources), etc. There needs to be some kind o
 f ruleset governing how we handle this case, but the goal of it all should be the proper r
 endering of the output file, hence the pull model. Also if you have a slow output clip, say 15fps, it's alot less cpu intensive than using a push model with 60fps source clips that are going to need to decode, filter, and render 4x as many frames as you need. Since everything in scripting is time-aware, it shouldn't matter what timecode we pull at, the renderer for transitions and such should be able to handle image processing based on the specified time value and the parameters in the control stream.

-- 
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm