Re: Design ideas for the editing system.

Toby Hudon wrote:

>>I agree if the architecture can be extended later. Any hacker will be 
>>able to add it's own format then. As long as we can handle the most 
>>tricky cases.
>>
> 
> 
> I agree, extensibility should be a major goal here. What's considered the tricky cases at this point?

WARP ? ;)
Also control tracks (which we partly know what they'll do), menus, frame 
references.

> I think our best best is the photoshop-like approach. Force everything into a single multitrack MKV for editing when we load it as a source, then perform conversion back to other formats like AVI at final render time if the user specifies that in options. Since editing assumes we're going to be making changes, there's no point in preserving an existing container structure since we'll need to completely tear it down to edit and rebuild it for final output anyway. For transcoding I suggest we just completely rip the existing streams out of the source file with appropriate container demuxers and re-mux them into the project MKV as direct stream copy, so there's no decode/encode. That should be possible, right?

Well, if the script is just an XML file it can be attached to an MKV 
file. The problem of storing everything in Matroska is that each stream 
has to be in sync. Or that will not be a file that is meant to be 
played, just stored. As so there's no need to use Matroska in this case. 
  A .tar should be enough (or .KAR/.MKR an EBML archive as planned a 
long time ago).

> How hard is it to just link up say an avi demuxer and plug the outputs into an mkv muxer? Does that actually work as a direct stream copy?

Sure, that's what mkvmerge does well. Direct Stream Copy just meant that 
coded data (in the sense of codec) are used untouched.

> Ok VFR is exactly why I want a pull model. Say our source file has source tracks at different framerates, some of which are varaiable. Say our output file has its own designated framerate, possibly vfr, possibly cfr, totally unrelated to the source framerates. If we want to render correctly, which frames do we NEED to generate? The frames in the output file. We don't care what the input was, as long as the output looks correct. So if our output is 24fps CFR we need to produce a frame every 1/24sec regardless of the input source rate or status. To do this we should request the current data at each 1/24 second, which may or may not fall on a discrete frame in any particular source track. That's when we need to worry about are we interpolating by mixing other frames, are we just copying the data from the last discrete frame (particularly with VFR sources), etc. There needs to be some kind
  of ruleset governing how we handle this case, but the goal of it all should be the prope
r r
>  endering of the output file, hence the pull model. Also if you have a slow output clip, say 15fps, it's alot less cpu intensive than using a push model with 60fps source clips that are going to need to decode, filter, and render 4x as many frames as you need. Since everything in scripting is time-aware, it shouldn't matter what timecode we pull at, the renderer for transitions and such should be able to handle image processing based on the specified time value and the parameters in the control stream.

BTW, that makes me think that we should be able to edit a video based on 
the audio timecodes, not video. So if the audio to start/stop the movie 
is in the middle of a video frame, we just cut the timecode of the video 
frame.

The problem of a pull model is that you know when a frame (audio/video) 
starts, but not when it ends. Or maybe it should be included in the 
pulling request (ending timecode)...