Storing header backups, and other important stuff

Hi all!

Previously we discussed this topic on the IRC channel and agreed to use 
Magic Blocks for storing header backups, this way:
{[MagicBlock with code for header backups]}(Main Header)...{[...
Legend: {}=Cluster, []=Block, ()=other Element, ...=something

There is one obvious problem in that approach: demuxers which already 
have those headers and don't need to read backups have difficulties 
skipping those headers.

Because of that, I propose that we store backups this way:
{...[MagicBlock that contains header backups]}{[...

Here players could easily skip that Block, just like they skip any 
unknown Block. When reading headers from it, it would contain Main 
Header (1024 octets or bigger) and optionally other Elements and then 
link (Clusters Element position) to the next Cluster start.

Albeu has his own way for storing headers (any (keyframe) Block could 
contain 'em and there would be a flag denoting if the header is there or 
not). His header suggestion is not related to any existing MCF header, 
but is a very stripped-down AVI-only header. The header he is suggesting 
is way too simple for advanced container like MCF, so it will need to be 
enlarged quite a bit until it gets usable.

What's good with Albeu's header is that it is clued to the video Block 
itself (Block explains its settings) and is quite a bit smaller than 
normal MCF headers.

What is bad is that the approach requires entirely different header for 
each Track Type, requiring a lot of work for implementing support for 
that. It also requires players (even if they already have all headers) 
to check that flag, then check header size (if that header is going to 
be extendable, it will need to be variable-sized), then skip it.

The option I propose is in specs now (not uploaded yet, as I'm cleaning 
it up a bit and waiting for your opinions).

--

Another issue is changing settings (video formats, etc) inside a file. 
In my opinion this shouldn't be supported (makes many things a lot more 
complex and makes recovering data from broken files more difficult). 
Instead, pieces of a stream with different settings should be saved to 
separate files. That system would work in streams/broadcasts too: when 
settings are changed, the first stream is terminated similar to the way 
files are terminated and then the new file starts just like MCF-files 
normally do, but with different settings.

--

Yet another issue is Linear Write (ie. it is impossible to seek back to 
the beginning of file to insert some values such as total file length).

--

Not enough already? Then take Seek Entries. The current implementation 
of those sucks.

--

My ETA for things listed here is next weekend, so it is better to hurry 
up a bit if you want to comment any of these things.

--

What comes to Transor (codec) API and other things, we need help from 
some people. That API is the biggest obstacle at the moment: we can't 
have any usable software until that API is working. Especially codec- 
and player developers could help there.

Most of you don't have a clue of what Transor is, so I'll try to explain 
it here.

Transor is a codec with API designed for MCF. Normal VfW or other codecs 
can't directly work with MCF because some extra header data must be 
added and some transformations might be required too. This allows 
different codecs for same format (ie. XviD and DivX for MPEG-4) to 
co-operate. Also, MCF headers contain much information about the data 
which AVI or other formats don't contain.

Of course it is impossible to get all codec developers to release 
Transors of their codecs, so some Transors will just pass that data for 
some other API, like VfW, and only add that extra data etc. One example 
of this is Vorbis: inside AVI, Vorbis is actually stored in Ogg (because 
AVI cannot contain Vorbis directly). MCF can store Vorbis without such 
hacks. If normal AVI Vorbis codecs were directly used with MCF, that 
would result in having Ogg inside MCF and this is definately not what we 
want.

What this means is that with MCF, players only need to support Transor 
API, nothing else. If VfW or other codecs are required, Transors take 
care of that.

Yes, this is more difficult to implement (in existing players) than Ogm 
(=AVI in different container), but in future this is the only way to get 
rid of AVI legacy (=pile of hacks). This should also make it possible to 
use both AVI and QT codecs with MCF, and to extract QT-compressed data 
with AVI-codec or vice-versa (only works with somewhat open formats, of 
course).

Transors are libraries, .dll, .so or whatever, so it is easy to install 
new ones (no need to recompile anything). They are identified by Format 
Name (16 octets) and other fields in MCF headers.

- Tronic -