decoder discoveries

Hello all,

I have been working on the media_decoder app to better understand how the communication 
between the extractor and decoder occurs in R5.  This all happened because I ran into a problem 
because I had an issue with the meta data fields.  (the were not being copied)  So, I went to R5 to 
see how/if they were copied.  Well, it turns out all fields are copied verbatim, with one little 
gotcha.  It seems the _area fields are somehow treated specially if they are not set to 
B_BAD_VALUE.  But this email isn't about that.

Initially we used the info field to pass header information to the decoder from the extractor.  
We abandoned that because our test case app "media_decoder" was able to succeed without 
passing any info in.  (also there was a related issue with the bebook description of the info field 
which is code-wise impossible)

So, I changed over to using the meta data field to store this information, since the "format" is the 
only argument passed in to the BMediaDecoder.  This works well for the ogg/vorbis pair, ogg/
speex, etc., and seems to have no theoretical problems associated with it, with the minor issue 
about copying between address spaces, which seems to be able to be solved by the conveniently 
located meta_data_area variable.

Well, I thought I had it all figured out when I went to test the copying behavior on R5.  Here's 
what I found out:

The format returned from EncodedFormat is fairly spartan.  In the case of mp3, it has the 
following fields set:  type, output.frame_rate, output.channel_count, output.buffer_size, encoding, 
bitrate.  In the case of ogg, it has the following fields set: type, output.frame_rate, 
output.channel_count, output.format, output.byte_order, output.buffer_size, encoding.

Notably missing from the above lists are: user_data, meta_data, or any other "miscellaneous" 
field where header information can be stored.  From my other observations, it's clear that the 
only thing that matters here is really the "encoding" field which maps directly to some specific 
decoder.

In media_decoder I used the format from EncodedFormat to create a BMediaDecoder.  Because 
of this lack of header information, it is impossible for the BMediaDecoder to be completely set 
up after this construction.  That is to say, it is generally unprepared to start processing encoded 
data. (header-less formats don't suffer from this but they are beside the point)

The next step in media_decoder is to get the DecodedFormat from the track.  This returns a 
format of RAW_AUDIO type, with some parameters set in the raw_audio type.  Perhaps the one 
interesting thing in it is the deny_flags which are set.  Still, none of meta_data or user_data, etc., 
are set to any interesting values.

At this point I call SetOutputFormat on the BMediaDecoder.  The decoder still lacks header 
information and can not completely initialize.  (a side note: GetNextChunk is not called during 
SetOutputFormat)  In my opinion the R5 decoders are in a pretty bad state right now.  They have 
only the bare-bones information passed to them from the extractor through a handful of fields in 
the media_format, and yet they are asked to negotiate the output format for data that they 
haven't even seen the headers for.

But moving on, the next step is to call Decode on the BMediaDecoder.  Remember, we still 
haven't seen the headers for the encoded data yet.  Finally, GetNextChunk is called on the 
BMediaDecoder.

The input media_header is not the media-header that I provided in my call to Decode.  In fact, 
the media_header is a pointer to a bunch of garbage data.  The media_header isn't even 
initialized to zeros.  In my current implementation for GetNextChunk I zero it immediately.  It 
worked before I added the zeroing, and it works after too.

Next I called BMediaTrack::ReadTrack(....).  For mp3, this set the following fields on the 
media_header: type, file_pos, orig_size.  for ogg, it set: type, start_time, file_pos, orig_size.  I 
returned the chunk data.

When I BMediaDecoder::Decode returned, it had modified my media_header that I provided. 
(remember: this is a different one from the one I saw in my GetNextChunk method)

For mp3 the following fields were set: type, start_time, orig_size.  This means that the mp3 
decoder added the start_time for the chunk during decode, because the start_time was not 
available earlier after ReadTrack.  This probably contributed to the R5 mp3 sync problems.  
Perhaps also notable is that the file_pos often does not get incremented sometimes, even though 
we advance through start_time.  This seems to be because the file_pos always points to the start 
of the chunk that the samples are in.

For ogg the following fields were set: size_used, start_time, file_pos, orig_size.  A notable 
difference between ogg and mp3 is that ogg provided the size_used field set to the appropriate 
value.  If I had used the size_used field from mp3, I would have dumped a whole bunch of 
nothing into my output file.  (my code used the buffer_size from the raw_audio format, but using 
size_used should have been okay too, I think)  Like mp3, file_pos is not always incremented and 
seems to be related to the start of the chunk the samples are in.

Okay, so at this point you should probably be asking: how did it work?  Perhaps we can jump 
into the middle of an mp3 stream and figure out things, but vorbis certainly needs codebooks 
from the header in order to perform any kind of sane decode.  It turns out that the mp3 reader/
decoder and the ogg reader/vorbis decoder used different strategies.  I examined the chunks 
returned from the NextChunk method for the mp3 track and the ogg track.

The ogg track seemed quite odd to me as I was expecting to see ogg packets in it.  There were 
none. (!)  In fact, the implementation of the ogg extractor in r5 is to completely decode the vorbis 
stream and pass the raw decoded data to the "vorbis decoder".  The chunks that I got in 
NextChunk were the same as the chunks that I got out of decode, and the same as I was writing 
into my file.

The mp3 track on the other hand, was the exact opposite.  The chunks that came through were 
identical to the input file.  (this may make more sense in light of the structure of mp3)  So, the 
mp3 decoder was basically operating on an unmodified file stream.

So, in both these cases there was no header communication required between the two levels.  I 
think that we may all agree that the ogg/vorbis strategy is not the path that we want. (we haven't 
taken it and it's working okay so far)  The approach that we have used that involves using the 
meta data is not as invulnerable as placing header parsing and data decoding in the same 
module, but it does get us back the power and elegance of re-usability of the extractors and 
decoders.  (this is already demonstrated by our ability to do useful things with ogg files that have 
some streams we can't yet decode)  Using the meta_data will have our kit be compatible with 
media_decoder once I fix the copying of the media_format to preserve meta_data on copying, as 
R5 does it.  It will also allow the codecs to give stronger and earlier feedback about stream 
properties and their ability to handle them.

additional notes on other formats:

I checked how the decoding works for an mpeg video/audio stream as well.  The chunks for the 
audio stream contained only the audio bits, as one would expect.  There was still no extra 
information passed through meta_data, etc.  The ReadChunk method set type (to 
ENCODED_VIDEO !), file_pos, orig_size.  The decoder also set type (to ENCODED_VIDEO !), 
start_time, file_pos, orig_size.

I checked how the decoding works for an avi video/mp3 stream as well.  The chunks for the 
audio stream contained only the audio bits, as one would expect.  There was still no extra 
information passed through meta_data, etc.  The ReadChunk method set start_time to some 
apparently useless thing (approx -9.223e20), and set file_pos, orig_size.  The file_pos was not the 
start of the chunk, may have been the end of the chunk but was not equal to the location of the 
data in the file + orig_size either.  The decoder set type, start_time.  It did not set file_pos or 
orig_size.

The version of media_decoder in cvs now finds the first audio stream in a file and decodes it.

Andrew

P.S. For those of you reading this who may possibly not know what our goals are with respect to 
codecs: we are not planning a binary compatible solution for the plugins.  we _are_ planning a 
binary and source compatible solution for the public APIs, in accordance with the general 
openbeos principle.  This exposition is for informative purposes only, and development will 
proceed according to our best judgement of how to implement the behavior specified and 
exhibited by the public APIs.  To the extent that we fulfill this goal we may or may not choose to 
implement it in a way that resembles the R5 implementation.  This is even more true since I 
haven't seen the R5 implementation. :-D