Re: NUBIE API design.

----- Original Message ----- 
From: "Toby Hudon"
Sent: Wednesday, December 31, 2003 11:56 AM
Subject: [Media-api] NUBIE API design.

/* Now, as for the design I was working on. I was trying to simplify things
as much as possible. The message passing idea can be useful as long as you
keep to the rules of not making other functions and the messages stay
organized and don't get out of control. To do this, you need to design a
flexible system that can change what kind of data is sent to the function in
case future needs change. So, I decided that the basis for the API would be
an object with a virtual function Do() that would take as input an integer
message and a data struct that would be defined for each message. After
talking with the Matroska team on this for awhile, I figured out that EBML
or XML would probably be useful in the design of these structures. */

EBML might be better since it can handle all data types fine.

/* So if objectA says objectB.Do(FOO, foo_data1), and objectB knows it
explicitly cannot support FOO or some condition specified by the FOO_DATA
struct foo_data1, it will return an error code in the struct indicating
failure. This code will be mandatory in all message structs. When objectA
gets the result and finds it failed, it has the responsability of handling
this condition and can perform whatever action it considers appropriate.
This may include retrying the message with different conditions (like asking
for VUYV, and if not asking for YV12, and if not asking for RGB), or it can
pass the failure code back to its caller, which will then do a similar
process.  */

This is a nice and simple way to handle this type of problem.  The
alternative is forcing everything to publish what all it supports and have
the connections automatically determine what to use.  This can potentially
create problems with longer chains if the automatically chosen format isn't
supported further down.

/* The application is in effect the "master" component of the system, and
has the final say on what happens when there is an exception. It may prompt
the user for more input, generate an error message, whatever. The API itself
doesn't need to be concerned with this, and shouldn't be because there's no
way to know what kind of application is running and what response is
appropriate. For a simple media player, a simple "file format not supported"
message might be fine. For a professional editing studio, a dialog box to
change options might be more appropriate. But this is for the application to
determine, not the API or the objects. */

'nuff said.  The API passes messages to the master, and then it determines
what to do.  DirectShow is the greatest blackbox of media.  Half the time
you don't really know what its doing or why.

/* Now, I know someone is just dying to say "What about multiple inputs and
outputs on an object?" I know this is coming up. Yes, objects can have
multiple input and output objects. This is done by a list of pointers to
inputs (jacks) and outputs (plugs). */

As a matter of fact I was about to ask this very question, so nevermind.

/* What about maintaining sync? Just mandate timestamp fields in the
relevant message data objects, or possibly all of them. Even if data is
coming in through different "paths" in the system like video and subtitles,
the timestamps will enable the application to maintain sync or perform
correct editing. In the end, all data has to come from the source object, so
if it is written to timestamp all the data it sends out, then when other
objects pass the data back they will have timing info even if other objects
in the pipeline are not time-aware, and the application can use this. */

This should be obvious to anyone.  The lack of timestamps is causing many of
the issues in the first place.

/* I know lots of people are already groaning about how inefficent and slow
this is going to be. But what does it look like in the big picture? You're
passing what, probably less than 10 messages per frame per object, even with
retry cases. */

I think that most of the comments that I've seen so far have been about
passing the video frames.  IE, if you have to pass the same video frame 5
times, that is a lot of memory bandwidth.  If you are working with
1920x1080x32bpp @24p, then you have almost 1.6 Gigabytes per second just on
video frames.  Any copying of the video buffer that you have to do at this
point is going to stretch the limits of many computers.


Pamel

Errrr.... Where is the news access?