NUBIE API design.

This is a design I was working on for an API called NUBIE, or Non-Universal Beta Interface Experiment. The concept was to make an extremely light and simple to use API that doesn't do much actual work itself and thus can be ported and extended easily.

I know this is extremely long, but please, if you wish to comment on anything, I ask that you take the time to read everything first as your question may be answered later in this email.

I'll probably make a bunch of references to VfW and DirectShow in here. That's because those are the API's I'm most familliar with so far, though I don't claim to be a master of either or even very proficient with them, I've just tried coding for them before. I'm sure most of the concepts in those APIs are repeated somewhere in some of the other APIs that have been discussed on this list, I'm just using them for examples of How Things Have Been Done Before(TM).

VfW's problem is it tries to mandate a strict synchronous flow of data by forcing input and output in the same function call every time, and does not allow any other API functions to be performed before return. The upsides are that makes it very precise for things like editing, but realtime playback is a little trickier as are some advanced techniques that are appearing in modern codecs like variable framerate and Bframes. The API structure itself is a fairly organized affair of message passing and switching, with some quirks in variation of parameters that MS seems to have everywhere.

DirectShow has some things right like timestamping and less strict data flow, but it takes too much control away from the application. Directshow graphs tend to want to run like their own autonamous programs and it can be difficult for an app to maintain control for things like editing. It also seems to me to be a fairly big mess of functions all over the place.


Now, as for the design I was working on. I was trying to simplify things as much as possible. The message passing idea can be useful as long as you keep to the rules of not making other functions and the messages stay organized and don't get out of control. To do this, you need to design a flexible system that can change what kind of data is sent to the function in case future needs change. So, I decided that the basis for the API would be an object with a virtual function Do() that would take as input an integer message and a data struct that would be defined for each message. After talking with the Matroska team on this for awhile, I figured out that EBML or XML would probably be useful in the design of these structures.

So, the API starts with a list of consts or #defines for people who use those. Basicly each valid message in the API is assigned an int number, and at the same time assigned a specific data structure associated with it, which can be just a simple ok/fail result indicator for messages that don't require i/o, but must always be defined. So we have Do(MESSAGE, MESSAGE_DATA) defined as the base function for the API. This is not to say it's the only function for the API, just that it's the primary one that most API objects will be concerned with.

The problem with message systems is that they can get very large, and expand from one version to the next as new messages are defined. Objects that support previous versions usually work fine with the messages they know about, but sometimes choke if they see something they do not expect. This can be solved by defining a default behavior for all messages that an object either does not support or does not recognize. Now, I just stated two examples, and most programming languages only have one default: in a switch statement, so what do we do with that? Well the default really should be for the case where we either do not recognize the message or do not handle the message. In this case, the message should be passed up the pipeline to the next object. For messages that we explicitly do not support, such as asking for data in a format the current object cannot generate, we return a defined fai
 lure code set in the struct that came with the message. Exception handling is the responsa
 bility of the caller. So if objectA says objectB.Do(FOO, foo_data1), and objectB knows it explicitly cannot support FOO or some condition specified by the FOO_DATA struct foo_data1, it will return an error code in the struct indicating failure. This code will be mandatory in all message structs. When objectA gets the result and finds it failed, it has the responsability of handling this condition and can perform whatever action it considers appropriate. This may include retrying the message with different conditions (like asking for VUYV, and if not asking for YV12, and if not asking for RGB), or it can pass the failure code back to its caller, which will then do a similar process. Also, some objects may require additional information or operations to complete a given message task. For example, if a codec was asked to render a frame, and it asks the container object to supply the next d
 ata, it may find this new frame is a B-frame. Thus it requires future data. It can then wi
 thin that initial message call generate more message calls to the container until it retrieves the data. Only once it can either fulfil the original message request or determine it is not possible will the original function call with the request for the frame as its message return.

Now, the problem with passing the exceptions up the pipeline is you eventually have to stop somewhere. In most encode/decode pipelines there are two ends, what some people call the source and sink. Sources can be anything like a file, video capture, etc, and sinks may be a file or a video renderer. These cases are different objects that don't support one side of the i/o system because they're endpoints. In this case, these objects will be defined differently. They still inheret the Do() function, but they also have other functions that the application talks to for the purpose of driving the pipeline. In the case of decode/playback, the pipeline is "pull" driven, that is the application will make requests of the rendering endpoint or sink. In the encode/capture case, the pipeline is "push" driven, and the application will make requests on the input source or source. Since all function req
 uests intiate at one point from the application, they all return to this point with either
  a success or failure. The application is in effect the "master" component of the system, and has the final say on what happens when there is an exception. It may prompt the user for more input, generate an error message, whatever. The API itself doesn't need to be concerned with this, and shouldn't be because there's no way to know what kind of application is running and what response is appropriate. For a simple media player, a simple "file format not supported" message might be fine. For a professional editing studio, a dialog box to change options might be more appropriate. But this is for the application to determine, not the API or the objects.

So, what do we have now?
We have determined that the application itself is master of the video pipeline (unlike directshow) and drives events.
A single non-varying message switch function with 2 parameters as the main interface (similar to VFW) that can be extended by defining new message/structure combinations, with an opt-out default case for unsupported messages and an explicit fail option that is handled by the caller.
Asynchronous two way communication (via messaging other objects before return and alternate method retries).

So far we have 3 object types, source, sink, and the two way. That latter case really should be split into two categories for ease of use. For two way objects, there are two variations seen very often. The case where input and output are the same format (filters), and the case where they are not (codecs). The main reason we might want to have filters as a seperate case is that by simplifying the situation so that only data content and not format is altered, the amount of work presented to the programmer is vastly reduced. By this I mean conversions such as one colorspace to another, which alters the content of an image. It does not include things like resizing, which alters its format and would affect the behavior of other components on the other side. Thus a fourth object type for filters would be useful. Filters need not be video filters, audio may be done as well, as can text (conside
 r a subtitle in/subtitle out that performs realtime language translation) or any other for
 mat defined in the API's message system. That's right, I said message system, not object. What does that imply? What if my video renderer that supports subtitle display sends a message request GET_SUBTITLE to the next component in the pipeline, which is a sharpening filter? The filter has no idea what to do with this message, so it applies the default case of pass up the pipeline. The codec recieves it next, and also has no idea what this message is, so passes it up the pipeline. The container parser then recieves it, and this object does know what a subtitle is and how to get it. It performs internal operations to extract the subtitle from the file, loads the data into the GET_SUBTITLE_DATA struct with timestamp and success flag, then returns. The codec recieves this and returns, the filter recieves it and returns, and the funtion resolves where the video renderer has just recieved a t
 ext subtitle packet from a video filter that has no concept or support for text. Neat huh?
  If the container did not contain text or the parser did not support it, what would have happend? Since the container parser is a source it knows it is a special case in that it cannot pass up the pipeline. So it would take the unknown message, locate the failure field in the struct (which remember has a standardized format like XML/EBML), and set it to fail, then send the result back. The video renderer would get a failure result, and could then notify the application that the request to play the video stream with subtitles failed. The application would then do whatever it deemed appropriate such as error messages, config boxes, or just retry playing without subtitles.

Now, I know someone is just dying to say "What about multiple inputs and outputs on an object?" I know this is coming up. Yes, objects can have multiple input and output objects. This is done by a list of pointers to inputs (jacks) and outputs (plugs). Each one points to the next component object in a particular chain. Doesn't this mess up the "pass the buck" system of handling messages? No. In the case of passing an unsupported message, it gets passed to the first component in the upstream list. If it comes back as a failure, pass it to the second component in the upstream list, etc, until we reach the last entry. If it still fails, return the failure to the calling function. Same thing but in the other direction in the case of encode instead of decode. This case may not even be necessary in many situations given the pass-through nature of the system. If you stuck an audio component inl
 ine with a video component, what would happen? All the video messages would be unsupported
  and pass through the audio object transparently. When the system asked for actual audio data, the audio component would intercept the message, and then if it sent requests upstream to a video component they would pass through and still get to the same source object.

What about maintaining sync? Just mandate timestamp fields in the relevant message data objects, or possibly all of them. Even if data is coming in through different "paths" in the system like video and subtitles, the timestamps will enable the application to maintain sync or perform correct editing. In the end, all data has to come from the source object, so if it is written to timestamp all the data it sends out, then when other objects pass the data back they will have timing info even if other objects in the pipeline are not time-aware, and the application can use this.

Now, there are a few other API functions the API needs to support, but this was the main one. Other functions would be loading and constructing the objects. This could work through simple lookup systems where a list of object types and the relevant code files are maintained. For example, say the application asks for an xvid decoder object to be created. The API would consult a list and look for that format in the list of decoders. It would then locate the specific dll or object file, load it and run its constructor, and then pass it back to the application so it can be initialized for use.

Format conversion is another often mentioned thing. I've got a generic frame object that can handle just about any video format (including obscure ones) with the same scan function. It does this by performing all the offset calculation and lookups for you. It also supports directly grabbing the format code and the raw bytes of video for objects that prefer to do their own work for speed. It would not be that difficult to add some conversion functions that internally convert colorspace on request. It would then be up to each specific object receiving the frame as part of the message object whether to use the internal conversion functions, or get the raw data and perform its own conversion for speed.

I know other people are now foaming at the mouth going "But we want to do it in pure C/asm/1890 census machine punch cards!" and that's fine if you can write something compatible. I know lots of people are already groaning about how inefficent and slow this is going to be. But what does it look like in the big picture? You're passing what, probably less than 10 messages per frame per object, even with retry cases. At a speed of what, maybe 100 frames per second tops? 1000 functions on a a system with a cpu speed in ghz? This is not going to be a significant impact. The main cpu intensive points are all going to be in the actual codecs and filters still, and this overhead will be minimal. As long as the option to do things the "traditional" way by getting the raw data is preserved, and I promise it will be a primary design goal to maintain that as a choice of the programmer (unlike java) 
 to allow that so that applications can remain as fast as they are now, while still being e
 asy to write with this system.

So in the end, what IS in the NUBIE API?

1. An easily extensible list of defined message and structure pairs.

2. Four specific object types regardless of data: sink, source, converter, filter.

3. A set of lookup functions to load appropriate object code.

4. A small amount of internal code in the message structures to promote ease of use by not forcing everyone to reinvent the wheel every time, with the option to bypass this and use raw access for speed in all cases at the programmer's choice.

5. A few basic rules for the behavior of objects and applications so that the system can be consistent.

That's it.
-- 
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm