Re: to be effcient or not to be efficient

----- Original Message -----
From: Steve Lhomme <[email protected]>
Date: Wed, 07 Jan 2004 15:03:34 +0100
To: "The attempt to create a new,  open standard codec API" <[email protected]>
Subject: Re: [Media-api] to be effcient or not to be efficient

> Steve Lhomme wrote:
> > Consider this :
> > struct _blah {
> >     uint16 Size;
> >     uint8  FieldA;
> >     uint32 FieldB;
> >     char   FieldC[30];
> > };
> > 
> > Now (after a few years) you realise that you need FieldA to be on 2 
> > octets instead of 1. You simply can't. And you can with EBML.
> 
> BTW, my point is that plain C structures are too simple to have good 
> backward/forward compatibility.
> 
> But I understand that EBML is too complicated for what we need. What we 
> need is a "classic" TLV system : [Type] [Length] [Value]. EBML is a 
> special TLV because the Type and Length have variable length. We don't 
> need this here. I think we just need the Type on 2 or 3 octets (2 should 
> be enough) and the Length on 4 octets (handle up to 4GB of data).
> 
> Then as for EBML, each Type (ID) would have its own meaning and type : 
> uint, sint, float, date, string, unicode string, custom, Master. The 
> Master type means that it contains other IDs. This way you could have :
> 
> [MessageID][Length][
> 	[FieldA][Length][Value]
> 	[FieldB][Length][Value]
> 	[FieldC][Length][Value]
> ]
> 
> Of course it depends if you prefer compatibility more (or less) than 
> speed. But as a communication means, it's more general. Then internaly 
> you are free to use structs directly. You just then have to convert the 
> data between the struct and the "EBML" data when you need communication.
> 
> Passing a message in such a way would also contain the data directly. 
> You just pass a pointer to the "Message ID" buffer. Instead of Message + 
> Data + Length of data.
> 

The main reason I was doing a two parameter Do(MSG, MSG_DATA) instead of just having one parameter with the message value embedded in the first field of the data is the speed issue that was raised before. Switching on a single int message code is fast, and decoding a complex data struct like EMBL is slow. Thus with just the message code as the first parameter we can do very fast message passing and switching because there's no decode involved, just a quick value compare. If you put the message in the data struct, then every object needs to access and decode every passed message, whether it handles it or not, and that's going to be much slower. Even if you just grab the first few bytes instead of doing a real full parse, you still need to seek the data pointer. That means time to resolve the adress, and grab the entire data object. This could potentially cause cache to start thrashing if 
 you're constantly trying to read a message object that potentially doesn't fit in the L1 o
 r L2 cache, at a potential rate of a thousand times a second. 

Other than that one message code, I agree we can just put everything in the one pass by reference data parameter.

As for latency, what do you think the benefits would be of a message "timestamp before call" latency field? That is, when the application issues a message, the last line before the actual passing to the pipeline is a timestamp into a field in the data. Then when each object gets the data, it does the same thing into a seccond field (so the original timestamp of the call is preserved but the second one overwrites with each "update"), but only if it modifies the data. Thus each app can compare the existing field values and know its own latency on the system it's being run on. When the call is finally finished and the app gets it back, it knows what the latency of the pipeline was. We could use this information at the app or even component level to decide when to drop frames or change types of processing to remain realtime.

As for threading, if we plan to have all frames on a timestamp system where we know the true relative time to the video of incoming data, then we can perform asynchronous operations and then resolve ordering issues later, as long as the latency (which we can track as mentioned above) does not cause performance to drop below realtime, assuming this particular application cares about that, which many editors/encoders don't anyway. Thus you can just thread multiple messages from the app, like get video, get audio, get subtitles, etc. If everything that needs to be is time-aware, it doesn't matter if we recieve requests for data out of order, that can be resolved by the time indexing. If you just thread the original app message calls, there's no reason you can 't have 8 pending frames in the pipeline with filter operations being run on seperate CPUs. They can be sorted out when they come in 
 by the timestamp and latency info. Yes, there's still potential issues with timeslicing an
 d the overhead threads incur. The point is if we have this as an OPTION, it's not so bad. Thus a realtime media player may not want to use this, because of the overhead. A non-realtime video editor may want to for the increased processing speed. Ideally it should be as simple as the application deciding how many message calls to be "in flight" at one time by creating a thread list and then dispatching message calls into it as threads read completed. It's the app's responsability to handle asynchronous data using the timing info provided if it CHOOSES to do this.

The main problem I have with implementing this approach right now is the seeking issue, which I'm still trying to figure out the best way to handle. Say we multithread and send out calls for frameA and frameB. If for some reason the request for frameB reaches the codec first, the codec (based on my codec here) could potentially need to change segments and thus would need to drop the existing data, and swap to the next multiframe block. Then when frameA comes in, it's in the block that was just discarded. That would mean needing to dump the existing data and get back the previous "keyframe" data, decode it, and serve the frame. But then we're going to need to repeat what we did for frameB for the next frame after that, so we've just tripled the work. There's probably a way to solve this issue with either some kind of code design or some kind of rule system about how threads are dispatched
  to make sure this doesn't happen at the codec level. I haven't figured it out yet. It doe
 sn't really "break" anything, it's just very inconvenient for performance if this happens.

The main use of multithreading is not likely to be at the container or codec levels. Codec multithreading will probably not work as frame to frame multithreading but as threading between processes on the same frame/group of frames while in decode. Thus that's a seperate issue and invisible to the API anyway as it's all in internal code. The point of API multithreading would be filter handling or multiple streams. If you have a lot of cpu-intensive filters, it would be beneficial to run them on multiple CPUs. Realtime players tend to have less of both cpu intensive filters and access to hardware with multiple cpus than non-realtime editing or encoding apps, so I'd imagine only these apps would want to use a multithreaded mode. If we make it as simple as just dispatching multiple messages in threads (and yes I know there's issues here about access to data and modification of data), people 
 can write their apps to do whatever they feel is best and the API shouldn't have to worry 
 about it other than asking component and app developers to "play nice" in case things are being done in threads. By this I mean attempting to avoid the above case with the out of order codec seek.
-- 
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm