RE: to be effcient or not to be efficient

My opinion about thread (however I am not a video/audio coder -yet-):

All processing from one filter/codec/demuxer to another is serialized,
meaning that not much can be parallelized (The codec must wait for the
demuxer output for the current video frame). Even if some may be
buffurized, this will just flatten the different processing time from
one frame to another, however this may be archived with a single buffer
at the end of the processing (Am I clear ? Is this right ?).

I don't see the benefit of using different thread for each stage of the
video/audio pipeline, however I may be wrong.

The real thread benefit is to use one thread for each entire processing
pipeline. For instance, pipelines are graphs in DirectShow, and each
graphs instance are encapsulated in one thread. This means that two
different videos playing in a GUI application will use two different
threads for each video (two pipelines, two DShow graph instances) and
one thread for the GUI (always a good thing to have a specific thread
for the GUI).

I think (abeit not sure) that VLC is actually using different threads
for different filter/codec/demuxer along the rendering pipeline (opposed
to mplayer/xine/dshow).


-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Attila Kinali
Sent: mercredi 7 janvier 2004 13:39
To: The attempt to create a new, open standard codec API
Subject: Re: [Media-api] to be effcient or not to be efficient

On Wed, 07 Jan 2004 09:36:46 +0100
Steve Lhomme <[email protected]> wrote:

> > There is a limited amount of information anyways, so you can add
> > 99% of it into the struct right now. If there will be any new
> > kind of information that could be added we can talk about a new
> > version of the api. And as it's really unlikely that elements
> > are removed you can even keep the structs compatible by adding
> > the new elements at the end.
> 
> This problematic case occured to me just 2 days ago. Where I had to
add 
> a boolean at the end of the structure. Hopefully for that structure 
> there is a Size parameter. Otherwise I should have used a new one (new

> DShow interface for the same functionality but extended). There would 
> have been 0 problem if the struct was in EBML (or similar). As you
said, 
> you can't design known for an unknown future. But you can make it
clean 
> for the future.

Yes, but if you seek extendability, you still don't need ebml.
You can do this just with normal structs, w/o any overhead beside
some pointer arithmetic, which is are 1-3 asm instructions compared
to a few hundred to thousand you need for ebml parsing.

> > Have a look at MPlayer G2s vp API (you have read my last mail, did
you ?)
> 
> Sure, but I expected something else than RTFM. As you obviously know
the 
> answer.

I can't give you that, because i don't know it myself.
I've read Richs design, asked him a lot of questions why
he did it like this an not otherwise. But still, i dont
understand more than 80-90% of it, because i lack a lot
of experiance of writing video filters and codecs which he has.

 
> OK. So what is the most successful video application ? I would say 
> DirectShow... But there is also Adobe Premiere or VirtualDub. I think 
> all of them use threads. BTW, multi-processor systems are best used
with 
> threads, right ?

I dont think VirtualDub uses threads for filters, at least i haven't
seen anything like this in the code. And as i already said, i don't
know DShow.
And your argument with multiprocessor machines is unfortunately void
as just a few freaks have more than one processor. The normal PC 
is something between a PII 400 and a PIII 1.5GHz and even lower
clock frequencies if you go to east europe, asia or africa.

 
> Sorry but I'm not a computer science man. I've learned mostly by
myself. 
> So I don't any books like what you mentioned. And I still like to
learn 
> from other persons like you. Anyway I'm almost sure that the context 
> switching for threads is faster than for process. It is equal in the 
> very worst case.

No worries, all good coders i know learned everything themselfs.
Context switches are the same for both threads and tasks, the only
difference is that you can save the TLB (translation lookaside buffer)
flush if you swtich between two threads of the same task. But this
is just a marginal saving IMHO (cannot proove it with numbers from
a real world application).
 
> Also my point is to have a very low latency system. Between 5ms and 
> 20ms. Which is far smaller than the duration of a video frame.

Trough a filter ? or to the output modules ?
If you mean the output modules, than you should reconsider it.
A video player wants as much buffer from the hardware as it can get
because it doesn't cost anything to use it (unlike self build buffers
in RAM). A interactive application on the other hand wants to minimize
overall delay and thus keeps those buffers small. IMHO the amount
of buffer in the output modules should be controlled by the application
and not by the API.


> So yes, 
> any performance impact has to be minimized as much as possible. But I 
> highly doubt this is the case for threads. At least on the Windows 
> platform where threads are a key feature. I know in the UNIX world 
> things are just starting to move (probably some people realised the 
> power of threading ?).

LOL, consider that windows didn't even have a proper multitasking
for until recently.
Well, threads, like all techniques, should be used there where they
make sense. If i'm going to write a GUI for something, i'll write
it as a frontend/backend system with threads. If i'm going to write
a video application i'll put everything in the critical path between
input (file/stream read) and ouptut (graphics, soundcard or file)
into one thread/task as i believe that it threads aren't worth
the overhead they have.

> > heh... The efficiency comes from this pushing and pulling.
> > If you manage the buffers the right way you can pass the data
> > right away trough filters that do not touch that part of the image
> 
> Sorry but this has nothing to do with push/pull/threads. This can be 
> achieved in all cases.

Depends, this is the real hard task when designing a API.

> > at all. Beside, a filter that takes a lot of time to do its work
> > will have an impact anyways as it will delay the picture.
> 
> Not if it has lower priority or even equal priority than other
threads. 
> If other similar threads have the same priority they will still have 
> processing time while the big thing is being done. This is not the
case 
> with a 1 task/1 thread system. Each stage has to cache a lot in case 
> another stage is too hungry for a while.

Why should a filter in the chain have a lower priority than others ?
The data must go trough it, no matters what.


> >>Also you didn't mention caching. What do you think is a good caching

> >>policy ?
> > 
> > 
> > Caching of what? Caching of pictures ? No need to do so.
> 
> IMO that's a bad design. You assume you always have enough CPU to 
> process everything. In reality that means that your system can handle 
> only the worst case (all stages take the max amount of time to process

> incoming/outgoing data). And you'll always end up working under the 
> worst case...

Yes i asume that there is enough CPU and I/O power to handle the
stream at realtime. Because otherwise i can clearly forget to 
watch a movie. If you think about preprocess the images and store
them until they are showed you go the same way as Nick Kurshev with
mplayerxp. IMHO it's a good idea and may help in cases where no
direct render is available and you are at the edge of the needed
computation power, ie you have enough cpu/io to process the stream
but at some scenes you're slightly under powered. But, you also
lose the power of direct rendering and get a lot of io by the
additional memcpy you have to do.

 
> My point here is that dropping frames and even more, dropping audio 
> samples is *no* option to me. Image someone playing live and once in a

> while there is no sound. Just because his system is fast enough for
the 
> general case but not enough for the worst case... Caching data should 
> make it possible to work even in the worst case (to prepare data in 
> advance).

For a player, dropping audio samples is a clear no-no. Even a few
dropped samples can be clearly heard. But dropping a frame once
in a while is not necessarily noticable.


			Attila Kinali


-- 
egp ist vergleichbar mit einem ikea bausatz fuer flugzeugtraeger
			-- reeler in +kaosu
_______________________________________________
Media-api mailing list
[email protected]
http://lists.matroska.org/cgi-bin/mailman/listinfo/media-api