Re: Gstreamer [WAS: Variable Framerate, plugin based video editing tool]

* Steve Lhomme <[email protected]> [2004-01-30 16:06:22 +0100]:

<snip>
> One issue I'm thinking about with a powerful/flexible media API is the 
> way it can handle OGG + Vorbis. Because the timecode can only be known 
> by the codec, not the container... That implies a rather tricky design.
hmm, it seems there are also containers which contain timestamps 
(matroska ?). So we should be able to handle both. We could say the
container may contain a quite crude ts, which helps in seeking over
long distances, but the exact (micro-)seeking and syncing has to go
over the decoder's timestamps. 

Since we may have longer chains where ts somewhere appeares and later
doesnt change, we could supply it the two kind of ts with each chunk
optionally (zero=not specified). Think of image filters, which operate
on some kind of pixmaps - they dont touch the ts, but others may do so
(i.e. changing the speed). So we require a way for asking the codec/filter
if its gonna change the speed or supplies or perhaps requires timestamps.

At least some output devices (i.e. display+speaker) will require timestamps,
so if they're not supplied up to this point in the chain, we somehow
have to "invent" them ... (dont know when this could happen)

<snip>
> >* muxers are implemented as filters (how shoud this work ?!)
> >  this brings us huge trouble! muxers brigde between frame-sequences
> >  and binary streams. we need a _clear_ decision between them, i.e.
> >  frames have timestamps/time intervals, binary streams are simply bytes
> 
> Well, this is the case in DirectShow too. What is the problem with that ?
It requires complex handling on many points. 

For example if we have a frame-based filter: it has to cope w/ splitting
of the input stream into frames and and merge the output frames into 
a output stream. This adds additionally complexity. Of course we could use
generic code for that, but each encapsulation requires time.

So for the case you're using the media-api for something like gstreamer
(intuitive assembling of a filter/processor chain), you can easily 
encapsulate the primitive objects (muxes, demuxes, codecs, filters, io) 
into proxy-objects, which do the necessary adaption automatically. But 
this requires computing time, which we do not want in projects where we
do not necessarily need the high level gstreamer model, but instead
require very good performance.

> I haven't checked GStreamer that far yet. One of the major problems of 
> GStreamer is that you have to compile the documentation :/
The documentation can be viewed on the website, but I really dont know
how up-to-date this copy is.

<snip>
> >* the pads could be left off, if we would divide all devices into 
> >  strict groups. For example a codec always converts _one_ stream
> >  from A to B - so it always has _one_ input and _one_ output.
> 
> I can think of examples with either multiple input or multiple outputs...
Okay, there are a few. For those we could invent a diffent kind of objects. 
If you really need a model which copes with all of them in a uniform way,
then put something like gstreamer on top of the small media-api.

<snip>
> >* gstreamer relies on glib - I do _not_ want to use it, since its
> >  bloated (in times of 1.x it was good), and many other projects 
> >  also dont want it. these days glib is good for GUI-centric 
> >  applications (together w/ gtk), but not for fundamental codec APIs.
> 
> Well, as long as it runs on many OSes I don't have any problem with it. 
> As long as it doesn't do things that are already done (better) in the OS.
glib is too big and requires too many resources (i.e. time and memory).
if it only would be a small library which does primitive memory management
(w/ garbage collection or refcounting) and some module handling (such as I
already proposed here in this list), coupled w/ a small devkit which 
takes most of the compilation process off the hands of the developer, 
it would be very nice. But glib is too big and doesnt solve the necessary
things clean enought.
 
> >>From start of this project, we were talking about a _CODEC_ API,
> >_NOT_ an allround a/v-processing-library.
> 
> Mmm, no. My real-time DJing software is a example of things I want to 
> do. It's not just about codecs and containers.
Okay, then you maybe require a higher level API, but this one should 
sit on top of the plain media-api. But the media-api itself should be
quite low-level and provide a strictly separated set of interfaces.
Evryone's free to build some abstraction layer on top of it, to have
high-level functionality like gstreamer.
 
<snip>
> >Gstreamer mixers things together, which are completely different
> >(i.e. codecs, io-streams, muxers, filters), so it implies a very
> >generic api for all this kind of objects. This is nice for intuitive
> >design of filter chains, but adds unnecessary overhead in things
> >like realtime applications.
> 
> Maybe GStreamer is not suited for real-time/low-latency. But that's 
> another problem. And I really want to know if that's the case or not.
The problem of gstreamer is, that it should solve evrything what could
be interesting from ground up. In German we've got a methaphoric symbol for 
that: it's called "eierlegende Wollmilchsau" - its a fable animal, which
produces all we need from animals.

Such things - if they really get working - require quite much resources
also for quite simple tasks. So better make it as simple as possible 
and put more abstract layers around primitive ones.

<snip>
> >So Gstreamer may be seen as a superset of our media-api, but I wan 
> >to keep the media-api small and performant, otherwise other projects
> >won't use it.
> 
> Well, maybe the media-api is name is not good. At least it's not what I 
> have in mind. It should handle more than just muxing/demuxing. And 
> GStreamer/DirectShow is the kind of thing I have in mind.

If our media-api should be adopted by many related projects, we have to
keep it simple and lightweight. This way the API falls into several parts:

* file/stream IO (VFS?)
* demuxers (as well file-based as network-based ones or capture devices)
* codecs (chunk-based, transform from one encoding to another)
* frame/sample-based filters (may eat up multiple frames, ie. 
  overlay, morph and may influence time flow, i.e. speed regulators)
* frame/sample-based output devices

These different kind of objects cannot be connected directly, therefore
we can place adaptors at those places, where they're really needed.

Nevertheless, at this API level the data control should lay completely
in the hand of the application or some higher API level.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT services

  phone:     +49 36207 519931         www:       http://www.metux.de/
  fax:       +49 36207 519932         email:     [email protected]
  cellphone: +49 174 7066481
---------------------------------------------------------------------
 DSL-Zugang ab 0 Euro. -- statische IP -- DSL-Flatfrate            --
 Diese Mail wurde mit UUCP versandt.      http://www.metux.de/uucp/
---------------------------------------------------------------------