Re: [gst-devel] Re: Common Opensource codec API

On Tue, 30 Dec 2003, Artem Baguinski wrote:

>not sure how vloopback's relevant to the discussion [my explanation aims
>to show its irrelevance ;)]
...
>may be the solution though is another layer, some video access library,
>that would provide uniform access to various video inputs and having OS
>dependant backends. e.g. OpenML (http://www.khronos.org/openml/) API
...
>again, a nice issue to discuss but i dont see how it relates to the
>Subject...

I try to enlighten you. Everything that happens to video/audio/multimedia
can be divided into three parts:

- Source
- A pipeline of format conversions
- Sink

For mplayer, the source is usually a file on disk or network. Then it does
format conversions which includes also codecs (e.g. mpeg decoder just takes
frames formatted as variable length packets and converts those into YUV)
and then maybe does some more processing to the frames, like deringing
(which can be considered another kind of format conversion), then maybe
converts YUV into RGB, and finally display in a window, which is the sink.

What if user wants to watch TV with mplayer?

The source is V4L device (e.g. /dev/video0) which delivers some specific
format, say YUV, and then the format converters convert YUV to RGB (assume
that the display takes only RGB) and displays or "sinks" that.

What if the TV driver delivers frames in MJPEG compressed frames from
/dev/video0? The same happens, except this time the pipeline has to convert
from MJPEG to RGB (possibly using multiple codec/format conversion
plugins).

What if the TV driver supports only frames in foo-format where foo is some
format specific to the device? Ideally mplayer would have a plugin (i.e.
codec or decoder) for the foo format, and everything would go smoothly.

Unfortunately, the TV driver author doesn't bother about mplayer, so he
doesn't do a plugin. Why should he anyway, it would add support only a
single program? A saner approach is to write a driver specific
userspace program that reads the foo-format from /dev/video0, converts to
YUV or RGB or some other common format, and then outputs that into
/dev/video1 using vloopback.

What I mean is that vloopback could in principle replace codecs in mplayer.
Have some new weird codec in AVIs? Why bother adding support only to
mplayer, when one could write a program that decodes frames from the AVI
and outputs to all other programs from /dev/video1 using vloopback?

This would work with every program, since V4L API is quite well supported.
In fact, V4L with vloopback could be considered a kind of plugin standard
in Linux. But the problem, probably, would be the low efficiency and
flexibility.

So no, I don't think vloopback is a good long term solution, although it
has one good thing: compatibility with most programs.

Kind of another extreme is mplayer, its plugins are efficient but they are
bound to a single program. Except that, of cource, libavcodec is not just
for a single program. But I'm still a bit suspicious whether it could
provide the necessary format conversions. Are dynamically loadable plugins
possible? Can the formats be well specified, i.e. if an AVI contains frames
in unknown format to mplayer, can mplayer be made aware of the format by
just writing libavcodec plugin or does it require modification of mplayer
too?

On the other hand, if implementing good codec/format conversion plugin
system, why not enhance the API slightly to allow also codecs with no
input, just output, and codecs with only input, no outputs--or source and
sinks. Enter Gstreamer.

>> Yeah, and Gstreamer. Probably others too. Multiplicity is the problem.
>Gstreamer isn't a codec api, it's a glue layer implented in C.

Glue layer? Gstreamer is an API for multimedia pipelines consisting of
source, format conversions and sink. And certainly format conversions
include codecs too.

>i guess before trying to invent it people should look at OpenML i

Hmm, good idea. Thanks for this, i didn't know about it.