Re: Re: [gst-devel] Re: Common Opensource codec API

Tuukka Toivonen wrote:
> I try to enlighten you. Everything that happens to video/audio/multimedia
> can be divided into three parts:
> 
> - Source
> - A pipeline of format conversions
> - Sink
> 
> For mplayer, the source is usually a file on disk or network. Then it does
> format conversions which includes also codecs (e.g. mpeg decoder just takes
> frames formatted as variable length packets and converts those into YUV)
> and then maybe does some more processing to the frames, like deringing
> (which can be considered another kind of format conversion), then maybe
> converts YUV into RGB, and finally display in a window, which is the sink.
> 
> What if user wants to watch TV with mplayer?
> 
> The source is V4L device (e.g. /dev/video0) which delivers some specific
> format, say YUV, and then the format converters convert YUV to RGB (assume
> that the display takes only RGB) and displays or "sinks" that.
> 
> What if the TV driver delivers frames in MJPEG compressed frames from
> /dev/video0? The same happens, except this time the pipeline has to convert
> from MJPEG to RGB (possibly using multiple codec/format conversion
> plugins).
> 
> What if the TV driver supports only frames in foo-format where foo is some
> format specific to the device? Ideally mplayer would have a plugin (i.e.
> codec or decoder) for the foo format, and everything would go smoothly.
> 
> Unfortunately, the TV driver author doesn't bother about mplayer, so he
> doesn't do a plugin. Why should he anyway, it would add support only a
> single program? A saner approach is to write a driver specific
> userspace program that reads the foo-format from /dev/video0, converts to
> YUV or RGB or some other common format, and then outputs that into
> /dev/video1 using vloopback.
> 
> What I mean is that vloopback could in principle replace codecs in mplayer.
> Have some new weird codec in AVIs? Why bother adding support only to
> mplayer, when one could write a program that decodes frames from the AVI
> and outputs to all other programs from /dev/video1 using vloopback?
> 
> This would work with every program, since V4L API is quite well supported.
> In fact, V4L with vloopback could be considered a kind of plugin standard
> in Linux. But the problem, probably, would be the low efficiency and
> flexibility.
> 
> So no, I don't think vloopback is a good long term solution, although it
> has one good thing: compatibility with most programs.
> 
> Kind of another extreme is mplayer, its plugins are efficient but they are
> bound to a single program. Except that, of cource, libavcodec is not just
> for a single program. But I'm still a bit suspicious whether it could
> provide the necessary format conversions. Are dynamically loadable plugins
> possible? Can the formats be well specified, i.e. if an AVI contains frames
> in unknown format to mplayer, can mplayer be made aware of the format by
> just writing libavcodec plugin or does it require modification of mplayer
> too?

Mmm, I'm surprised that DirectShow hasn't been mentioned yet. Because it 
does many things, is quite extensible (even though we faced some 
limitations with Matroska), and is used by many different programs for 
different purposes. The only problem is that it's Microsoft specific and 
based on DCOM... IMO if we take a lot of stuff from DirectShow and get 
around the limitations we have a much better uniform API. If it is close 
enough to DirectShow all Windows application could move to that easily !