An open, standard video/audio API?

Hello,

(SUMMARY. A general-purpose standard API for all multimedia processing,
including codecs, filters, demuxing etc., would be very useful in
GNU/Linux, and resolve many existing problems, including licensing. The
MPlayer team is the best source for a standard that could be widely
adopted. I would volunteer to do tech writing work on the spec).  

(DISCLAIMER. The following may need a sanity check; i.e. I may have lost
contact with reality. If so please tell me where).

I'm a user of mplayer, and I like it a big darn lot, so thanks people.
(I know it's not exactly the right list for this but I had to put it in
my first message to an mplayer devel list).

I'm also a big fan of Open Standards. Free Software too - but while Free
Software would totally rule in an ideal world, Open Standards totally
rule *now*, they make the Internet run. They also encourage the
development of Free Software, and minimize the effects of non-free
software.

And the one thing that I feel is missing in [GNU/]Linux, while present
(although crippled) in the Non-Free OS, is a clearly defined standard
API for all multimedia stuff.

A Free Software masterpiece, VirtualDub, is made possible by DirectShow.
A clear (hmm, theoretically) interface to all possible codecs, the AVI
file format, and some other stuff is just already there. Yes, it's
stinky (the 4GB limit, and of course the lack of freedom). 

But GNU/Linux has none whatsoever. Video4Linux2 contains some Codec API,
but who uses it? 

As a result, a GNU/Linux program that wants to use codecs, filters, etc,
has to care about interaction with every library out there. Look at
transcode, which had to actually include a copy of ffmpeg. (And no, one
can not just forget everything else and go ffmpeg only).

>From the discussion I saw here, it seems that while the MPLayer G2 core
could be a sort of "clearing center" for multimedia processing under
GNU/Linux, it will still have a pretty specific API.

My question: why not aim for a standard, generic, published,
well-described multimedia processing API, one that would (because of its
technical benefits and open nature) far surpass DirectShow?

It should work across machines (not to speak of working across very
different programs). X-Windows might provide an example. And yes, I
would support sockets as the main tool, and TCP sockets where possible.
Among other things, this will allow interoperation of different
machines, as already done in X.

Just imagine: one machine to grab the source and apply pre-procesing
filters, another to do the coding, and a third to stream the content,
all communicating over the Open Multimedia Protocol. This would be the
*easy* scenario; hard ones would include load balancing.

In most cases of course the API would work on the same machine - but
still provide nice separation for different tasks, so loved by the Unix
world. A codec, a filter, a muxer or demuxer, a player would all
communicate via socket streams. Each one easily replaceable, without
micro-management from the player (common today). 

<Use_case> A user has installed a cool hardware MPEG5 decoder. He
previously used to run MPlayer G2 for playing MPEG5 files; his brother
ran XMMS G3, and used Transcode to convert MPEG5 to Theora for watching
on his Zaurus handheld. 

Now the just replaces the primary system codec for MPEG5. Neither
MPlayer nor XMMS nor Transcode will ever notice (unless they were
configured to run a soecific codec by name, overriding system settings).
</Use_case>

Among other matters, an open socket-based standards might resolve the
recent licensing issue.

The problem with hardware player developers, as it seems to be, is that
they just can't release all of the source to their firmware, because of
licensing matters with other companies. 

But what if the components they use will all interact via a standard
socket interface? In this way, as the GPL FAQ exlicitly says, they can
use GPL-ed and non-GPL modules together!

So they can use some modules from MPlayer, AND be forced to release any
changes to these modules, AND be able to pay MPlayer developers for
enhancing them.

But if their proprietary modules use the standard protocol, they'll be
able to keep them - and honor any licensing requirement with any third
company.

It's stricter than the LGPL. They will be forced to adhere to the
protocol and use sockets, rather than access system calls in a
particular library. In fact, any DRM they install will be easy to
circumvent - the output will go through the standard stream protocol, so
go intercept it, and whoops! we're done.

If this standard is adopted widely enough, the DRM-mongers will either
have to stop using all newer versions of Linux video tools altogether,
or see their data intercepted and their DRM broken by the nearest hacker
who knows how to read a manual for a standard. 

(They still have the option of preventing physical access inside the
hardware, so that one will be unable to do this interception; but one
can't fix that with software licenses. That would work with 100% GPL
too, just keep the key values [data not code!] in a chip inside).

MPlayer is widely accepted as the best multimedia player under
GNU/Linux; ffmpeg/libavcodec, as the best codec library; MEncoder and
transcode are the two commonly used processing solutions. Therefore, I
think the MPlayer team would be the best people for developing and
promoting this standard.

Now for a bit of personal stuff. I am a tech writer with over 5 years of
experience behind me. English is not my native language, but I am
employed as a tech writer in English by a major international company.
For money, I write development docs for internal-use software (it's not
distributed outside of the company) - and they like the work I do.

I would be willing to take up the writing of the spec. Not the technical
side of the development - I don't know *nearly* enough to do that. But I
can glean information from programmer-speak, list discussions, etc., and
convert it into an organized, readable document. And that's what an open
standard spec should be, if it's to be widely adopted. (Ultimately, why
not make it an RFC?) 

But of course, this is useful only if the developers here are willing to
go that way. To develop an open standard for interaction between
multimedia processing components. And to base the development of MPlayer
G2 on this standard, not an internal API. 

This could be a crucial step for development of GNU/Linux as a user OS.
This may be an insane raving by a misguided guy. ANyway, replies will be
very much appreciated :)

Yours, Mikhail Ramendik
Moscow, Russia