Re: RFC: Embedded documentation

Hi.

Michael Niedermayer (HE12026-08-14):
> yes

:)

> I consider the general decision to have embedded documentation separate
> from the review of the particular APIs needed to implement it.

Of course. But at least the question of whether it “belongs in ffmpeg”
can be answered: we decided we want embedded docs, it is necessary for
it, propose something else that fits the bill or it belongs in ffmpeg.

> This is an ambitious project; it is not simple.

I am aware of it, but I see a path to achieving it incrementally, only
adding change that either can be rolled back or is in itself an
improvement.

> For example, localization becomes more important once applications use
> documentation obtained directly from the libraries.

That is true, and it is an excellent question to raise.

(Side note: I hope we agree that translating the diagnostic messages is
a can of worms we do not want to open. Bug reports with messages in
Swedish, <shudder>.)

I think handling translated documentation is much simpler than handling
translated diagnostic messages, because we do not have to worry about
variants (singular/plural) or placeholders and format specifier changing
order in the sentence.

That leaves only the issue of loading the documentation in another
language, which would be quite easy as long as the system is capable of
loading the documentation from a file, see below.

One thing that would help for the translation of documentation is to
make sure its source code clearly distinguish the parts that are in
English from the parts that are code and must absolutely not be
translated. And to make sure the HTML output has “lang="en"”. And
“lang="x-code" translate="no"” everywhere that is relevant so that
applications can sic a LLM on it to translate on the fly.

But let it be on record that translations are the worst case of
duplicated documentation that inevitably gets out of sync.

> Also, where and how should the data be stored? Our texi documentation is
> currently around 2-3 MB, or roughly 500 kB compressed, although one could
> argue that compression does not belong at this layer.

One interesting feature of ffmpeg is that the default build is almost
entirely self-contained: “./configure && make” will give a binary that
works by itself, can be copied on any similar system and still work. It
does not require files to be installed at specific places.

The price for that is that our binaries are big, 30M.

I think the default should be to have the documentation in the binaries,
or in the shared objects if --enable-shared. And if people want to save
a little space, --disable-embedded-doc or --enable-small.

And 30M binaries mean the documentation is only +10%.

Let me remind everybody here that if the documentation is in the binary
or shared objects, it is memory-mapped by the operating system, only the
parts that are used are loaded into memory, they are shared between all
instances running at the same time and can be unloaded in case memory is
necessary.

We can later implement loading (or rather mmap()ing) from an external
file, but that requires hard-coded paths that can be overridden with
environment variables, global mutable state, etc. But that would be
useful for translations, as said above.

>			   or roughly 500 kB compressed, although one could
> argue that compression does not belong at this layer.

Compression would lose us all the nice benefits of mmap()ing, unless it
happens between mmap() and the filesystem.

> I think some important goals are:
> 
> * It should be easy to maintain. Once implemented, it should not add a
>   significant maintenance burden compared to today.

Once it works, I believe the API itself should require very little
maintenance: it is not connected to protocols that evolve or anything
like that, and it does not interact with untrusted data at all.

As for the documentation itself, I believe the effort to clean it up to
make it possible to implement the project would have the side effect to
make it slightly easier to maintain.

> * Documentation should not be duplicated, since duplicated documentation
>   will inevitably drift.

Absolutely. I intend to use the existing documentation as is, or, more
probably, after a once-and-for-all semi-automatic conversion in a more
convenient format (I am thinking a dialect of Markdown with one file per
C source file that requires documentation). No duplication at all.

It could even reduce duplication ever so slightly, because it would
allow to remove the .descr field of components and options.

> Code and API details will of course be subject to the usual review.

Of course.

> However, objections to the project in principle need to be raised now.
> It would not be reasonable for Nicolas to spend substantial time
> implementing this and then have the result rejected because embedded
> documentation is considered not to belong in FFmpeg.

Thanks, that is all I am asking.

In a way, it is a little like if we were negotiating to commission me to
do it for a fee: it would not be acceptable, once the work is done and
meets the requirements, to say “sorry, all things considered we do not
want it” and not pay. Except I do not want money for it.

> that said, iam in favor of this, if nicolas wants to work on this!

Let us see where the discussion goes before talking technical details
further.

Thanks.

Regards,

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]