Re: RFC: Embedded documentation

Hi Nicolas

On Sun, Aug 16, 2026 at 09:25:50PM +0200, Nicolas George via ffmpeg-devel wrote:
> Hi.
> 
> Michael Niedermayer (HE12026-08-14):
> > yes
> 
> :)
> 
> > I consider the general decision to have embedded documentation separate
> > from the review of the particular APIs needed to implement it.
> 
> Of course. But at least the question of whether it “belongs in ffmpeg”
> can be answered: we decided we want embedded docs, it is necessary for
> it, propose something else that fits the bill or it belongs in ffmpeg.

something like it, is needed. Yes.


> 
> > This is an ambitious project; it is not simple.
> 
> I am aware of it, but I see a path to achieving it incrementally, only
> adding change that either can be rolled back or is in itself an
> improvement.

Yes, incrementally implementing things like this is a very good idea.


> 
> > For example, localization becomes more important once applications use
> > documentation obtained directly from the libraries.
> 
> That is true, and it is an excellent question to raise.
> 
> (Side note: I hope we agree that translating the diagnostic messages is
> a can of worms we do not want to open. Bug reports with messages in
> Swedish, <shudder>.)
> 
> I think handling translated documentation is much simpler than handling
> translated diagnostic messages, because we do not have to worry about
> variants (singular/plural) or placeholders and format specifier changing
> order in the sentence.

The parts an application would want to show in its user interface like
buttons, would need to be translated.


> 
> That leaves only the issue of loading the documentation in another
> language, which would be quite easy as long as the system is capable of
> loading the documentation from a file, see below.

> 
> One thing that would help for the translation of documentation is to
> make sure its source code clearly distinguish the parts that are in
> English from the parts that are code and must absolutely not be
> translated. And to make sure the HTML output has “lang="en"”. And
> “lang="x-code" translate="no"” everywhere that is relevant so that
> applications can sic a LLM on it to translate on the fly.
> 

> But let it be on record that translations are the worst case of
> duplicated documentation that inevitably gets out of sync.

In the age of AI, updating translations should be almost as easy as building
object from source.


> 
> > Also, where and how should the data be stored? Our texi documentation is
> > currently around 2-3 MB, or roughly 500 kB compressed, although one could
> > argue that compression does not belong at this layer.
> 
> One interesting feature of ffmpeg is that the default build is almost
> entirely self-contained: “./configure && make” will give a binary that
> works by itself, can be copied on any similar system and still work. It
> does not require files to be installed at specific places.
> 
> The price for that is that our binaries are big, 30M.
> 

> I think the default should be to have the documentation in the binaries,
> or in the shared objects if --enable-shared. And if people want to save
> a little space, --disable-embedded-doc or --enable-small.

yes, i agree


> 
> And 30M binaries mean the documentation is only +10%.
> 
> Let me remind everybody here that if the documentation is in the binary
> or shared objects, it is memory-mapped by the operating system, only the
> parts that are used are loaded into memory, they are shared between all
> instances running at the same time and can be unloaded in case memory is
> necessary.
> 
> We can later implement loading (or rather mmap()ing) from an external
> file, but that requires hard-coded paths that can be overridden with
> environment variables, global mutable state, etc. But that would be
> useful for translations, as said above.
> 
> >			   or roughly 500 kB compressed, although one could
> > argue that compression does not belong at this layer.
> 

> Compression would lose us all the nice benefits of mmap()ing, unless it
> happens between mmap() and the filesystem.

I dont think so actually
If you keep the documentation in its compressed form and only decompress
when something actually needs a string (and dont cache it or store it)
then all the compressed data can
be shared is read only, static and identical between memory and file.
And we can do that because decompression is fast and documentation
retrieval is not speed critical.
What we would need is a fast decompression for small pieces of documentation.
THere are bascially
2 ways, either, you have a fixed global dictionary so each part of the
documentation is a list of indexes into that dictionary OR you have a
statistical model that provides the probability of the next symbol
based on the k prior. Both these would be non adaptive and efficient
for small pieces of text

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Take away the freedom of one citizen and you will be jailed, take away
the freedom of all citizens and you will be congratulated by your peers
in Parliament.

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]

-----BEGIN PGP SIGNATURE-----

iF0EABEKAB0WIQSf8hKLFH72cwut8TNhHseHBAsPqwUCaoNFLQAKCRBhHseHBAsP
q+tzAJ9dVOQufsr3pDyr64tLw78vWgbmsgCfT48eUxCwuM5LmEFfhazilatxofs=
=jLOH
-----END PGP SIGNATURE-----