buffer management and libvo2...

Hi again. It's almost time for more vp coding (actually I've already
done a little), but I'm running into a problem interfacing with
libvo2.

With the old mpi system, the destination vf/vo you were direct
rendering into knew exactly how many buffers of each type it would
have to provide. This was based on assumptions about standard IPB
codec behavior, which we want to do away with in G2 to allow direct
rendering by more advanced codecs with multiple reference frames and
such. Even in the absence of such codecs, it's very useful for a
filter to be able to keep multiple reference frames for temporal
filtering without copying. Thus, I really don't want to sacrifice this
flexibility.

So, what's the problem?

Suppose your vo module can only provide 3 buffers. Suppose all three
are already in use (either displayed, if in video memory, or still
locked by the filter/codec). Then the filter chain gives you an
export-type or automatic-type (allocated) image for the vo to display.
Where does it go??

Before you object that filters should not grab many dr buffers at a
time, let's look at a few examples...

1. vf_tfields. Let's say we've optimized tfields so that it does slice
   rendering. Thus, it grabs 2 dr buffers from the vo, and whenever it
   gets a slice, it draws the deinterlaced top field into the first
   buffer, and the deinterlaced bottom field into the second buffer.
   Think 2 buffers is no problem? Now consider what happens when the
   source codec doing the slice-rendering through tfields uses B
   frames (i.e. out of order rendering)... :)))

2. Codecs with multiple reference frames (h264, vp3, theora, ...). The
   codec might request and lock two readable buffers and also request
   another buffer for B-type (non-reference) frames.

3. vf_pullup. Right now it allocates its own buffers for deep
   buffering, but it would improve performance to use dr buffers from
   the vo when possible. It requires at least 3 buffers, and probably
   4 or more.

Cases 2 and 3 apply when the vo's buffers are readable, which
_probably_ means they're in system memory and the vo can allocate as
many as it wants. But I don't think it's good to rely on this
assumption. Case 1 is definitely a problem, since it occurs with vo's
that are letting you write directly into video memory (and therefore
which have a very limited number of buffers!).

Let's see how case 1 can actually lead to a deadlock:

A. Codec is decoding a sequence IBP. The I frame was already finished.
   Now it's time to decode the P and then the B.
B. Codec requests indirect buffer (slices) from tfields (for P frame).
C. tfields gets 2 dr buffers from the vo to render slices into.
D. ... P frame gets rendered ...
E. Codec requests indirect buffer (for B frame).
F. tfields tries to get dr buffers, but all buffers are exhausted, so
   it has to fall back to using auto (allocated) buffers.
G. tfields returns the first field extracted from the B frame, which
   happens to be in an automatic buffer.
H. vo tries to display the image, but all buffers are in use!!!
I. Deadlock!

What can be done about it?

Solution 1. Never allow dr to the last buffer.
Pros: Deadlock impossible. Simple.
Cons: Sometimes we can't know how many buffers are available. Even if
we can, refusing to use one buffer may prevent DR entirely!!

Solution 2. When we run into deadlock, shuffle buffers. I.e. allocate
new buffer in system memory and move the contents of one dr buffer
there to free up space.
Pros: Deadlock impossible. All buffers can be used.
Cons: Very slow if dr buffers are in video memory! Also requires
horrible hacks to change a dr-type mpi into an auto-type one.

Solution 3. Require codecs/filters to report how many buffers they
will need at config-time.
Pros: Deadlock impossible. All buffers can be used. No ugly hacks.
Cons: Very confusing! Requires considerably more logic in filters. For
example, in case 1 above, tfields has to figure out it will need 4
buffers based on the info the codec gives it. How to do this?!

Solution 4. ???

Any better ideas? I don't like any of my 3 proposals at all!

BTW, vo_mga should definitely be updated to allow the full max of 4
buffers rather than just 3!

Rich