Re: more on frame timing, framerate-changing filters

On Sun, Jun 15, 2003 at 12:46:51AM +0200, Arpi wrote:
> Hi,
> 
> > Anyway, to get to the point, I'm hoping to make this code friendly for
> > porting to G2 (I already have the timing-change stuff worked out for
> > it), but I'd like to clear up how we want timing to work. Right now,
> > it seems G2 has a duration field for the image, but I'm not sure
> > whether that's preferred, or whether we should instead have a
> > timestamp relative to the previous frame... IMO, for some filters, we
> > may need to know *future* information to decide a duration, whereas we
> > should already know the pts relative to prev frame. And of course, for
> > performance purposes (and coding simplicity), it's optimal not to have
> > to grab future frames from up the chain before we output a given
> 
> Yes.
> Actually I spent a lot of time thinking on this, and got to the conclusion
> that no optimal (or near optimal) solution exists.
> At least in mplayer world, where lots of containers and codecs with
> different behaviour and timing models are supported.
> 
> I've reduced this game to 2 basic types:
> - absolute timestamps (when to display the frame)
> - frame durations (how long display the frame)

Yes, and I'm asking if we can change it to absolute timestamp and
relative timestamp...

> > frame. Consider for example the case of vf_decimate (near-duplicate
> > frame dropper). If it wants to output duration, it has to grab frames
> > into the future until it finds a non-duplicate. But with pts relative
> > to previous, it can defer the task of decoding those extra frames
> > until the current frame has been displayed, and *then* decoding those
> > extra frames takes place during the "sleep time" anyway, so it won't
> > cause the a/v sync to stutter on slow systems.
> 
> Actually you should (have to) report dropped frames too, by returning NULL.
> It ensures that next filters know about frames were dropped (some temporal
> filters or for example field<->frame splitters/merge filters require this),
> and the final a-v sync code also know about frame being dropped.

I disagree. You're right if the filter is just doing something silly
like blindly dropping frames to lower output framerate. But if your
filter is actually adding or dropping frames for a good reason
(inverse telecine, duplicate removal, etc.), then temporal (noise?)
filters and field split/merge stuff should have no problem getting the
output stream as-is with no information about what was dropped. In
fact, I don't even like thinking about the chain in terms of "drops"
or "added frames", since from my point of view, the whole idea is that
there is no inherent 1-1 correspondence between input frames and
output frames.

As for A/V sync being preserved, this does not require notifying the
next filter about drops either. You just have to merge the duration of
the frame you dropped in with the durations of the other frames you're
sending out so that the sum duration is not changed.

> Imho the vf_decimate should run one frame before the playback pointer,
> so it always return the previous frame if different enough, or return NULL
> if similar enough.

Hmm, I'd have to think about it more, but I'm not sure this can be
done right without shifting a/v sync by at least one frame...which
would be bad.

> Filters altering playback rate should modify the duration of incoming
> frames, and reset timestamp of generated new (==inserted) frames.
> See the tfields port for example.

Let me explain how I'd prefer to do it. I'd like to have relative_pts
rather than duration, which is exactly the same thing as duration of
the previous frame. Then, what tfields would do is output both fields
with relative_pts of input.relative_pts/2. Here, the exact same thing
could be done with duration as-is.

> And, yes, i know it's not the optimal solution, so i'm open to better
> models, although i know there is no better way (within the given
> constraints)

>From the perspective of implementing the player loop, relative_pts and
duration are pretty much exactly the same. However, as I was saying in
my original mail, relative_pts is much better from the filters'
perspective since you can know it earlier.

Also relative_pts seems to be a more natural quantity to work with
when thinking of output frames as the basic unit rather than input
frames. For instance, smoothing out frame times with inverse telecine
requires adding 1/120 to pts for frames that are shown for 3 fields,
and subtracting 1/120 for frames that are shown only 2 fields. To
emulate this with durations, you'd need to know whether the *next*
frame is shown for 3 fields or just 2, which requires buffering up
even more -- bleh. (You can't just adjust durations of the current
frame based on whether it's shown 2 fields or 3. That will work with
perfect telecine, but in the wild perfect telecine does not exist, so
in all practical settings it will break a/v sync!!) Keep in mind that
I'm already having to buffer at least 6 fields for the new smart
algorithm, so if I'm limited to duration rather than relative_pts,
I'll also have to buffer an additional frame...

> Kabi once doen some calculations on ffmpeg-devel when this topic was
> discussed there (about the ticker code) and with double it's accurate
> enough to run over several thousands of hours without a single frame delay.

OK, I guess it can be ignored then...

> > would be much perferred for precision encoding and video processing
> > work, and nice for output to containers like nut (mpcf) which will
> > support such things. I'm envisioning my inverse telecine wanting to
> > use a time base of 1/120 second, for flawless output of mixed
> > telecine, 30fps progressive, and 60fps interlaced content as a single
> 
> you shouldn't rely on assuming any fps, think of if anyone is doing -speed
> 1.356 and then use teh telecine filter. you should use only rates, not
> absolute values. ie for inv. telecine multiply incoming duration by 4/5...

Obviously I didn't mean having numbers like this hard-coded... :) Just
one possible setup for the output muxer, which a user might configure.

And anyway you mean 5/4, not 4/5, but as I descrived above that will
NOT work for real-world content, just ideal 3:2 telecine.


Rich