Re: more on frame timing, framerate-changing filters

Hi,

> > I've reduced this game to 2 basic types:
> > - absolute timestamps (when to display the frame)
> > - frame durations (how long display the frame)
> 
> Yes, and I'm asking if we can change it to absolute timestamp and
> relative timestamp...

why?

btw note that duration != relative timestamp

   f(x-1)             f(x)              f(x+1)
     |   f(x).rel_pts  |     f(x).dur      |
     |<--------------->|<----------------->|
     ^                 ^                   ^
  f(x-1).pts         f(x).pts           f(x+1).pts


> > > frame. Consider for example the case of vf_decimate (near-duplicate
> > > frame dropper). If it wants to output duration, it has to grab frames
> > > into the future until it finds a non-duplicate. But with pts relative
> > > to previous, it can defer the task of decoding those extra frames
> > > until the current frame has been displayed, and *then* decoding those
> > > extra frames takes place during the "sleep time" anyway, so it won't
> > > cause the a/v sync to stutter on slow systems.
> > 
> > Actually you should (have to) report dropped frames too, by returning NULL.
> > It ensures that next filters know about frames were dropped (some temporal
> > filters or for example field<->frame splitters/merge filters require this),
> > and the final a-v sync code also know about frame being dropped.
> 
> I disagree. You're right if the filter is just doing something silly
> like blindly dropping frames to lower output framerate. But if your
> filter is actually adding or dropping frames for a good reason
> (inverse telecine, duplicate removal, etc.), then temporal (noise?)
> filters and field split/merge stuff should have no problem getting the
> output stream as-is with no information about what was dropped. In
> fact, I don't even like thinking about the chain in terms of "drops"
> or "added frames", since from my point of view, the whole idea is that
> there is no inherent 1-1 correspondence between input frames and
> output frames.

it depends.
i wanted to keep the option of seeing dropped frames in filters.
it isn't used yet, but i see several ideas for it.
for example, think of a filter merging fields to frame.
for example you want to split decoded frame to fields, apply
similar-frame-remover filter (on fields) and then merge fields.
the merge filter should know about the number of fields being dropped.
(ok i know it's not a good example, tell a better one).
but i can imagine other cases too, when it's useful to a filter to know
about dropped frames. and since it's near zero work to implement
(it's already done in core, see VFCAP_NOTIFY_DROPPED_FRAMES) why to drop it?

> As for A/V sync being preserved, this does not require notifying the
> next filter about drops either. You just have to merge the duration of
> the frame you dropped in with the durations of the other frames you're
> sending out so that the sum duration is not changed.

this is good idea

> > Imho the vf_decimate should run one frame before the playback pointer,
> > so it always return the previous frame if different enough, or return NULL
> > if similar enough.
> 
> Hmm, I'd have to think about it more, but I'm not sure this can be
> done right without shifting a/v sync by at least one frame...which
> would be bad.

why?

> > Filters altering playback rate should modify the duration of incoming
> > frames, and reset timestamp of generated new (==inserted) frames.
> > See the tfields port for example.
> 
> Let me explain how I'd prefer to do it. I'd like to have relative_pts
> rather than duration, which is exactly the same thing as duration of
> the previous frame. Then, what tfields would do is output both fields
> with relative_pts of input.relative_pts/2. Here, the exact same thing
> could be done with duration as-is.

the problem is that duration of previous frame is not (always) available
when decoding a frame. so again it needs extra complexity in decoders, to
keep it in their private area or so.

> > And, yes, i know it's not the optimal solution, so i'm open to better
> > models, although i know there is no better way (within the given
> > constraints)
> 
> >From the perspective of implementing the player loop, relative_pts and
> duration are pretty much exactly the same. However, as I was saying in
> my original mail, relative_pts is much better from the filters'
> perspective since you can know it earlier.
> 
> Also relative_pts seems to be a more natural quantity to work with
> when thinking of output frames as the basic unit rather than input
> frames. For instance, smoothing out frame times with inverse telecine
> requires adding 1/120 to pts for frames that are shown for 3 fields,
> and subtracting 1/120 for frames that are shown only 2 fields. To
> emulate this with durations, you'd need to know whether the *next*
> frame is shown for 3 fields or just 2, which requires buffering up
> even more -- bleh. (You can't just adjust durations of the current
> frame based on whether it's shown 2 fields or 3. That will work with
> perfect telecine, but in the wild perfect telecine does not exist, so
> in all practical settings it will break a/v sync!!) Keep in mind that
> I'm already having to buffer at least 6 fields for the new smart
> algorithm, so if I'm limited to duration rather than relative_pts,
> I'll also have to buffer an additional frame...
> 
> > Kabi once doen some calculations on ffmpeg-devel when this topic was
> > discussed there (about the ticker code) and with double it's accurate
> > enough to run over several thousands of hours without a single frame delay.
> 
> OK, I guess it can be ignored then...
> 
> > > would be much perferred for precision encoding and video processing
> > > work, and nice for output to containers like nut (mpcf) which will
> > > support such things. I'm envisioning my inverse telecine wanting to
> > > use a time base of 1/120 second, for flawless output of mixed
> > > telecine, 30fps progressive, and 60fps interlaced content as a single
> > 
> > you shouldn't rely on assuming any fps, think of if anyone is doing -speed
> > 1.356 and then use teh telecine filter. you should use only rates, not
> > absolute values. ie for inv. telecine multiply incoming duration by 4/5...
> 
> Obviously I didn't mean having numbers like this hard-coded... :) Just
> one possible setup for the output muxer, which a user might configure.

ah ok

> And anyway you mean 5/4, not 4/5, but as I descrived above that will
> NOT work for real-world content, just ideal 3:2 telecine.

:)


A'rpi / Astral & ESP-team

--
Developer of MPlayer G2, the Movie Framework for all - http://www.MPlayerHQ.hu