Re: Analogue artifact estimation

> > I originally posted these ideas to the Theora developers list, but 
> they suggested that this list might be more appropriate, (I thought 
> that the Tarkin project had died off :-) ), so here goes:
> 
> Well, it depends on your definition of "dead" :-). We've had longer 
> periods of inactivity before, and then someone mails something and 
> it flares up again. Or not. :-).

Oh, right, I definitely mis-understood then, I originally thought that the idea was to shoe-horn as many existing ideas in to the new codec as possible.  Glad that's not the case!  So, Tarkin is really meant to be the next-but-one codec, where as Theora is a solution for today's applications, I get it now.

> > > Just batting a few ideas around here, but would it be possible
> > > to include in to the codec, estimation for common video
> > > artifacts that occur in the analogue world?
> 
> That's interesting. If we can recognise them we could probably even 
> compensate for them as well. Ofcourse, for certain artifacts the 
> original information is simply missing so the best we can do is try 
> and synthesise something.

Interesting...  I was just thinking of recognising them to remove difficult-to-compress data from the datastream, but an option to fix the problem is even better.

> > > For example, anything that's gone through a composite stage
> > > will likely have dot-crawl and false colour - if we can
> > > recognise this effect in the encoder, we can treat it as a
> > > special case.
> > >
> > > Other artifacts that come to mind are:
> > >
> > > * 3-2 Pull down
> > >
> > > * 30/25 and 25/30 FPS conversion artifacts
> > >
> > > * Dropouts, (both noisy, a-la VHS, and solid bars, a-la BetaSP,
> > > (which I believe halves the chroma resolution - can we possibly
> > > take this in to account as well?) )
> > >
> > > * Stobe effect of a CRT filmed by a different scan-rate camera
> > >
> > > * Colour changes in multi-layered painted cel animation, (where
> > > a character is, say, talking, and they repeatedly add and
> > > remove a cel from the top layer, their face flashes bright and
> > > dark :-) )
> > >
> > > Infact, existing video compression codecs do a *terrible* job
> > > of encoding hand painted animation, especially anime, despite
> > > claims by well known studios that it is not an issue.  Good
> > > motion estimation to compensate for poor registration of
> > > animation would be worth investigating.
> 
> Agreed, simple RLE does a lot better than MPEG on cartoons for 
> example. However, I think cartoons are better off with a separate 
> codec that's optimised for that. Probably something that can 
> extract curves and surfaces and store them efficiently.

Definitely - a vector-extracting codec would be interesting, because then you could do all sorts of fun things like:

* Interpolate between frames for really smooth conversion between frame rates
* Interlace and de-interlace cleanly
* Properly scale up existing material, for HDTV and film applications.

OK, I am getting well ahead of what is practical to implement, but those are nice ideas for a future codec.

> > > I think that the problem with all codecs to date is that they
> > > are designed to compress perfect material originating directly
> > > from a camera.  In my opinion, the above *analogue* artifacts
> > > are what give the digital codecs a hard time.
> 
> It's definitely an interesting idea. I'm not sure how difficult it 
> would be to reall integrate this into the codec, and whether it 
> wouldn't be easier to try and filter out the artifacts before 
> giving the stream to the codec.

A pre-processor might be better, because just about everything we've mentioned would be considered wrong by some people, especially fixing dropouts, etc, as that's really moving away from trying to preserve the input video, and moving towards trying to guess what the original video looked like.

> Do you know of any existing algorithms to filter out these artifacts?

Not really.  Also, to be honest, I'm not really very knowledgeable in that specific field.  I am a programmer, but not a great mathmatician, (just a reasonable one ;-) ).

Reverse 3-2 pulldown looks fairly straightward, but there is the problem of working out which fields go together to make the original frames.  As long as field dominance, (I know that's the wrong term, but the correct one escapes me, and you know what I mean), doesn't change partway through, this should be obvious at the first scene change, because there is going to be one combination that is obviously wrong, and simply doing an RMS of the differences between lines should reveal that one.

FPS conversion artifacts - there are loads of ways to do 25-30 and 30-25 conversions.  Not sure where to start.

Dropouts - just repeat the line above, maybe?  Or use the other field of interlaced video?  Awful, I know, but what else can you do, unless the drop out is 1 line, when you could interpolate above and below.

Unfortunately I've got no useful suggestions at all about how to implement the other ideas :-(.

John.
--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to '[email protected]'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.