Re: New paper on motion compensation

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 25 July 2002 22:52, Lourens Veen wrote:
> I just got this off Slashdot, so at least some of you probably
> already know, but here's a post for those who hadn't seen it yet,
> and ofcourse for discussion of the used techniques.

Well, I've read some more and while I haven't really studied the 
math very hard, it does look promising indeed. The authors describe 
an algorithm that is meant to take a sequence of partially 
overlapping frames shot by a stationary camera that is free to 
zoom, rotate, pan and tilt, transform them all to a single 
coordinate system, and then combines all the images to create a 
single high-resolution image. If the object displayed is planar (ie 
a flat face) the camera is also allowed to move. Furthermore the 
algorithm is pretty robust; small violations of these rules still 
give good results, and according to the paper it can be implemented 
quite efficiently. Page 25 in the aforementioned paper shows a nice 
example of the technology at work. Another interesting result is at 
the top of page 28. Finally, there is GPLled code available on 
sourceforge (http://comparametric.sourceforge.net/), I'll have a 
look at that soon as well.

So, how to use this in video compression? Assuming a static scene 
for a moment (ie no moving objects, as in the image on page 25), we 
could throw the image sequence through this algorithm, wavelet or 
simply JPEG compress the resulting high-res image, and for each 
frame store the 8 floats that describe the transformation from our 
source image to the frame at hand. Using texture mapping hardware 
this costs about 0 CPU to play back. Seeking would not require 
playing back from the last keyframe, since once you have the 
high-res image any frame based on it can be rendered at once.

However, most scenes feature moving objects. The image at the top of 
page 28 in the paper shows what happens then. Big mess, if you want 
to use it in the way I described above. However, once you have 
overlapping frames, it's trivial to find areas that don't match up. 
These areas could be masked out according to a majority rule (ie in 
that image, if there are 5 frames showing grass, and 1 frame 
showing a player, the codec decides it's grass). After transforming 
this mask back to the video frames, we can split the frames into 
objects, which can then be pushed through the algorithm again to 
yield a separate high-res image for each object, and a separate 
bunch of transformations for each object. A frame then consists of 
a list of objects and a set of masks (which could be RLE 
compressed, or perhaps a zerotree-encoded quadtree is more 
efficient) where each object is a reference to a high-res image and 
a set of transformation coefficients.

This splitting up into objects might also remove the problem of a 
moving camera, if the objects are sufficiently similar to planar 
surfaces.

So, what do you think?

Lourens
- -- 
GPG public key: http://home.student.utwente.nl/l.e.veen/lourens.key
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9QTLvvmNyqZHWDvURAhV9AKCwFWfKb0gTlezJRoxGU4XySN68lACfUIIQ
RsJYW8GJgXdV7GEi9tBk5Bc=
=3Fta
-----END PGP SIGNATURE-----


--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to '[email protected]'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.