Re: crazy idea for encoding

> What you have outlined would indeed provide significant compression if
> it could effectively be implemented. The problem is the implementation.
> Getting structure data from moving images is actually remarkable difficult.
>
> Consider that we have bi-nocular vision or a known camera trajectory.
> Also assume that the points we observe are approximately stationary
> between frames. With such assumtions it is relatively trivial to derrive
> from perspective geometry the position in 3D space relative to the
> oberver a given point given two or more frames.
Well, as we've got only 1 2d-image to construct our data from, the problem is 
easier :-)

> The problem comes when trying to track multiple points (e.g. the corners
> of a wall) in that it is remarkably difficult to say which points in
> image 'A' correspond to points in image 'B' (it is known as the
> 'correspondance problem') consider the problems:
>
>   - Points in image A might disappear off the edges in image B
>   - Our 'wall-corner-detector' might fail and 'lose' one of the points
> in image B. How do we know which point in image A isn't in B?
>   - Spurious points might show up in either image, how do we discount them?
>
> Many techniques (such as the 'epipolar constraint' method) have been
> devised to reduce the search space in such problems but generally the
> problem is very difficult.
>
> A pragmatic person might argue that we are not aiming for an accurate 3D
> model of the scene, only one which looks adequately like it when
> rendered from the observers vantage. Certainly such an algorithm could
> be created but one then has to worry about corner cases. The ones that
> leap to mind are:
>
>   - Glass objects -- How do you handle transparency?
>   - Moving objects -- In order to get a 3d model you must assume
> stationary scent and a known camera trajectory (or at least a scene
> which remains stationary with respect to itself). Removing this
> condition effectively increases your search space exponentially.
>
> This is just a brain dump of things which should be considered if the
> coded were to be able to deal with 'real-life' videos.
Well, I'm with you in that it won't be easy.

But suppose the following:
- If the image changes radically between frames, it's either a new scene or 
some heavy action - which can't be encoded with my idea.
- If the scene doesn't change much - I'll further assume a star trek movie :-] 
where the uss enterprise is shown in space - -- that is, if many pixels stay 
the same from one frame to the other, than we can go looking. I expect that 
that happens nowaday too with video encoding (looking for similarities 
between frames).
This looking goes from the continuous pixels to their edge (via an edge filter 
or where the frames differ). In my example this will be the stars or the 
outline of the starship. In some part of the scene I expect that the scenery 
doesn't change much, ie., the stars move very slowly across the screen, and 
from that point (normally start or end of scene) these areas of change can be 
tracked as they move along. As long as there's continuous movement or 
acceleration this should be possible.
So we can identify corresponding points in the frames. Using these (and the 
connections between them, which are identified by an edge filter [where color 
changes]) we can guess some "sprites". Initially these sprites are 2d (and 
some will stay so - eg. the star background). But if a sprite itself changes 
in that for example the middle goes to the right but the outline goes to the 
left, areas disappear to the right edge and new ones appear to the left side, 
then we can assume that this object is spinning while moving.
Or the background: If the image shifts to the right, we can append the new 
bits from the left to the already existing image.

I don't propose to construct complete 3d implementations - that's too much 
overhead both encoding and decoding. [But it would be cool to once 
reconstruct the enterprise as a 3d object - some scenes would then just say 
"move object Y along path Z over background W", possible with a zooming 
factor - that would save bandwidth as this object needs to be described only 
once, although it may be a problem with streaming] [Would be cool to get 
"indefinite" resolution - in such a movie we could zoom in ...]

So if I'm not completly wrong in assuming that there has to be a comparision 
between frames I think it won't be so much overhead to (first) identify 2d 
sprites and then try to get them as 3d working.


Regards,

Phil



--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to '[email protected]'
containing only the word 'unsubscribe' in the body.  No subject is needed.
Unsubscribe messages sent to the list will be ignored/filtered.