Re: Projective image transformations

Around 10 o'clock on Sep 13, Allen Akin wrote:

> Sorry for the delay -- I have only once-a-day email access at the moment.
> I hope this is still relevant.

Yes, we're quite a ways from concensus here.  Thanks much for your input.

> First, beware of feature creep.  You're really getting into 3D territory and
> image-processing territory now.  Both are littered with wheels waiting to be
> reinvented.

An image of the Oregon trail springs to mind here...

Applications want to perform image transformations; especially as monitor 
resolutions depart from the standard 96dpi values that several desktop 
environments assume.  Switch to a 200dpi display, and you'd like the 
images in your web page to grow larger so that the page is scaled together.

Another important application is in accessibility -- we need to 
dynamically magnify selected areas of windows to make them accessible to 
users with limited visual abilities.

There are a host of other reasons to provide our 2D system with basic 
image reshaping operations.   I don't think we can just skip this ability, 
but we certainly don't want to specify anything known to be broken, and 
I'd also like to limit the functionality to what we know is necessary and
let future additions to the protocol add new functionality as needed.

> My understanding is that resampling really means "reconstruct (a continuous
> representation of the signal), filter, then sample."

In the DSP world, resampling is done by upsampling to the LCM, filtering, 
downsampling to the target sample rate and then filtering again.  I think 
this is essentially the same thing.  Of course, implementations generally 
have just a single FIR that walks over the input and generates the output, 
but the filter coefficients contain the composite of both filters and the 
resampling stages.

Nearest neighbor is the simplest form of this; it's just a box filter, 
which isn't particularily good at reconstructing the signal, nor does it 
effectively eliminate aliasing as it doesn't clip values above the Nyquist 
limit.

I guess bilinear interpolation can be considered a filter as well, kind of 
a two element FIR.

> Hard to tell where this technology is going from a point of view outside
> the IHVs, but I'd bet we'll see more options.

I think that just means we need to make sure the set of filters we allow 
is extensible.  Owen had a suggestion that we provide aliases for virtual 
filter names like "fastest", "smooth-but-not-too-slow", "best-results" so 
that applications could use these names instead of attempting to pick one 
of the physical filter names.  That would permit redirection of 
applications using the generic names to specific filters which provided 
the expected performance tradeoff.

> Current 3D hardware generally supports projective tranformations of texture
> coordinates before texel lookup.  (It's needed to avoid artifacts when
> geometric primitives are broken into triangles.

I was already assuming that the hardware could perform projective
transformations; thanks for verifying this.

>  You may need a 4x4 matrix (like the OpenGL texture matrix) for this,
> though.

I'll read through the OpenGL spec more carefully then; homogeneous 
transformations in the 2D world need only a 3x3 matrix, but perhaps I'm
missing something important here.

> OpenGL has several ways to handle edge pixels.

Right now, I'm exposing three different mechanisms, repeat, nearest and
transparent.  The one case I've eliminated is replacing missing pixels with
a solid color.  The reason for this is that there isn't a representation 
for transparent in every color space, so I'd have to use color values, but 
that leads to interesting questions if dithers get introduced in the 
future.

> Clamping to a mix is used when available texture memory is much smaller
> than the size of the texture image, and you must use multipass techniques
> to draw a piece of the texture image per pass.  This latter case is rarely
> supported in hardware these days.

I don't constrain source image sizes, so any operation like this would 
have to be synthesized within the driver itself.  Fortunately, the days of 
limited off-screen memory are pretty much gone.

> The full mipmap stack costs only 1/3 as much memory as the base texture.  If
> you're doing a lot of minification, it's usually considered a good tradeoff.

As our source operands are writable objects, it seems like it will be 
somewhat difficult to provide good semantics for mipmaps, but perhaps we 
could build synthetic source pictures from sources at multiple resolutions 
to do the same thing.  However, I think this remains just a performance 
optimization at some level; I'm guessing most mipmaps are generated by 
shrinking the larger image with a more accurate filter.

> Current consumer hardware does, both for full-scene antialiasing and for
> anisotropic filtering of textures.  Current high-end hardware can do a lot
> better, so maybe we'll see that filter down to consumer hardware before long.

Perhaps we should add some of the common consumer hardware filters to the 
basic package then.  Anything to improve on bilinear interpolation.

Keith Packard        XFree86 Core Team        HP Cambridge Research Lab