Re: Some perspective from the cheap seats...

On Tue, Mar 25, 2003 at 01:27:45PM -0800, Allen Akin wrote:
> On Mon, Mar 24, 2003 at 11:29:18PM -0500, Havoc Pennington wrote:
> | I think the answer to that is "no" - in general, we have near-zero
> | understanding of where hardware is going. ;-) Some people might.
> 
> It's too big a topic to handle in a short note, and I might not be the
> best-qualified person to attempt it.  But for desktop systems:
>

Thanks a lot for the detail here, I've saved your mail in a text file
for my future reference. Especially the reading material suggestions
should be helpful.
 
> | Here are some things I can think of quickly:
> | 
> |  - WYSIWYG. We need the same API to draw to screen, printer, and a
> |    raster image to be dumped to a file, generating results that are
> |    basically the same. A pixel off here and there is OK in most cases,
> |    but using a different font or doubling line widths or something
> |    would not be. And we need client-side code available that can draw
> |    the primitives, even if there's also a server-side option.
> 
> When you say you want to draw to devices that differ as much as screens
> and some printers, that tells me we also need to think about larger
> issues in the rendering model.  (For example, where the responsibility
> for sorting lies when the rendering operations are order-dependent, as
> in blending.)  Solving this in a completely general way requires a
> higher-level knowledge of the scene to be rendered, something that
> low-level APIs like OpenGL offer assistance for, but don't address
> entirely.
> 
> As for client-side drawing code, I'm not clued-in enough to understand
> all the issues.  For example, do you intend to rule out
> hardware-assisted direct rendering when running locally?  Is this
> something that the application can be expected to decide, or is it up to
> the rendering library?  What are the key factors in making the
> decision?

I'll try to be a bit more verbose about when we'd use client-side
rendering or software fallbacks, hopefully I'll accidentally answer
your questions in here somewhere. ;-) 

1. Alternative output devices

Ideally, there is one API that applications use with various targets
(printer, server-side buffer, client-side buffer), and we simply
accelerate it when possible.

Consider the case of the GIMP, an image editor. It needs to use the
same font rendering API used in a word processor, or just to draw your
buttons and menus. But, it needs to draw them to a raster image on the
client side that can be saved in the GIMP file format.

Or more simply, if I have a diagram or something in a document, I need
to be able to print it, and export it to PNG format.

App developers aren't willing to write their code 3 times in order to
do this - and they aren't happy if the PNG or printed output is
different enough from what was on the screen for users to notice.

So on the GTK+ level, what we would want to present is one API that
has a Device or RenderTarget object, that can represent your various
output devices. Then you can write:

 void
 render_my_diagram (GtkOutputDevice *device)
 {
   gtk_render_line (device, x1, y1, x2, y2);
 }

and call render_my_diagram() for the various output devices.  If the
output device is the server-side buffer, then gtk_render_line() would
use acceleration when available, automatically.

This is why RENDER basically just draws triangles or trapezoids or
whatever, right - because then your "if (server-side) foo() else
bar()" code is limited to the part that pushes those primitive shapes
and you can just have one copy of all the higher-level code such as
"draw a circle." But on the GTK level things will be virtualized
enough that you could later decide to move "draw a circle" into the
server and we could take advantage of that. Another advantage of
keeping most code client-side is just that you aren't likely to have
different bugs/misfeatures in the server vs. client implementation.


2. Old X servers

We can't really rely on features where the software fallback is
totally unusable. For example, if you run certain games on an XFree86
server without hardware acceleration, it pretty much locks up the X
server due to the slowness of the software GL.

In cases like that, we would either need to have some way to
automatically disable animations/effects wholesale, or we'd need the
animations/effects to be adequate in software and use hardware only as
a "make it even snappier" option.

It's not really acceptable to launch a game and have your system lock
up; those games really should be opening a dialog saying "sorry" and
then exiting again. I don't know if there's currently a way for them
to detect the problem and do so. Similarly for the desktop, if we had
effects that were only usable with hardware accel, we'd need to be
able to disable them automatically and use some fallback.

Lots of things can't be disabled - say, drawing text, or drawing a
diagram in a spreadsheet. We can't open a dialog saying "sorry, you
need a better graphics card to draw a pie chart" ;-) So in these cases
the software fallback is required to be "fast enough" and hardware
accel is only allowed to be icing.

Not only does the software fallback have to be good enough, a
client-side software fallback has to be good enough (in case servers
lack the extension).



I guess the short summary here is that for most "real work" rendering,
we can only use features that are usable with client-side software
fallback, and hardware accel can never be more than icing on the
cake. It can be pretty *significant* icing - if some users get 10x
faster rendering, cool. But the software fallback does have to be
usable. *How* usable depends on how many people have the hardware
accel. ;-)

For features that are kind of optional, like animated genies, we can
go ahead and just write "if (have_accel) do_nice_animation() else
do_ugly_xor_hack()" assuming there's a way to ask whether we have the
acceleration that matters for the given effect.
 
> |  - Performance not such an issue. We care about performance, but not 
> |    *as much* as others might. Not that it hurts to be faster.
> 
> :-)

I think in the above text I managed to come up with the *reason* why
performance isn't as crucial, namely all the cases where we can't rely
on having hardware accel anyhow. So we definitely can't rely on its
exact speed in those cases.

But we might care about performance a lot for a feature that 
was only really usable with hardware accel, since we'd then need it to
be fast enough for whatever we were trying to do, and it's not a given
that it is.

If there's more cool stuff available with server extensions and
hardware accel there will probably be more pressure to drop support
for old X servers over time.
 
> I'm reminded of a conversation I had in the Olden Days with Dave Brown
> shortly after he left SGI and joined DEC.  Dave described how people
> were stunned by the text-scrolling speed on the then-current SGI machine
> (probably a GTX, I don't remember for sure).  Dave laughed and explained
> that the terminal emulator didn't scroll; it just cleared the window and
> re-rendered the text in its new position.  When you've got enough
> horsepower to render 3D scenes at interactive rates, you might be able
> to find new ways to handle 2D effects, including brute force. :-)

Nalin (author of the gnome-terminal "engine", vte) was telling us
about his plan to use GL to accelerate vte the other day - it sounded
a little bit scary, but I'm curious to see the outcome. ;-)

Havoc