Re: GLX in bad state

You ran the demo on a system with a GeForce FX 5600, Athlon 2500 with 1 GB 
system memory.  You did not cut-and-paste the benchmark file summary so I 
do not know what options you used.  You are the CTO of TG and you should know 
better.  It is possible that you are deliberately doing this so as not to 
provide publicly accessible information which will allow a performance 
comparison between WineX and DirectX.

I continue to encourage Transgamers to provide their results.  When you 
(Transgamers) do compile the patch please change the line in glx_private.h:
from #define CACHESIZE 64
to   #define CACHESIZE 128

I have added some comments below.  However, some of the items are 
convoluted and irrelevant.

On Mon, 12 Apr 2004, Gavriel State wrote:

> Nadav Frum wrote:
> >>As we've stated in the past, we are very particular about the patches that
> >>we allow into CVS, and we don't like checking things in that have either
> >>known issues or that have known paths that might cause issues, unless there's
> >>a very compelling reason for it.
> >
> >
> > I think I know a bit about the WineX code.If you have an invalid pointer 
> > to an unimplemented function, WineX will crash when that function is
> > called.Therefore, there are no negative issues with adding a stub.
> 
> The issue is not whether adding the stub would cause problems, it was whether
> putting the stub in CVS would force him to waste time updating his unfinished
> pixel shader patch.The answer was yes, it did, but he did it anyway.

Really?

> > Now perhaps you will come forward with some benchmark results.
> 
> I wasn't actually using the benchmark mode before, since visually I wasn't
> getting more than about 2fps.Leaving out the pbuffer->texture copy didn't
> change that significantly.
> 
> Thanks to Ove's suggestion, I'm now getting 19fps.More on this later.
> 
> The machine is an Athlon 2500 with a GeForce 5600 and 1 GB of RAM, my
> home gaming box.
> 
> >>As Ove stated, this is the overall philosophy of the OpenGL design in
> >>general, not ours.Obviously performance is very important to us.
> >
> > How is the overall philosophy of the OpenGL design (whatever it may be)
> > relevant to this discussion?
> 
> It's relevant because you started the discussion raising the idea that
> there should be a new OpenGLextension added to support render-to-texture
> in GLX.

This GLX extension would not be implemented by the OpenGL designers, so 
how is their philosophy relevant?  This is not a Philosophy forum.
 
> >>If you'd like to determine where there might be further performance issues
> >>with your updated video memorymanagement code, you might try doing a trace
> >>with +timestamp,+x11drv,+ddraw to narrow down what's actually taking up
> >>time.
> >
> >
> > This is a bad approach for the following reasons:
> >
> > 1.This level of debugging will slow the performance to such an extent 
> >   that the overall analysis of performance will become irrelevant.
> 
> Not true.Slowdown will generally be proportional, so it's unlikely to
> distort the results.

if,   Time(Routine A) = k * Time(Routine B)
then, Time(Routine A + TRACE) != k * TIME(Routine B + TRACE)
i.e. not proportional.  

> > 2.The overall performance is an accumulation of routines which on their 
> >   own waste less CPU time than is measurable by the time stamp.
> 
> But the problem we see here, as I mentioned, is showing gaps of 200ms and
> higher in bits of code within the glx.c code.

While #define CACHESIZE 64.  This is no longer an issue. 

> In general, that technique is good so long as lots of time is being spent
> within a given Win32 API call from the app's perspective.If an app has a
> tight loop calling a Win32 API which takes less than one millisecond, it's
> less useful.

True.

> > 3.The performance may be slow because the graphics card is used in an 
> >   inefficient way.  Such performance losses will occur when the video 
> >   card executes the command and not when the command is issued.
> 
> This is impossible to determine with any standard profiler - for that you need
> tools that know the hardware intimately.

Without NVidia's insider analysis, benchmarks offer some help.
 
> > 4.The timestamps will also include time spent executing the game's
> >   binaries and libraries other than x11drv and ddraw.
> 
> Yes, but not while we're within an API call.
> 
> > 5.The game's binary may be doing some excess computation in response to 
> >   unexpected feedback from WineX.  This will not be revealed.
> 
> Possible, but unlikely, and impossible to determine with any other kind of
> profiler.

A benchmark comparison between similar systems where the only difference 
is CPU power may be very revealing.

> > A much better approach is a profiler like 'oprofile'.It has the 
> > limitation, however, that it does not show what the video card and the
> > games's binary is working on.The information received from differently 
> > spec'ed systems would benefit such an analysis.
> 
> Yes, we're quite aware of the different profiling tools available on the
> system, having been involved in writing them.Most Linux profilers are
> nice in theory, but less good in practice.That's why I designed cprof
> at Corel, which was implemented by Andrew Lewycky, who later worked at
> TransGaming.Sadly, cprof also turned out to be less good in practice
> (it generated too much output data too quickly, on the order of hundreds
> of megs a minute).
> 
> Regardless, since I wanted to see how it was progressing, I had another
> go at oprofile to see how it was doing.Sadly, it hasn't advanced much
> from the last time I looked at it.

I like the work they have done.

> oprofile is not yet capable of providing call graph profiles, though they're
> working on it.Once they get it going, we'll probably put some effort into
> making sure it can read Win32 stacks as well as Linux stacks so that we can
> get some useful data from the Windows side.

This is a good idea and one I have not mentioned before.

> Currently. without call graph profiles, you end up with results that tell
> you almost nothing useful, like this:
> 
>             7806 81.0676 libGLcore.so.1.0.5336
>              764  7.9344 c4demo.exe
>              200  2.0771 vmlinux
>              183  1.9005 libd3dgl.so
>              176  1.8278 libx11drv.so
>              116  1.2047 libGL.so.1.0.5336
>              111  1.1528 zero
>              106  1.1008 nvidia
> 
> The culprit here is of course libx11drv, but you wouldn't know it from that
> output.

If these are the results of running the benchmark on Oprofile it is very 
useful and interesting.  My results look like this:

   268510 37.2090 c4demo.exe
   164817 22.8396 libGLcore.so.1.0.5336
    84837 11.7564 libd3dgl.so
    72135  9.9962 vmlinux-2.6.4-custom
    27270  3.7790 libd3d8.so
    24785  3.4346 libx11drv.so
    14861  2.0594 libGL.so.1.0.5336
    12153  1.6841 libc-2.3.2.so
    10476  1.4517 nvidia

If you are not using hardware shaders and this is not a cache size 64 
run, then the large figure for c4demo.exe on my system is significant.
It suggests that the c4demo.exe may be running an idling routine.
This means that the CPU is waiting for the GPU and not the other way around.

If it is a GPU problem try finding out where the bottleneck is as follows:
1. Run at different resolutions, with and without FSAA.
2. Disable and enable the indexed drawing routines in prim.c.
3. Disable and enable texture copying. 

> oprofile also lacks the ability to extract names from stripped libraries
> (ie: libGLCore.so.1.0.5336), so when we ask for a symbolic breakdown, we
> simply don't get any of those symbols.The best we can get, with the application
> of a patch I've made to oprofile that gets the dynamic symbol previous in the
> image to the point in question is something like this:
> 
> samples%        image name               app name                 symbol name
> 6102   62.1765  libGLcore.so.1.0.5336  wine                     _nv000054gl
> 764     7.7848  c4demo.exe               wine                     (no symbols)
> 536     5.4616  libGLcore.so.1.0.5336    wine                     _nv000061gl
> 499     5.0846  libGLcore.so.1.0.5336  wine                     _nv000073gl
> 324     3.3014  libGLcore.so.1.0.5336    wine                     _nv000077gl
> 
> This is equally useless, though some asm poking shows that the driver is spending
> all of its time copying some buffers around.Some debugging back to the call point
> of the slow routine shows that it's being called from (drumroll)....the GL call to
> create a pbuffer, exactly what I said it was half a dozen emails ago.
> 
> With this confirmation, and Ove's additional comments on the caching scheme, I had
> a go at increasing the cache size from 64 pbuffers to 128.That instantly gave me
> a 10x performance boost, as we were no longer running out of cached pbuffers.

2x on my system.  Old news.  I guess you did run Oprofile before changing 
the cache size to 128.  Try to move on.

> So, what can we conclude:
> 1) NVidia's drivers aren't good at allocating new pbuffers for some reason

PBuffers are not meant to be allocated on a frame by frame basis.  NVidia 
is not to blame here.

> 2) Over a scene's lifetime c4 is setting up lots of textures for rendering, and
>    blowing our cache.  It's not clear how active these are though, as I mentioned
>    earler that I'd only seen 7 such textures before.

Change to 128 for the cache size and let's move on.

> 3) If some of these are not actively used for rendering during a scene, it's
>    possible that our cache policy is to blame.  We may want to switch to a true
>    LRU cache.  That could potentially eliminate the problem without requiring
>    a significant increase in the cache size.

Get ARB_render_texture.  Otherwise do a fallback implementation.

> 4) It may be useful to set the default cache size larger in any case.

Point 2.

> and, most importantly,
> 
> 6) Performance improvements aren't about how you profile, they're about how you
>    analyse the data you have.

Obviously.

> Anyhow, I'll leave it to Ove and others to decide whether the cache algorithm
> needs significant changes, and about the arguments for and against upping the
> cache size further or leaving it alone.

Start at 128 and double it thereafter with a WARN when it isn't enough.

On Mon, 12 Apr 2004, Gavriel State wrote:

>>  3) If some of these are not actively used for rendering during a scene,
>>     it's possible that our cache policy is to blame.  We may want to switch
>>     to a true LRU cache.  That could potentially eliminate the problem without
>>     requiring a significant increase in the cache size.
>
> So, some further experimentation bears this out.  With some randomness (but not
> real LRU) added to the victim-selection algorithm, but with a stable cache size,
> we get behaviour which is better, but still kind of choppy.

Is the 19 fps or 2 fps choppy?

>An LRU would do much better.
>
> And we probably want to look for ways that we can preserve a pbuffer even if
> it's a cache victim if it happens to be the same size as the new pbuffer that
> will replace it.  We are perhaps a bit zealous about destroying the victim
> entry completely and then allocating a new entry in its place.

I think your Least Requested Unit (LRU) scheme is a bad idea for the 
following reason:

Your LRU scheme would presumably be applied when the GPU refuses to 
allocate additional memory.  However, as the video memory approaches the
limits the GPU will gradually start working slower and slower without causing
that final failure.

You could start the LRU much sooner but then you are risking under-utilising
your video memory (e.g. go back to the 32 MB for all scheme).  
You are also using CPU power.  The best approach is to let the video driver
decide how to allocate memory, which brings us back to ARB_render_texture.

I think it would be helpful if you admit that ARB_render_texture is the 
best solution and that the rest are just less satisfactory ways of 
managing without it.

Nadav Frum
----------