Re: X11 BadValue errors on Solaris 10..

Wed Sep 28 2005, [email protected] wrote:
> 

> x: 2147483496 7fffff68,  y: 0 0, w: 1023 3ff, h: 576 240

> Any ideas?

Yes, this happens when the window is smaller than the "scaled" image.
There was a bug in set_videoarea, two unsigned ints that were subtracted
and then divided by two and then converted to a signed int.
If the result from the subtraction was 'negative' the division made the
resulting signed int very large.
For some reason this worked on my Xserver anyway and on yours without xshm.
So there must be some error correction built in or maybe there are some 
unsigned/signed/short wrap arounds in the server that 'corrects' this.
Anyway, here is the patch (I have comitted it to CVS too)

/Bj|rn

RCS file: /cvsroot/ogle/ogle/mpeg2_video/video_output_x11.c,v
retrieving revision 1.123
diff -u -r1.123 video_output_x11.c
--- video_output_x11.c  25 Sep 2005 08:51:04 -0000      1.123
+++ video_output_x11.c  28 Sep 2005 23:31:44 -0000
@@ -850,10 +850,10 @@
 {
   w->video_area.width = width;
   w->video_area.height = height;
-  w->video_area.x = (w->window_area.width -
-                    w->video_area.width) / 2;
-  w->video_area.y = (w->window_area.height -
-                    w->video_area.height) / 2;
+  w->video_area.x = ((int)w->window_area.width -
+                    (int)w->video_area.width) / 2;
+  w->video_area.y = ((int)w->window_area.height -
+                    (int)w->video_area.height) / 2;

   return;
 }