PATCH] glx/drisw: fix xshm_opcode never initialized, add XShmAttach fallback

=46rom b87c39149704a03872fd02adc59d56f16a9396f1 Mon Sep 17 00:00:00 2001
From: Peter Molfese <[email protected]>
Date: Tue, 12 May 2026 15:05:34 -0400
Subject: [PATCH] glx/drisw: fix xshm_opcode never initialized, add =
XShmAttach
 fallback

fixes https://gitlab.freedesktop.org/mesa/mesa/-/work_items/15445

glx/drisw: fix xshm_opcode never initialized, add XShmAttach fallback
=20
 xshm_opcode was declared and initialised to -1 but never assigned the
 actual MIT-SHM major opcode from the X server. This caused two bugs:
=20
 1. handle_xerror compared event->request_code against -1, which can
    never match a real opcode, so xshm_error was never set.
    Consequently XShmPutImage was called against a segment the X server
    had already rejected at XShmAttach time, producing:
=20
      X Error of failed request: BadShmSeg (invalid shared segment =
parameter)
      Major opcode of failed request: 133 (MIT-SHM)
      Minor opcode of failed request: 3 (X_ShmPutImage)
=20
    This is the failure mode seen on Mesa main with XQuartz on macOS.
=20
 2. Because xshm_opcode stayed at -1, the assert(xshm_opcode !=3D -1) at
    the top of handle_xerror fired the moment any X error arrived while
    the handler was installed, aborting the process with:
=20
      Assertion failed: (xshm_opcode !=3D -1), function handle_xerror,
      file drisw_glx.c, line 62.
=20
 Fix by:
=20
 a) Calling XQueryExtension(\"MIT-SHM\") once per Display the first time
    XCreateDrawable attempts SHM, storing the returned major opcode in
    xshm_opcode. If the extension is absent, jump to the non-SHM path.
=20
 b) Explicitly clearing xshm_error to 0 before installing the error
    handler, removing a latent dependency on the global being zero from
    a previous call.
=20
 c) Adding a comment explaining why the XSync after XShmAttach is
    required: it forces the X server to process the attach before
    XShmPutImage is issued, which is the pattern required for servers
    (like XQuartz) that process the attach lazily.
=20
 d) Replacing assert(xshm_opcode !=3D -1) in handle_xerror with a
    conditional return, so a defensive path remains without aborting
    the process.
=20
 e) Emitting a one-shot mesa_logw_once when the attach is rejected so
    the fallback to XPutImage is observable in the log without spam.
=20
 f) Adding a no_shm label so the opcode-query failure path can jump
    directly to the non-SHM XCreateImage fallback.
=20
 Tested on macOS with XQuartz using glxgears from mesa-demos 9.0.0.
 Previously crashed with the assertion; now runs correctly and emits:
=20
   MESA: warning: MIT-SHM attach rejected by X server (error 10);
   falling back to XPutImage
=20
 Error 10 is BadAccess, confirming XQuartz rejects the SHM attach due
 to macOS inter-process memory restrictions. The XPutImage fallback is
 transparent to the application.

A demonstration Homebrew formula applying this patch is available at:
https://github.com/pmolfese/homebrew-mesa-fix
=20
 Cc: [email protected]"
---
 src/glx/drisw_glx.c | 53 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 51 insertions(+), 2 deletions(-)

diff --git a/src/glx/drisw_glx.c b/src/glx/drisw_glx.c
index 2a3b630fa20..712cfd9a724 100644
--- a/src/glx/drisw_glx.c
+++ b/src/glx/drisw_glx.c
@@ -59,7 +59,14 @@ handle_xerror(Display *dpy, XErrorEvent *event)
 {
    (void) dpy;
=20
-   assert(xshm_opcode !=3D -1);
+   /*
+    * xshm_opcode is populated by XCreateDrawable before this handler =
is
+    * installed.  It should never be -1 here, but guard defensively =
rather
+    * than asserting so that a misbehaving X server cannot abort the =
process.
+    */
+   if (xshm_opcode =3D=3D -1)
+      return 0;
+
    if (event->request_code !=3D xshm_opcode)
       return 0;
=20
@@ -78,6 +85,33 @@ XCreateDrawable(struct drisw_drawable * pdp, int =
shmid, Display * dpy)
    }
=20
    if (!xshm_error && shmid >=3D 0) {
+      /*
+       * Populate xshm_opcode before installing the error handler.
+       * Previously xshm_opcode was never assigned after being =
initialised to
+       * -1, so handle_xerror's request_code comparison never matched =
an MIT-SHM
+       * error and xshm_error was never set.  This meant XShmPutImage =
was called
+       * with a segment that the X server had silently rejected at =
XShmAttach
+       * time, producing BadShmSeg (seen with XQuartz on macOS).
+       *
+       * XQueryExtension fills in the major opcode we need.  If it =
fails the
+       * extension is not available and we fall through to plain =
XPutImage.
+       */
+      if (xshm_opcode =3D=3D -1) {
+         int opcode, ev, err;
+         /*
+          * XQueryExtension gives us the major opcode that the X server =
uses
+          * for MIT-SHM requests.  We need it so handle_xerror can =
correctly
+          * identify MIT-SHM errors by request_code.  Previously =
xshm_opcode
+          * was left at -1 forever, so the error handler's comparison =
never
+          * matched and xshm_error was never set, causing XShmPutImage =
to fire
+          * against a segment the server had already rejected =
(BadShmSeg on
+          * XQuartz/macOS).
+          */
+         if (!XQueryExtension(dpy, "MIT-SHM", &opcode, &ev, &err))
+            goto no_shm;
+         xshm_opcode =3D opcode;
+      }
+
       pdp->shminfo.shmid =3D shmid;
       pdp->ximage =3D XShmCreateImage(dpy,
                                     NULL,
@@ -92,13 +126,26 @@ XCreateDrawable(struct drisw_drawable * pdp, int =
shmid, Display * dpy)
          /* dispatch pending errors */
          XSync(dpy, False);
=20
+         xshm_error =3D 0;
          old_handler =3D XSetErrorHandler(handle_xerror);
          /* This may trigger the X protocol error we're ready to catch: =
*/
          XShmAttach(dpy, &pdp->shminfo);
+         /*
+          * Force a round-trip so the X server processes XShmAttach =
before we
+          * ever issue XShmPutImage.  Without this sync, XQuartz (and =
other
+          * servers that process the attach lazily) reject the =
subsequent
+          * XShmPutImage with BadShmSeg because the segment is not yet
+          * registered server-side.
+          */
          XSync(dpy, False);
=20
          if (xshm_error) {
-         /* we are on a remote display, this error is normal, don't =
print it */
+            /* X server rejected the SHM attach (e.g. XQuartz sandbox,
+             * remote display).  Fall back to plain XPutImage silently;
+             * the caller treats ximage =3D=3D NULL as the non-SHM =
path. */
+            mesa_logw_once("MIT-SHM attach rejected by X server (error =
%d); "
+                           "falling back to XPutImage",
+                           xshm_error);
             XDestroyImage(pdp->ximage);
             pdp->ximage =3D NULL;
          }
@@ -107,6 +154,8 @@ XCreateDrawable(struct drisw_drawable * pdp, int =
shmid, Display * dpy)
       }
    }
=20
+no_shm:
+
    if (pdp->ximage =3D=3D NULL) {
       pdp->shminfo.shmid =3D -1;
       pdp->ximage =3D XCreateImage(dpy,
--=20
2.50.1 (Apple Git-155)