bug#81407: 31.0.90; Crash in display_line: stale glyph_row after window-config change from menu-bar :enable eval during redisplay

Aaron Iba <[email protected]>
Newsgroups gmane.emacs.bugs
Message-ID <CAH_c0bc9M=H8Os=D0DGJ12P0dqKDE+TQtwS9yMw+obqJR8ON-A@mail.gmail.com>
> Try the patch below, it seems to avoid the trouble here.

Thanks!  I applied it to the emacs-31 branch (builds as 31.0.91 now)
and tested on macOS/NS, arm64, --without-native-compilation.  Results:
the patch helps but the guard is unreliable here -- I still get the
crash in a fair fraction of runs, and I found out why.

Test results with the -Q recipe (5 plain runs + 3 instrumented runs):

  - guard fires, evil Lisp aborted, no crash:  ~1/4 of runs
    ("Error during redisplay: (my-evil-fontify 1157) signaled (error
     \"fontification-functions cause glyph matrix reallocation;
     disabled\")" appears in *Messages*, Emacs is fine afterwards)
  - guard does NOT fire, no crash (freed block still readable): ~1/4
  - guard does NOT fire, SIGSEGV as before: ~1/2

Why the guard misses: on this code path the window's matrices are not
adjusted in place -- they are freed and new matrix structs are
allocated.  I put a logging breakpoint on adjust_glyph_matrix in the
patched build, printing MATRIX, its rows/rows_allocated on entry, and
the current value of window_desired_matrix.  After the evil
delete-other-windows, every adjust_glyph_matrix call comes in with a
fresh struct (rows == NULL, rows_allocated == 0):

  run where the guard MISSED and Emacs crashed:
    [adjust] matrix=0x80e1fc4d0 rows=0x0 alloc=0
window_desired_matrix=0x80efa2e60 MATCH=False
    [adjust] matrix=0x80e1fc540 rows=0x0 alloc=0
window_desired_matrix=0x80efa2e60 MATCH=False
    [adjust] matrix=0x80e1fc7e0 rows=0x0 alloc=0
window_desired_matrix=0x80efa2e60 MATCH=False
    [adjust] matrix=0x80e1fc850 rows=0x0 alloc=0
window_desired_matrix=0x80efa2e60 MATCH=False
    ... none of the new structs equals the old desired matrix; the old
    struct (0x80efa2e60) was freed; display_line crashes afterwards at
    xdisp.c:26855 (it->current_y += row->height) reading the freed row.

  run where the guard FIRED:
    [adjust] matrix=0xbd8faf870 rows=0x0 alloc=0
window_desired_matrix=0xbd8faf870 MATCH=True
    ... note rows == NULL and rows_allocated == 0: this is also a
    brand-new struct -- malloc just happened to reuse the freed old
    struct's address, so the pointer comparison matched by accident.

So `matrix == window_desired_matrix' compares against a pointer that
is dangling by the time the comparison runs; whether it matches is
heap-layout luck.  (I suspect the same was true when it "avoided the
trouble" on your machine.)

This also means the crashing scenario is slightly different from what
the patch assumes: the desired matrix is not enlarged behind
redisplay's back -- it is destroyed and replaced.  Perhaps the check
belongs where matrices are freed or detached from the window rather
than (only) in adjust_glyph_matrix: e.g. remember the window (or
compare w->desired_matrix against the matrix captured at
handle_fontified_prop time after the fontification call returns), or
error in free_glyph_matrix when it is called on window_desired_matrix.
The post-fontification check in handle_fontified_prop could then
detect "w->desired_matrix != saved pointer" and abort the window's
redisplay the way your patch already does.

Two smaller observations, in case they matter:

  - disable_fontification_functions_p is never reset, so one offense
    permanently disables fontification in that buffer (until it is
    killed).  Maybe intended as punishment, but a buffer whose hook
    misbehaved once (e.g. transiently, via some state in another
    library) stays unfontified with no way back that I can see short
    of killing it.

  - The crash backtrace in the guard-miss case is the same as in my
    original report: try_window (xdisp.c:21693) -> display_line
    reading the stale ROW right after the fontification call returns.

One more scope finding: fontification-functions is not the only
Lisp-during-window-redisplay entry point that can do this.  The same
evil call fired from a mode-line :eval form also crashes the patched
build (SIGSEGV inside display_mode_line <- display_mode_lines), and
window_desired_matrix is not set around mode-line evaluation, so the
new guard cannot see it at all:

;;; repro-modeline.el --- emacs -Q -l repro-modeline.el
(defvar my-armed nil)
(setq-default
 mode-line-format
 (list "%b "
       '(:eval (progn
                 (when my-armed
                   (setq my-armed nil)
                   (delete-other-windows))
                 ""))))
(defun my-go ()
  (switch-to-buffer (get-buffer-create "*top*"))
  (split-window-below)
  (select-window (next-window))        ; select bottom window
  (switch-to-buffer (get-buffer-create "*bottom*"))
  (run-at-time 0.5 nil (lambda ()
                         (setq my-armed t)
                         (force-mode-line-update t)
                         (redisplay t)
                         (kill-emacs 0))))
(add-hook 'window-setup-hook #'my-go)
;;; end

Here the top window's mode line is being displayed when the :eval form
deletes that very window, so the mode-line display continues with
freed matrices.  I mention it because a fix scoped to
fontification-functions would still leave this (and presumably
header-line/tab-line eval, window-scroll-functions, etc.) exposed;
whatever invariant the fix enforces probably wants to hold for any
Lisp called from inside the window-redisplay loop.

Happy to run further instrumented tests or try a revised patch.
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.