bug#81623: 32.0.50; Long-line optimization severely slows redisplay of invisible newlines

> From: Scott Guest <[email protected]>
> Date: Sat, 15 Aug 2026 14:16:49 -0700
> Cc: [email protected]
> 
> Thanks for the reply. I see the argument that this isn't an outright
> bug, and I understand that raising or disabling `long-line-threshold`
> is a possible workaround.
> 
> My concerns, however, are that:
> 
> - The limitation that there are no significant invisible regions is
>   not documented (to my knowledge). "Significant portions" is also
>   relative: the cost depends on the absolute number of hidden
>   newlines, so the problematic region need only be a small relative
>   portion of a larger buffer.

It is hard to provide useful and yet accurate description of these
aspects, because there are too many specific cases.  The limit is
exposed to Lisp because one-fits-all solutions are impossible.  In
each case where the limit produces unacceptable results, one should
look for a better value by trial and error.

> - The long line in the reproducer is not itself invisible and is
>   off-screen during the timed redisplay. A non-invisible but
>   off-screen long line enables a per-buffer optimization flag which
>   then causes pathological work in a separate invisible region. This
>   feels quite surprising.

This is intended, because the causes of slow redisplay due to long
lines happen also when the long lines are outside of the viewport.
This is due to how the current Emacs display engine works: it many
times scans lines before and after the window, especially when looking
for the current place to start a window's display.

> - Once a buffer has been recognized as containing a long line,
>   `long_line_optimizations_p` is never cleared. Changing
>   `long-line-threshold` afterward therefore does not disable the
>   optimizations in that buffer; the workaround must be applied before
>   the first redisplay that detects an over-threshold line, which may
>   be much earlier than the pathological redisplay.

Killing the buffer and revisiting the file is the usual solution.

The flag is never reset because it is very hard to know when the
conditions are no longer applicable.  We could have a function to
reset the flag, but it was never requested, and it could be dangerous
in quite a few cases.

> For my use case, visible long lines are quite frequent, while this
> pathological case is fairly rare (but still just frequent enough to
> cause issues). It would be unfortunate to lose the optimization's
> benefit for the common case because of this rarer pathological case.

You can just enlarge the threshold, instead of setting it to nil.

> The redisplay time with the optimization enabled grows roughly
> linearly with the number of invisible lines.

That's expected.  Making large portions of a buffer invisible is a bad
idea in Emacs, at least with the current display engine, because most
algorithms don't take this case into consideration.

> The real input where I encountered this had a few MiB of such lines,
> which made the slowdown severe enough that redisplay appeared to
> hang.

I wonder what is the reason for hiding so many lines.  Why not leave
them displayed instead?

> More importantly, this appears to be caused by avoidable, redundant
> work. [Disclaimer: the analysis below was developed with LLM
> assistance, then checked against the source and verified with an
> instrumented build. I may still have missed something.]
> 
> Redisplay calls `back_to_previous_visible_line_start`, which walks
> backward over all 40,000 invisible newlines. After the first step,
> each iteration already begins at BOL, immediately after the newline
> ending the line just traversed. (This occurs regardless of whether that
> line was empty or contained text; the reproducer has bare newlines
> solely for minimality.) With long-line optimization enabled, it
> nevertheless calls `get_nearby_bol_pos` to find that BOL again.

Why is that a problem in this case?  get_nearby_bol_pos will be called
regardless of whether there is or isn't invisible text, so how is it
the cause for slowdown when there are many invisible lines?

In any case, are we supposed to have optimizations for specific cases
of buffers with many empty lines hidden by the invisible property?
There will be no end to such special cases.

> For almost all iterations in this reproducer, each call scans a
> 500-character chunk and invokes `find_newline1` about 500
> times. Inserting counters around this path, each full backward
> traversal made 39,753,460 calls to `find_newline1`. The timed
> redisplay invoked that expensive traversal three times. The non-ASCII
> character is not required for these redundant scans; it only amplifies
> the associated byte/character conversion cost.

find_newline1 is very fast, it calls memchr to search for a newline.
I fail to understand how it could be a problem here.

> A fast path returning `pos` when the preceding byte is a newline
> preserves the result of the existing scan and eliminates the slowdown
> in this reproducer. [I have omitted the concrete patch since it was
> generated by an LLM.]

What happens if the buffer is narrowed, and the BOL we are at is
outside of the narrowed region?

Also, the job of get_nearby_bol_pos is to find a BOL position _before_
the given position, so being at BOL does not help, AFAIU.

Bottom line: given that a solution by enlarging the threshold exists
and is very simple, I'm reluctant to make significant changes in this
mechanism for the benefit of a rare use case.  Any such change will
need to be tested against all the cases we had back when this feature
was implemented (there's a bug in the bug database about it, which
should describe at least the moist frequent cases).