bug#68861: 30.0.50; ERC 5.x: Introduce a modern message-insertion API

"J.P." <[email protected]> writes:

> Severity: wishlist
>
> Some recent ramblings on this topic (by me) were crudely tacked on to
> the end of the now closed bug#67677.

This bug is also closely related to what I've been calling "part two" of
bug#49860, which aims to add IRCv3 features that basically require not
only insertions away from prompt but also random splicing, deleting, and
refreshing of those inserted messages. These are all currently foreign
concepts to ERC. I'd say the surest route to attaining them is by
revamping ERC's insertion handling so it's based on message objects. By
this I mean organizing a message's makeup into a structured and
persistent constellation of individual ingredients to be rendered in an
elegant and straightforward manner. This should make influencing
everything from portions of messages to spans of multiple messages more
predictable, extensible, and maintainable.

> In summary, the solution originally delivered by that bug of having
> all chat messages formatted by literal `format-spec' templates was
> left wanting in terms of an easily exposable and extendable public
> API.

Unfortunately, what I'm proposing here will only prolong that deficit in
terms of moving first to a solely internal API to be trialed and
improved upon indefinitely or until complaints arise. To that end, I
think we ought to introduce this as two separate but related areas in
the code base:

 - An optional library and module providing the underlying "refreshable"
   insertion framework, which new code can depend on directly

 - An interface for existing code to alter its behavior when refreshable
   insertion is in effect

This two-pronged approach at least superficially favors caution in the
name of preserving compatibility with existing interfaces, like
`erc-insert-modify-hook', and all the behavioral nuances baked into them
[1]. In the end, though, this change is quite radical in that it upends
fundamental assumptions underpinning the client's design: no longer will
ERC be presenting an append-only log to users but instead a dynamic
"palimpsest" of sorts that can, in a very general sense, even be likened
to a graphical canvas.

>                              Instead, I think it'd be preferable to take
> a more traditional tack and preserve the various constituent parts of a
> template as separate nodes in a tree-like structure for deferred
> assembly just prior to formatting, perhaps in conjunction with some
> caching mechanism. The literal, propertized base template would still
> feature prominently as part of a template class's definition, but its
> role would be reduced to serving as input for a validation scheme.

I've taken a stab at something loosely resembling the approach
described, although most existing template-based messages won't be
decomposed into meaningful hierarchies beyond simplistic partitioning
into {head} and {body} components. However, those with a speaker-based
{head} could conceivably be broken down further into:

   __________________{head}__________________
  /                                          \
  {left-bracket}{speaker}{right-bracket}{space}

The reason they're not expanded to that level of granularity at the
moment is because I want to resist introducing contiguous but discrete
intervals of common text properties, which can slow down scanning. Such
entropy can be mitigated somewhat by consolidating them into a
`category' text property, although there are compatibility trade offs
with that as well.

> In terms of proposals, I think a successful candidate for adoption
> should provide at least two working demos, preferably among the
> following:
>
>   - Bidirectional input and display
>   - Headline style messages w. speaker on a separate line
>   - Text substitution (language translation, encryption, etc.)
>   - Display names (e.g., for bridges, perhaps reversible)

(To this list, I'd add a top-down layout, where the prompt sits atop the
buffer and new messages spawn just below it.)

Anyway, here is a simplified sketch of the proposed design (full PoC
available soon):


Message objects
===============

Each message is represented by an instance of a struct-based type that
marries a hierarchical model of arbitrary message subdivisions (called
"divs") with attributes derived from common `erc--msg-props'.
Traditional modules that don't want to `require' the library can
interact with message objects before, during, and after insertion via a
"refreshable" compatibility API defined in erc-common.

A div tree's root is represented by a function assigned to the message
object's "div" slot. Its job is to insert the entire contents, including
any bounding newlines. It does this either by deferring to a list of
similar "child" div functions, implemented as a familiar Lisp hook, or
by inserting text directly, or both. A special wrapper that runs the
hook ensures its members (children) are inserted depth-first, with
smaller hook depths corresponding to earlier buffer positions. So, a div
is in fact more than a function: it depends on supporting properties as
well.

A div function's own behavior can be modified by normal Lisp advice,
which, like its hook members, is added via the help of an associative
labeling scheme and a dedicated helper utility. Advice can also be
applied to its children in this same manner as well as their children,
and so on. Once again, for the sake of clarity: each div consists of an
insertion function, a label, a hook, and an advice stack, with the
latter two only defined as needed.

For example, a div tree for a foldable "erc-multiline--code" message
displaying a block of syntax-highlighted source code might look like:

  - :label "root"
    :inserter #'erc-div--insert-children
    :advice [:after #'erc-read-only]
    :hook
     - :label "above"
       :inserter #'erc-div--insert-hard-newline

     - :label "head" :inserter #'erc-div--insert-speaker
       :advice
        - :override #'erc-gravitar--iconify
        - :after #'erc-fill--wrap-add-indent-prefix

     - :label "body"
       :inserter #'erc-div--insert-children
       :advice [:after #'erc-fill--wrap-add-wrap-prefix]
       :hook
        - :label "multiline-bookend-beg"
          :inserter #'erc-multiline--insert-bookend-beg

        - :label "body/inner"
          :inserter #'erc-div--insert-body-from-backing-store
          :advice
           - :after #'erc-multiline--code-fontify-by-major-mode
           - :after #'erc-multiline--add-folding-overlay

        - :label "multiline-bookend-end"
          :inserter #'erc-multiline--insert-bookend-end

     - :label "tail" :inserter #'ignore
     - :label "below" :inserter #'ignore

While a div's shape is arbitrary, the three "well known" message types
that serve as adapters for traditional template-based messages all
subdivide their "root" div into top-level divs named "above", "head",
"body", "tail", and "below". Thus, a module that has no interest in
defining its own message types or integrating with the framework at all,
can still piggyback on other div-based messages using more abstract,
albeit less convenient, hooks and advice. For example:

  (add-hook 'erc--refreshable-modify-div-functions
            (erc--refreshable-modify-div-functions
             #'erc--make-invisible :after 40 'head 'tail))

The above adds the function `erc--make-invisible' as advice that runs
after the "head" or "tail" div has inserted its contents in any buffer.

  (add-hook 'erc--refreshable-modify-div-functions
            (erc--refreshable-modify-div-functions
             #'erc-foo--insert-right 'foo-right 10 'head)
            0 'localp)

This example instead adds a *new* div called "foo-right" as a child of
"head" that runs after the latter's own insertion code because the depth
is positive; a -10 would have it run before. Each generated root div is
cached with a composite key composed of relevant factors available just
before insertion, such as the default and buffer-local values of
`erc--refreshable-modify-div-functions'.


"Mutable" buffers
=================

A buffer's contents are modeled as an ERC-flavored Ewoc with a few minor
modifications. Modules that manage state between messages, such as those
providing unique speaker tags or timestamps, or those needing to adapt
to splices and deletions for other reasons can do so by providing
handler functions conforming to local-advice-based interfaces defined as
function-valued variables. A handler can react to changes by requesting
that arbitrary nodes be deleted or refreshed. These actions can cascade,
so care must be taken to avoid cycles:

  (defun erc-foo--handle-splice (i n inst prev next)
    "Consider INST, the Ith message object currently being inserted.
  Weigh it in the context of neighbors PREV and NEXT, the I-1th and
  I+1th (of N total) messages in the sequence, respectively. If INST is
  the last of its sequence and has an `erc-foo' property, schedule NEXT
  for refreshing if it too has one."
    (when (and prev
               next
               (= i n)
               (erc--refreshable-prop-get inst 'erc-foo)
               (erc--refreshable-prop-get next 'erc-foo))
      (list next)))

The function above would ostensibly belong to an older `erc-foo' module
that doesn't directly depend on the library providing the framework.
Instead, it treats params INST and co as opaque object handles, and it
uses the abstract "refreshable" interface to interact with them. By
contrast, a "native" module built directly atop the framework would
interact with such objects directly, through their slot accessors and
various library utilities. The primary reason for maintaining this
designation and the consequent indirection is to avoid foisting a rather
massive library on users who only ever use traditional modules.

For sequential insertions, such as history playback, modules may need to
maintain state apart from what's normally reserved for "live" arrivals
inserted at prompt. Otherwise, important data, like that contained in
`erc-server-users', will be munged. For this shadowing to work, a module
provides a piece of local :around advice invoking a special macro:

  (defun erc-foo--wrap-state (inner)
    (erc--refreshable-with-locals
        ((erc-foo-var1 (erc-foo-init 1))  ; value form runs once
         (erc-foo-var2 (erc-foo-init 2)))
      (funcall inner)                     ; defer to insertion code
      (erc-foo-teardown)))                ; clean up resources

The macro captures and restores the values of the `let'-style VARLIST
before and after insertion for every message in the sequence.


The WIP framework currently addresses both concerns outlined above, that
is, both message objects and mutable buffers, as a single module in a
single library. It may be prudent to split these up in into separate
files, although I'd rather preserve the library-module correspondence if
at all possible. That said, the library is quite large, at around 1,700
lines, with around half being compatibility oriented.

Additional details to follow at some point. Thanks for reading.


[1] Some compatibility related limitations of the current version:

    - When redrawing an inserted message, it currently only restores
      the `erc-parsed' text property for commands backed by a data
      store, since these sustain enough info to reconstruct an
      `erc-response' object. Members of traditional insertion hooks
      that rely on `erc-get-parsed-vector' for other message types,
      like a 353, must adapt to the property's absence.

    - It discards the trailing newline normally retained in the narrowed
      buffer after `erc-insert-modify-hook' and friends have visited.
      Thus, any markers left at `point-max' by a hook member no longer
      sit between a newline and the start of a message but between the
      end of a message and a newline.

    - The value of the quasi-internal `erc--msg' text property is now a
      message object instance, and it may appear at a later position
      relative to the start of a message because it now demarcates the
      boundary between any preceding newlines and the logical contents.
      Some utilities, like `erc--get-inserted-msg-prop', have been made
      aware of this difference.