emacs-31 6b670907fa9: Update tree-sitter manual for embed level

branch: emacs-31
commit 6b670907fa9040b58f54d6958b1f551a7519585b
Author: Yuan Fu <[email protected]>
Commit: Yuan Fu <[email protected]>

    Update tree-sitter manual for embed level
    
    * doc/lispref/parsing.texi (Using Parser): Mention embed level.
    (Multiple Languages): Explain embed level, and local vs
    non-local parsers.
---
 doc/lispref/parsing.texi | 131 +++++++++++++++++++++++++++++++++--------------
 src/treesit.c            |   1 -
 2 files changed, 93 insertions(+), 39 deletions(-)

diff --git a/doc/lispref/parsing.texi b/doc/lispref/parsing.texi
index 4da72b512b0..91680f89cd2 100644
--- a/doc/lispref/parsing.texi
+++ b/doc/lispref/parsing.texi
@@ -455,20 +455,19 @@ is non-@code{nil}, this function always creates a new parser.
 @code{nil}.  Different parsers can have the same tag.
 @end defun
 
-Given a parser, we can query information about it.
-
-@defun treesit-parser-buffer parser
-This function returns the buffer associated with @var{parser}.
-@end defun
-
-@defun treesit-parser-language parser
-This function returns the language used by @var{parser}.
-@end defun
-
-@defun treesit-parser-p object
-This function checks if @var{object} is a tree-sitter parser, and
-returns non-@code{nil} if it is, and @code{nil} otherwise.
-@end defun
+@findex treesit-parser-buffer
+@findex treesit-parser-language
+@findex treesit-parser-embed-level
+@findex treesit-parser-p
+Given a parser, we can query information about it:
+@code{treesit-parser-buffer} returns the buffer associated with a
+parser, and @code{treesit-parser-language} returns the language used by
+the parser.  A parser's tag can be retrieved by
+@code{treesit-parser-tag}.  Each parser also optionally carries a embed
+level that can be set by @code{treesit-parser-set-embed-level} and
+retrieved by @code{treesit-parser-embed-level} (for embed level,
+@pxref{parser-embed-level}).  Finally, @code{treesit-parser-p} tells you
+whether an object is a tree-sitter parser.
 
 There is no need to explicitly parse a buffer, because parsing is done
 automatically and lazily.  A parser only parses when a Lisp program
@@ -1808,10 +1807,11 @@ Again, @var{thing} can be either a symbol or a predicate.
 
 If @var{parser} is non-@code{nil}, only use that parser's parse tree.
 Otherwise try each parser covering point, from the most specific
-(deepest-embedded) to the least specific.  If there are multiple parsers with
-the same embed level at @var{position}, which parser is tried first is
-undefined.  If @var{parser} is a language symbol, the function limits
-the parsers it tries to the ones for that language.
+(deepest-embedded) to the least specific.  If there are multiple parsers
+with the same embed level (@pxref{parser-embed-level}) at
+@var{position}, which parser is tried first is undefined.  If
+@var{parser} is a language symbol, the function limits the parsers it
+tries to the ones for that language.
 @end defun
 
 @defun treesit-thing-next position thing &optional parser
@@ -1875,8 +1875,9 @@ position must be strictly greater than @var{position}.
 If @var{parser} is non-nil, only use that parser's parse tree.
 Otherwise try each parser covering point, from the most specific
 (deepest-embedded) to the least specific.  If there are multiple parsers
-with the same embed level at @var{position}, which parser is tried first is
-undefined.  @var{parser} can also be a language symbol.
+with the same embed level (@pxref{parser-embed-level}) at
+@var{position}, which parser is tried first is undefined.  @var{parser}
+can also be a language symbol.
 @end defun
 
 @findex treesit-beginning-of-thing
@@ -2032,11 +2033,12 @@ for nodes in a region.
 @defun treesit-language-at pos
 This function returns the language of the text at buffer position
 @var{pos}.  Under the hood it calls
-@code{treesit-language-at-point-function} and returns its return
-value.  If @code{treesit-language-at-point-function} is @code{nil},
-this function returns the language of the deepest parser by embed level
-among parsers returned by @code{treesit-parsers-at}.  If there is no
-parser at that buffer position, it returns @code{nil}.
+@code{treesit-language-at-point-function} and returns its return value.
+If @code{treesit-language-at-point-function} is @code{nil}, this
+function returns the language of the deepest parser by embed level
+(@pxref{parser-embed-level}) among parsers returned by
+@code{treesit-parsers-at}.  If there is no parser at that buffer
+position, it returns @code{nil}.
 @end defun
 
 @heading Supporting multiple languages in major modes
@@ -2153,10 +2155,38 @@ a @var{query} preceded by zero or more @var{keyword}/@var{value}
 pairs.  Each @var{query} is a tree-sitter query in either the string,
 s-expression, or compiled form, or a function.
 
+@code{treesit-update-ranges} uses @var{query} to figure out how to set
+the ranges for parsers for the embedded language.  It queries
+@var{query} in a host language parser, computes the ranges which the
+captured nodes span, and applies these ranges to embedded language
+parsers.
+
 If @var{query} is a tree-sitter query, it should be preceded by two
 @var{keyword}/@var{value} pairs, where the @code{:embed} keyword
-specifies the embedded language, and the @code{:host} keyword
-specifies the host language.
+specifies the embedded language, and the @code{:host} keyword specifies
+the host language. The captured nodes determines the range of embedded
+parsers. The capture name doesn't matter, but names that starts with an
+underscore are ignored, and the @code{@@language} capture name is
+reserved.
+
+If the embedded language is dynamic, then @code{:embed} can specify a
+function rathar than a static language. The function takes the node
+captured with @code{@@language} capturename. Care have to be taken that
+the @code{@@language} capture and the range capture are in the same
+match group. The function should return either a language symbol, or nil
+if no valid language can be found.
+
+If the query is given a @code{:offset} keyword whose value is a pair
+of numbers, each captured range is offset by those numbers.  For
+example, an offset of @code{(1 . -1)} will update a captured range of
+@code{(2 . 8)} to be @code{(3 . 7)}.  This can be used to exclude things
+like surrounding delimiters from being included in the range covered by
+an embedded parser.
+
+By default, a parser sees its ranges as a continuum, rather than
+treating them as separate independent segments.  Therefore, if the
+embedded ranges are semantically independent segments, they should be
+processed by local parsers, described below.
 
 @cindex local parser
 If the query is given the @code{:local} keyword whose value is
@@ -2164,19 +2194,44 @@ If the query is given the @code{:local} keyword whose value is
 otherwise the range shares a parser with other ranges for the same
 language.
 
-By default, a parser sees its ranges as a continuum, rather than
-treating them as separate independent segments.  Therefore, if the
-embedded ranges are semantically independent segments, they should be
-processed by local parsers, described below.
-
-Local parser set to a range can be retrieved by
+Local parsers set to a range can be retrieved by
 @code{treesit-local-parsers-at} and @code{treesit-local-parsers-on}.
 
-@code{treesit-update-ranges} uses @var{query} to figure out how to set
-the ranges for parsers for the embedded language.  It queries
-@var{query} in a host language parser, computes the ranges which the
-captured nodes span, and applies these ranges to embedded language
-parsers.
+Emacs treats local embedded parsers and non-local embedded parsers quite
+differently:
+
+@itemize @bullet
+@item
+Local parsers are managed by Emacs. Emacs create new local parsers when
+they are needed and deletes them when they are not.  Non-local parsers
+must be created by major modes, and Emacs only make use of them if one
+exists.
+
+@item
+Because local parsers can be freely created by Emacs, they are suitable
+for recursively nested embedded parsers, where the embed level can be
+arbitrarily deep.  On the other hand, since non-local parsers are major
+mode created, they can only handle a fix number of embed levels.
+@end itemize
+
+@anchor{parser-embed-level}
+@cindex embed level, tree-sitter
+The embed level of the top-level parsers is 0, or omitted, the first
+level of embedded parsers get embed level 1, and so on.  For local
+parsers, each embed level creates a new parser; for non-local parsers,
+since Emacs cannot create non-local parsers freely, it tries to find a
+usable parser at the right embed level, if none exists, Emacs just gives
+up.
+
+If the query is given a @code{:range-fn} keyword whose value is a
+function, Emacs uses that function to compute the ranges to use for the
+embedded parser.  The function takes captured node and the offset given
+by the @code{:offset} keyword, and returns a list of ranges, where each
+range is a cons of the start and end position. For local parsers, each
+@var{query} creates one local parser, and the ranges set by the
+@code{:range-fn} is applied to that parser; for non-local parsers, all
+the ranges (including ranges created by @code{:range-fn}) in that embed
+level are joined together and applied to the same parser.
 
 If @var{query} is a function, it doesn't need any @var{keyword} and
 @var{value} pair.  It should be a function that takes 2 arguments,
diff --git a/src/treesit.c b/src/treesit.c
index 55e6f8b94a1..7fb11048d18 100644
--- a/src/treesit.c
+++ b/src/treesit.c
@@ -2565,7 +2565,6 @@ embedding increments the embed level by 1.  */)
   return XTS_PARSER (parser)->embed_level;
 }
 
-/* TODO: Mention in manual, once the API stabilizes.  */
 DEFUN ("treesit-parser-set-embed-level",
        Ftreesit_parser_set_embed_level, Streesit_parser_set_embed_level,
        2, 2, 0,