Re: Local treesit parsers with disjoint ranges

(Adding emacs-devel back to CC)

> On Jul 12, 2026, at 6:47 PM, Ruby Iris Juric <[email protected]> wrote:
> 
> Yuan Fu <[email protected]> writes:
> 
>>> On Jul 8, 2026, at 5:24 AM, Ruby Iris Juric <[email protected]> wrote:
>>> 
>>> This is the point where Emacs comes in. I've attached an image to this
>>> email that should help explain exactly what I'm trying to achieve. I
>>> want to allocate a local Typescript parser for each html_interpolation
>>> node, that covers all of it's direct permissible_text descendants, and
>>> excludes any other nodes that appear within, such as elements.
>> 
>> I feel like the astro parser took a shortcut that makes itself much
>> harder to use effectively. It makes much more sense for the astro
>> parser to also parse typescript. Embedded typescript seems like a very
>> commonly used pattern; having to embed external parsers for common
>> language features like this just makes everything harder. Imagine tsx
>> parser needs a html parser for handling html 😃 But anyway, range-fn
>> should work for your use-case, let’s figure out why it doesn’t.
> 
> I agree that it's a bit of a hack, but I think that's largely because
> tree-sitter doesn't offer a good solution for the recursive embedding
> use-case. I feel like the Astro parser shouldn't have to reimplement all
> of Typescript in it's parsing, just so it can add it's own syntax
> extensions to Typescript. I'm also not sure what a good solution here
> would look like, though.

It would be nice if tree-sitter lets you easily combine/import grammars. For “recursive embedding”, astro seems to be in the same situation as jsx? If it’s part of the language syntax, the grammar needs to handle it.

> 
>>> I would expect that two parsers are created for this situation, to match
>>> the attached image. However, it seems like in practice, only one parser
>>> is created to cover the innermost html_interpolation. I've attempted to
>>> debug the treesit.el code for managing local parser ranges, and it seems
>>> like the following is happening:
>>> 
>>> 1. treesit.el processes the range for the outermost html_interpolation
>>> 2. A Typescript parser is created that correctly covers both disjoint
>>>  ranges, along with a single overlay spanning the start of the first
>>>  range to the end of the last range (!)
>> 
>> At this point everything looks fine, the overlay is mainly used to
>> store the parser, so one parser <-> one overlay.
>> 
>>> 3. treesit.el proceeds to processing the range for the inner
>>>  html_interpolation
>>> 4. treesit.el determines that the given range is contained within the
>>>  overlay for the previously created parser (see
>>>  treesit--update-ranges-local), and opts to update the existing local
>>>  Typescript parser and overlay to match the new range, instead of
>>>  creating a new Typescript parser
>> 
>> This seems like where the confusion is, when treesit.el processes the
>> node one level deeper, it should’ve looked for a parser with embedded
>> level +1. It didn’t, because you aren’t using local parsers (:local
>> t).
> 
> I am using a local parser here[1] - note that I called out the behaviour in
> treesit--update-ranges-local, as opposed to
> treesit--update-ranges-non-local. Specifically, it's this snippet here
> that I think is the problem:
> 
>        (dolist (range ranges)
>          (let* ((r-start (treesit--range-start range))
>                 (r-end (treesit--range-end range))
>                 (existing-local-parser
>                  (catch 'done
>                    (dolist (ov (overlays-in r-start r-end) nil)
>                      ;; Update range of local parser.
>                      (when-let* ((embedded-parser
>                                   (overlay-get ov 'treesit-parser))
>                                  (parser-lang (treesit-parser-language
>                                                embedded-parser))
>                                  (parser-level (treesit-parser-embed-level
>                                                 embedded-parser)))
> 
> [...]
> 
>              ;; Create overlay and local parser.  Refer to
>              ;; (ref:local-parser-overlay) for more explanation of
>              ;; local parser overlays.
>              (let ((embedded-parser (treesit-parser-create
>                                      embedded-lang nil t 'embedded))
>                    (ov (make-overlay r-start r-end nil nil t)))
> 
> Assuming there's an existing local Typescript parser that covers the
> ranges ((10 . 20) (50 . 60)), r-start will be 10 and r-end will be 20,
> and an overlay for the range (10 . 60) will be created for it. If I now
> try to create a new parser for the range (30 . 40), (overlays-in 30 40)
> will return the previously created overlay for (10 . 60), since
> (30 . 40) is entirely within that range. Because the parser returned by
> (overlay-get ov 'treesit-parser) is the same language and embed level as
> the one being created, it reuses that previous parser instead of
> creating a new one.
> 
> Same embed level seems like it shouldn't be the case, but I suspect
> because we have to jump back to the host parser to handle the Astro
> expression that it parsed within the interpolation, the embed level is
> always 1, no matter how deep the nesting is, because any embedded
> Typescript parser will always be created as a direct descendent of the
> host Astro parser. So I agree that yeah, the parser is doing a bit of a
> hack here that makes it tricky to use. I guess the question now is
> whether we want to handle this case in treesit.el, or if we consider it
> a parser issue?

Ah, I see the problem. Yeah, the embed parser code is designed for parser embedding another parser, which embeds another parser. But here we want the same parser to create different levels of embedded parser. I need to think about whether we can support this kind of paradigm.

Yuan