Re: Local treesit parsers with disjoint ranges

[apologies for anyone who got two copies of this - i'm only just now
realizing i did not reply all initially, whoops!]

Yuan Fu <[email protected]> writes:

>> On Jul 8, 2026, at 5:24 AM, Ruby Iris Juric <[email protected]> wrote:
>>
>> This is the point where Emacs comes in. I've attached an image to this
>> email that should help explain exactly what I'm trying to achieve. I
>> want to allocate a local Typescript parser for each html_interpolation
>> node, that covers all of it's direct permissible_text descendants, and
>> excludes any other nodes that appear within, such as elements.
>
> I feel like the astro parser took a shortcut that makes itself much
> harder to use effectively. It makes much more sense for the astro
> parser to also parse typescript. Embedded typescript seems like a very
> commonly used pattern; having to embed external parsers for common
> language features like this just makes everything harder. Imagine tsx
> parser needs a html parser for handling html 😃 But anyway, range-fn
> should work for your use-case, let’s figure out why it doesn’t.

I agree that it's a bit of a hack, but I think that's largely because
tree-sitter doesn't offer a good solution for the recursive embedding
use-case. I feel like the Astro parser shouldn't have to reimplement all
of Typescript in it's parsing, just so it can add it's own syntax
extensions to Typescript. I'm also not sure what a good solution here
would look like, though.

>> I would expect that two parsers are created for this situation, to match
>> the attached image. However, it seems like in practice, only one parser
>> is created to cover the innermost html_interpolation. I've attempted to
>> debug the treesit.el code for managing local parser ranges, and it seems
>> like the following is happening:
>>
>> 1. treesit.el processes the range for the outermost html_interpolation
>> 2. A Typescript parser is created that correctly covers both disjoint
>>   ranges, along with a single overlay spanning the start of the first
>>   range to the end of the last range (!)
>
> At this point everything looks fine, the overlay is mainly used to
> store the parser, so one parser <-> one overlay.
>
>> 3. treesit.el proceeds to processing the range for the inner
>>   html_interpolation
>> 4. treesit.el determines that the given range is contained within the
>>   overlay for the previously created parser (see
>>   treesit--update-ranges-local), and opts to update the existing local
>>   Typescript parser and overlay to match the new range, instead of
>>   creating a new Typescript parser
>
> This seems like where the confusion is, when treesit.el processes the
> node one level deeper, it should’ve looked for a parser with embedded
> level +1. It didn’t, because you aren’t using local parsers (:local
> t).

I am using a local parser here[1] - note that I called out the behaviour in
treesit--update-ranges-local, as opposed to
treesit--update-ranges-non-local. Specifically, it's this snippet here
that I think is the problem:

        (dolist (range ranges)
          (let* ((r-start (treesit--range-start range))
                 (r-end (treesit--range-end range))
                 (existing-local-parser
                  (catch 'done
                    (dolist (ov (overlays-in r-start r-end) nil)
                      ;; Update range of local parser.
                      (when-let* ((embedded-parser
                                   (overlay-get ov 'treesit-parser))
                                  (parser-lang (treesit-parser-language
                                                embedded-parser))
                                  (parser-level (treesit-parser-embed-level
                                                 embedded-parser)))

[...]

              ;; Create overlay and local parser.  Refer to
              ;; (ref:local-parser-overlay) for more explanation of
              ;; local parser overlays.
              (let ((embedded-parser (treesit-parser-create
                                      embedded-lang nil t 'embedded))
                    (ov (make-overlay r-start r-end nil nil t)))

Assuming there's an existing local Typescript parser that covers the
ranges ((10 . 20) (50 . 60)), r-start will be 10 and r-end will be 20,
and an overlay for the range (10 . 60) will be created for it. If I now
try to create a new parser for the range (30 . 40), (overlays-in 30 40)
will return the previously created overlay for (10 . 60), since
(30 . 40) is entirely within that range. Because the parser returned by
(overlay-get ov 'treesit-parser) is the same language and embed level as
the one being created, it reuses that previous parser instead of
creating a new one.

Same embed level seems like it shouldn't be the case, but I suspect
because we have to jump back to the host parser to handle the Astro
expression that it parsed within the interpolation, the embed level is
always 1, no matter how deep the nesting is, because any embedded
Typescript parser will always be created as a direct descendent of the
host Astro parser. So I agree that yeah, the parser is doing a bit of a
hack here that makes it tricky to use. I guess the question now is
whether we want to handle this case in treesit.el, or if we consider it
a parser issue?

-- ruby

[1]: https://git.isincredibly.gay/srxl/astro-ts-mode/src/branch/range-fn-for-interp/astro-ts-mode.el#L155-L159