Re: How do I return more than one tag from production?

On 02/19/2016 01:36 PM, Gracjan Polak wrote:
> Hi all,
>
> Due to how reparsing interacts with Haskell block structure it may
> happen that after reparsing a block that had one function definition now
> will two of those. For example, before:
>
> f = x
>    where
>      z = w
>      k = p
>
> After edit:
>
> f = x
>    where
>      z = w
> k = p
>
> So now there are two functions (f and k), but only one block is
> reparsed. How to create a rule that covers this case?

Hi Gracjan,

You have a lot of questions that didn't get answered, so I'm not sure 
where you are at this time, so I'll try to cover a few things.  I'll use 
C as an example since that is what I'm most familiar with.

In C, if you have a code snippet such as:

int a;

int b;

int c;

and you change "int b" into "int bob", the incremental parser will find 
the full extents of the previous tag "int b" which were laid out with 
buffer markers, which means they move when you edit.  It then expands 
out to just after "int a;" and just before "int c" to capture any edits 
that might include white-space, comments, etc.  It then looks into the 
tag scope of b to see what the reparse token is.  It will be empty in 
this case, so it kicks of a lex/parse starting with your %start token.

It takes however many tags it finds (0, 1, or more) and splices that 
list back into the original tag list.  Thus, if you edit in multiple 
tokens, it will splice them all in for you.

If the lex/parse step fails (ie - unterminated string or block) then it 
assumes you aren't done editing, and does nothing, and kicks of a 
reparse again later.  Sometimes it might throw up it's hands and dirty 
the whole buffer and start from scratch.

If you instead have:

struct foo {
   int a;

   int b;

   int c;
};

and do the same operation, when the lex/reparse occurs, it discovers the 
parent token is for "foo", and kicks off the parse using the nonterminal 
saved in "int b" and restricts other operations to within the bounds of 
foo.  If the edit block of 'foo' is exceeded, then all of "foo" is 
reparsed using the top %start token.


For all this to work, the structure of the language is one where 
"EXPANDFULL" is used to recurse into the parse on a section of text that 
is itself meant to create a list of tags whose 'parent' is another tag. 
  ie - you need to implement "semantic-tag-components" for your language 
and teach it about each type of tag that is a parent of other tags, and 
the parsing of those parent tags should be calling EXPANDFULL to create 
those tags.

It should not be necessary to override the various parts of the 
incremental parser. In the past I thought that might be useful, but 
AFAIK, all languages so far conform to the structure it uses, so it has 
not be necessary.

You had a different question regarding INDENT_BLOCK, which you would 
probably find useful.  In your grammar you need to define tokens for 
INDENT and DEDENT, and then you define a block token for INDENT_BLOCK 
that indicates that the block starts with INDENT and ends with DEDENT. 
This needs to work with the lexer.  For langauges using single character 
block delimiters like { and }, you can use the built-in 
semantic-lex-paren-or-list when you define the parser.  It uses the 
syntax table to find \s( as a match, and uses "forward-list" to create 
the BLOCK token.  It looks at the lexer depth flags to decide if it 
instead needs to create the { token instead.

In Python, the DEDENT token is not part of the syntax table, so you will 
see that wisent-python-lex-beginning-of-line will try to determine what 
kind of indentation it is, and convert that into either a DEDENT (see 
comment "indentation increased"), whitespace (see comment "no change in 
indentation"), or INDENT_BLOCK token (see comment "add INDENT_BLOCK 
token".  In that way, all beginning of lines are handled the same way, 
but different kind of lexical tokens are emitted.

I hope this helps.
Eric


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140