Re: {Spam?} Re: state in parsers
Kristof Bastiaensen <[email protected]> Sat, 21 Nov 2015 14:43:04 +0100
| Newsgroups | gmane.emacs.semantic |
|---|---|
| Message-ID | <[email protected]> |
Not quite. If parens are balanced inside the context, then no implicit
'}' is inserted. Also
other tokens may close the context, like "case {context}of", "let
{context}in",
"[expr, do {context}]", etc..., depending if the expression is inside
the context or not.
So I'd need to track every expression, basicly parsing it.
Here is how it is implemented in the ghc haskell compiler parser:
Parser.y.source, line 2938:
> -- Layout
>
> close :: { () }
> : vccurly { () } -- context popped in lexer.
> | error {% popContext }
The close terminal is either a vccurly (virtual '}') character, emitted
by the lexer when the indentation is less than the context, or if an
error occurs, the layout context is popped by the parser.
Can this be done in wisent?
Thanks,
Kristof Bastiaensen
On 21-11-15 03:29, Eric Ludlam wrote:
> Sorry for the delay in replying. Kids take up a lot of time.
>
> Based on your more recent example, I think your lexer can do the
> tracking. In this case, the lexer probably needs to track several things.
>
> 1) when it sees "do" it needs to create "do" and "{"
> 2) when it sees ")", it needs to close any pending "}"
> 3) it needs to track if it is in a "{" block, and convert indent into ;
>
> I think that is all possible in the lexer.
>
> Use semantic-lex-test to show a dump of what your lexer is outputting.
>
> Eric
>
> On 11/03/2015 10:57 AM, Kristof Bastiaensen wrote:
>> Ok, so let's say I emit a 'layout-next' token for implicit ';'. When
>> inside a layout,
>> this should be part of the grammar rule, and when outside the layout, it
>> should
>> be ignored (like whitespace). How do I track whether I am inside or
>> outside the layout?
>> Do I need to specify each rule, for example 'expression', twice, one for
>> inside, one for
>> outside? Or can I set a 'variable' to track this in the parser?
>>
>> Regards,
>> Kristof
>>
>> On 03-11-15 16:28, Eric Ludlam wrote:
>>>
>>> I am replying on my phone so I'll be a bit brief.
>>>
>>> I was providing python as an example of whitespace handling g so you would
>>> have a starting point. At a guess, it looks like your keywords like do or
>>> let also define some sort of indentation depth. The key for your lexer is
>>> to create tokens that the parser will use to reason on how it will
>>> interpret the system. Your parser would need to accept and use or ignore
>>> the special tokens depending on context. I don't think the lexer would
>>> convert whitespace into braces and ; though if it isn't completely
>>> deterministic from the lexical point of view. If you can track what you
>>> need through the lexer, then you could do the substitution. The c lexer
>>> handles preprocessor statements by tracking state while lexing so maybe
>>> that style will work for you.
>>>
>>> Eric
>>>
>>> On Nov 3, 2015 9:48 AM, "Kristof Bastiaensen" <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> messed the formatting up again :-( Sorry for the noise...
>>>
>>> Here is a better (contrived but real) example, that demonstrates how
>>> layout depends on the parser context:
>>> f x = length (
>>> do a <- [20]
>>> return (30)) + x
>>> * 2
>>>
>>> This translates to:
>>>
>>> f x = length (
>>> do {a <- [20]
>>> ;return (30)}) + x
>>> * 2
>>>
>>> Regards,
>>> Kristof Bastiaensen
>>>
>>>
>>> On 03-11-15 15:34, Kristof Bastiaensen wrote:
>>> > Actually the first problem can be solved at the lexer, by
>>> > remembering the indentation after the 'let', 'where', 'do' or 'of'
>>> > keyword. The second problem is harder, since where to ignore
>>> > insert ';' tokens would depend on the parser context.
>>> >
>>> > The mailer messed up the formatting, so I'll try again,
>>> hopefully it is
>>> > right now:
>>> >
>>> > let a = 20
>>> > b = 30
>>> >
>>> > and
>>> > let a = 20
>>> > b = 30
>>> >
>>> > are not the same thing. The first translates to
>>> >
>>> > let {a = 20
>>> > ;b = 30 }
>>> >
>>> > and the second to (the illegal)
>>> >
>>> > let {a = 20}
>>> > b = 30
>>> >
>>> > And for example the following
>>> >
>>> > let a = 20
>>> > b = 30
>>> > in a + b
>>> >
>>> > translates to
>>> >
>>> > let {a = 20
>>> > ;b = 30
>>> > }in a + b
>>> >
>>> > not to
>>> >
>>> > let {a = 30
>>> > ;b = 30
>>> > ;in a + b}
>>> >
>>> > Which is illegal. Where to insert ';' depends on the parser
>>> context,
>>> > it's not possible to do at the lex phase.
>>> >
>>> > Regards,
>>> > Kristof Bastiaensen
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On 03-11-15 15:21, Kristof Bastiaensen wrote:
>>> >> Hi,
>>> >>
>>> >> the problem in haskell is that simply specifying indent is not
>>> enough,
>>> >> for example:
>>> >>
>>> >> let a = 20
>>> >> b = 30
>>> >>
>>> >> and
>>> >> let a = 20
>>> >> b = 30
>>> >>
>>> >> are not the same thing. The first translates to
>>> >>
>>> >> let {a = 20
>>> >> ;b = 30 }
>>> >>
>>> >> and the second to (the illegal)
>>> >>
>>> >> let {a = 20}
>>> >> b = 30
>>> >>
>>> >> And for example the following
>>> >>
>>> >> let a = 20
>>> >> b = 30
>>> >> in a + b
>>> >>
>>> >> translates to
>>> >>
>>> >> let {a = 20
>>> >> ;b = 30
>>> >> }in a + b
>>> >>
>>> >> not to
>>> >>
>>> >> let {a = 30
>>> >> ;b = 30
>>> >> ;in a + b}
>>> >>
>>> >> Which is illegal. Where to insert ';' depends on the parser
>>> really,
>>> >> it's hard to do at the lex phase.
>>> >>
>>> >> Regards,
>>> >> Kristof Bastiaensen
>>> >>
>>> >>
>>> >>
>>> >> On 03-11-15 13:22, Eric Ludlam wrote:
>>> >>> On 11/02/2015 07:13 AM, Kristof Bastiaensen wrote:
>>> >>>> Hi,
>>> >>>>
>>> >>>> I'd like to add support for haskell to semantic. Haskell
>>> syntax is
>>> >>>> sensitive
>>> >>>> to indentation, and I'd like to know if it is possible to use
>>> bovine or
>>> >>>> wisent for
>>> >>>> it.
>>> >>> Yes.
>>> >>>
>>> >>>> The layout rule is explained here:
>>> >>>>
>>> https://www.haskell.org/onlinereport/haskell2010/haskellch2.html#x7-210002.7
>>> >>>>
>>> >>>>
>>> >>>> What it does, is after certain keywords, whenever a '{'
>>> character is
>>> >>>> ommited, it
>>> >>>> enters a layout context. In the layout context it will
>>> remember the
>>> >>>> indentation
>>> >>>> of the next token. When a line has the same indentation as
>>> this token,
>>> >>>> a ';' token is
>>> >>>> inserted before the expression. Whenever the indentation is
>>> less than
>>> >>>> this, a '}' token
>>> >>>> is inserted, and the layout context is ended. The layout
>>> context is
>>> >>>> also ended when a parser error would occur.
>>> >>> The python grammar in semantic/wisent/python.wy and python.el
>>> is an ok
>>> >>> example. The premise is that when you develop the lexer, you
>>> teach it
>>> >>> to convert your indentation into lexical tokens that you then
>>> use in
>>> >>> your grammar.
>>> >>>
>>> >>> In the python.el support file, you will notice it uses
>>> >>> (current-indentation) and uses that to derive INDENT and DEDENT
>>> >>> tokens. The grammar uses those to create INDENT and DEDENT
>>> blocks so
>>> >>> it can recurse into code bodies. You could just as easily
>>> skip them
>>> >>> too if you prefer.
>>> >>>
>>> >>>> So I would set a state variable to all the layout context
>>> indentations,
>>> >>>> and then change the behaviour of the lexer or parser based on
>>> this
>>> >>>> state. Is that possible? Would it interfere with
>>> incremental parsing?
>>> >>> Definitely start with the lexer, and there is good
>>> documentation on
>>> >>> creating a lexer in the semantic language developers guide.
>>> Use the
>>> >>> lexer testing function to make sure it outputs what you
>>> expect, and
>>> >>> then you can continue on with the grammar.
>>> >>>
>>> >>> Eric
>>> >>
>>> ------------------------------------------------------------------------------
>>> >> _______________________________________________
>>> >> cedet-semantic mailing list
>>> >> [email protected]
>>> <mailto:[email protected]>
>>> >> https://lists.sourceforge.net/lists/listinfo/cedet-semantic
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > _______________________________________________
>>> > cedet-semantic mailing list
>>> > [email protected]
>>> <mailto:[email protected]>
>>> > https://lists.sourceforge.net/lists/listinfo/cedet-semantic
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> cedet-semantic mailing list
>>> [email protected]
>>> <mailto:[email protected]>
>>> https://lists.sourceforge.net/lists/listinfo/cedet-semantic
>>>
>>
>> ------------------------------------------------------------------------------
>>
>>
>>
>> _______________________________________________
>> cedet-semantic mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/cedet-semantic
>>
> ------------------------------------------------------------------------------
> _______________________________________________
> cedet-semantic mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/cedet-semantic
------------------------------------------------------------------------------