Re: Two minor bugfixes

Hi Eric,

>> Second bug:
>> File: 0001-Save-global-variable-wisent-lookahead-before-calling.patch
>>
>> The wisent parser uses some global variables, which is a bit of a hassle
>> considering that it calls itself (via EXPANDFULL and the like). If the
>> last token of a recursive call does not match, it can corrupt the
>> unmatched-syntax-cache or even lead to wrong parse results. If recursive
>> calls of the parser are only done with parenthesised blocks and rules
>> are accordingly, then the last token must be something like ')' and thus
>> always matches and the bug is not triggered. For languages like fortran
>> this is different.
>> I am very sure that the commit indeed fixes a bug and nothing more, but
>> the wisent parser and its token stream organisation is a rather complex
>> automaton, so I might have missed something.
> This is a great find.  Thanks.  I've installed it here and it passes the
> test suites.  Was there an example that exposed this that is short you
> can share?  Can the bug be exposed using some existing language?  It
> would be nice if I could add a test for this.
With my fortran grammar the followng short exampe produces the error:

subroutine test(arg)
    call something()
end subroutine test

The token sequence produced by the lexer looks like
((SUBROUTINE 1 . 11)
  (NAME 12 . 16)
  (PAREN_BLOCK 16 . 21)
  (NEWLINE 21 . 22)
  (PROG_FUNC_SUBR_BODY 22 . 42)
  (END 42 . 45)
  (SUBROUTINE 46 . 56)
  (NAME 57 . 61)
  (NEWLINE 61 . 62))

PROG_FUNC_SUBR_BODY is something like the semantic-list in bovine. It
always needs to be expanded (via an expandfull-like macro with depth 0), 
as types of arguments to subroutine (not present here) are declared
inside this block (unlike C, where the type is inside of the PAREN_BLOCK 
token). As "call something()" is not a variable declaration, it fails to 
match anything. The last token is a NEWLINE at 41.42, which then gets 
wrongly reinjected, fails once more at the top level goal and ends up on 
the unmatched syntax cache.
If I add a rule for NEWLINE in the top-level goal
func-subr-mod-prog
   : one-function
   | one-subroutine
   | one-module
   | one-program
   | NEWLINE
     (TAG "error" 'error :region $region1)
   ;
I indeed get the error tag ("error" error (:region (41 . 42)) ...),
which clearly is a parse error, as there is no such newline at this level.

You might add this example once the fortran wisent grammar is included
in cedet. For some reason I cannot produce this with python or some
other language, even with some manipulations in the grammar. However,
the error recovery in wisent and some other issues makes it hard to
predict anything. In fact, I gave up to have a correct 
unmatch-syntax-cache with wisent, as there are some other issues, though 
I would like to come back to that (I like the unmatched syntax
highlighting I saw with some C/Qt Editors, thats helpful, but requires a 
more complete grammar anyway).

BTW: Is there any reason not to declare a wisent-parser class, with
slots for all those variables. All those indirections via global
variables (like with wisent-lexer-function) looks crazy. Clearly one
reason would be that its much work for not much gain.

Martin


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=190641631&iu=/4140/ostg.clktrk