Re: FreeMarker, ANTLR & New FreeMarker Template Notations [was: JetBrains / IntelliJ IDEA v8 to Support FreeMarker!]

Randall R Schulz wrote:
> On Wednesday 02 April 2008 15:41, Jonathan Revusky wrote:
>> Randall R Schulz wrote:
>> JavaCC pulls 
>> tokens off the token source on an as-needed basis. In JavaCC you can
>> even do something like token_source.setState(...) from within the
>> expression parser.
> 
> I consider that a far more defensible design, personally.
> 
> 
>> I use this in at least one point in the FreeMarker 
>> parser, for example, when the parser enters a macro invocation with a
>> set of named parameters, so the token source enters a different
>> (though only slightly different) lexical state. As best I can tell,
>> something like this is not even possible in Antlr. Of course, it
>> could be that it's a the wrong way to do it, a horrible hack, but
>> still, it strikes me as better than putting a separate
>> token-massaging step in between lexing and parsing that is not even
>> specified in the grammar file!
> 
> Well, real-world problems often get messy (and what FreeMarker must do 
> is probably one of the more challenging parsing tasks going).
> 
> I guess the thing to do is learn about their (ANTLR's) notion of "island 
> grammars" which allow you to mingle grammars... Or something like that.


I had a look at that, but it just seems like an enormous kludge to me.
At the bottom of the page here:

http://www.antlr.org/wiki/display/ANTLR3/Island+Grammars+Under+Parser+Control

TP writes:

"This does seem to be a pretty complex way of doing things, but it also 
seems to work."

It would seem that he himself recognizes that the Island Grammar thing 
is pretty horrible.

The thing I figured out finally is that, yes, you can simulate lexical 
states, you use something called a "gated semantic predicate."

EQUALS: {lexState == EXPRESSION}?=> '=';

And you define and set the lexState variable. The problem is that you 
have to put that boilerplate in front of every lexical rule to which it 
applies. So, here is a snippet of the grammar I have:

FALSE : {inExpression}?=> 'false';

TRUE : {inExpression}?=> 'true';

NULL : {inExpression}?=> 'null';

INTEGER : {inExpression}?=> ('0'..'9'+);

DECIMAL : {inExpression}?=> (INTEGER '.' INTEGER);

and another bunch of lines which all start the same way. In JavaCC which 
has the lexical states built in, you just go:

<EXPRESSION> TOKEN : {

...
}

and all the tokenizing rules defined in that block only apply to the 
EXPRESSION lexical state. And also, you can define blocks that apply to 
a set of different lexical states:

<EXPRESSION, INSIDE_PARENTHESIS> : {
...

}

The other big problem that I have never had in JavaCC is generating a 
lexer with methods with too large a code size, you know, the 64k 
limitation to the size of a method. I kept hitting this constraint and 
finally, scaled back my goal, so that it would only recognize square 
bracket syntax.

Another way in which ANTLR is vastly inferior to JavaCC is the quality 
of error messages it gives under various circumstances. I think this is 
mainly because in JavaCC, the snippets inside your actions is just java 
code, so it is validated as java code by the JavaCC parser and it tells 
you the error location in that file in those cases. In ANTLR, what is 
inside {....} is not validated, just passed through to the .java file 
which, if there is a problem, fails to compile, except you get an error 
message that provides a location that has nothing to do with where you 
made the mistake.

I have made significant progress on an ANTLR-based FTL grammar, but my 
tentative conclusion is that, pragmatically speaking, ANTLR is a pretty 
seriously inferior tool to JavaCC -- again, unless you want to target 
another language besides Java, in which case, JavaCC won't work for you.

Maybe I'll change my mind on this. I haven't given up completely on 
ANTLR: I'm going to finish an alternative FTL grammar and see how it 
works just for heck of it. I think I'm most of the ways there. It may be 
that as I learn more about ANTLR, I will see ways of simplifying the 
grammar and so on that I wasn't aware of. So far, it's pretty miserable 
compared to using JavaCC.

> 
>> I have to think the 161 subscribers to freemarker-user is a very
>> small proportion of the people who use the tool nowadays. OTOH, maybe
>> the docs are sufficient for most people and it is a fairly
>> straightforward thing to use anyway. A parser generator, no matter
>> how good the docs would have more people subscribing to the list, I
>> guess.
> 
> I don't have a feel for that, actually. I'd have to think there are more 
> uses for templating than for parsing.

Oh, I'm sure of that. I meant that, assuming you need a tool like JavaCC 
or ANTLR, it is far more likely that you would join the mailing list and 
ask questions and so on than for something like FreeMarker.

JR


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone