Re: FreeMarker, ANTLR & New FreeMarker Template Notations [was: JetBrains / IntelliJ IDEA v8 to Support FreeMarker!]

Randall R Schulz wrote:

>>>  I'm not sure. I wonder if that goes back to my writing a TPTP
>>> parser using ANTLR 2.x or my more recent creation of a CLIF parser
>>> using ANTLR 3.0. Could you send me a link to the mail archive you
>>> found?
>> http://www.antlr.org:8080/pipermail/antlr-interest/2006-November/0184
>> 05.html
> 
> That would probably correspond to my work on a TPTP parser using ANTLR 
> 2.7. I completed that without too much trouble. I think I may have been 
> asking about lexical states so I could add a kind of documentation 
> comment capability. In my vague recollection, I ended up with a kind of 
> simple state machine overlayed on top of the lexer output so I could 
> associate the proper preceding documentation comment text with the 
> non-comment statements that followed.

The problem with what you're doing there, if I understand it right, is 
that you're  massaging the token stream in a certain way between the 
lexing and parsing steps and this is not at all specified in your 
grammar file, right?

OTOH, I guess that does work since Antlr reads in all the tokens before 
doing any expression parsing. That surprised me, as it strikes me as far 
less flexible than JavaCC in this regard. JavaCC pulls tokens off the 
token source on an as-needed basis. In JavaCC you can even do something 
like token_source.setState(...) from within the expression parser. I use 
this in at least one point in the FreeMarker parser, for example, when 
the parser enters a macro invocation with a set of named parameters, so 
the token source enters a different (though only slightly different) 
lexical state. As best I can tell, something like this is not even 
possible in Antlr. Of course, it could be that it's a the wrong way to 
do it, a horrible hack, but still, it strikes me as better than putting 
a separate token-massaging step in between lexing and parsing that is 
not even specified in the grammar file!

>>>  (I don't want to start a big flame-fest between partisans, but I
>>> find
>> The problem with ANTLR is not exactly a *lack* of docs. Terence,
>> along with collaborators, have written a fair bit of verbiage on it.
>> However, it's a totally disorganized mass, some of it is simply
>> utterly dense and impossible to understand. Other parts of it are
>> downright misleading.
> 
> This is, in my experience, the biggest criticism that can be generally 
> leveled against open-source projects. So many of them, while valuable 
> as software artifacts, are quite poorly documented. (But I'll add, 
> quite sincerely, that FreeMarker is a welcome exception to this.) I'm 
> using Grails for a Web Interface, and it suffers from exactly the same 
> problem. No reference manual, disorganized, incomplete documentation, 
> etc. The only way to successfully use it is to rely on the collective 
> understanding of the principals and users via its mailing list.

I'm sure that's true, but it's a pretty sorry comment about the state of 
all these projects.

Well, OTOH, if one's unstated goal in using open source is as a kind of 
social networking mechanism, it's fine. You get to socialize with 
people, even in paid work time, because ostensibly, it's all about 
learning how to use FOOBAR software, which you need for your work. (It's 
still a rather limited social networking mechanism, since for one thing 
you're not going to meet many women that way...)

OTOH, if the point of the OSS software is to actually get any work done, 
it's pretty terrible. I guess the very big projects tend to get books 
written about them and you can just buy a book or three.

> 
> 
>> Terence also wrote a template engine called StringTemplate. ...
>>
>> (This is not a parody or a joke. See
>> http://www.antlr.org/wiki/display/ST/Introduction )
>>
>> That's the text you get via navigating www.stringtemplate.org->main
>> documentation->introduction.
> 
> Oh, I know. I've been there.

:-) Isn't that just beyond belief though?

> 
> I've written parsers using JavaCC and ANTLR (version 2.7 and 3.0). I 
> guess we can safely say that they each have their strengths and 
> weakness.

THe only comparative strength I can identify in ANTLR vs JavaCC is the 
ability to target multiple languages.

> 
> 
>> To what audience is the above paragraph addressed? A post-graduate
>> class in computer science? Or, put another way, Is the typical
>> programmer who is in the market for such a tool going to hit this
>> introduction and think to himself: "Aha! Suitable formalisms! Great,
>> that is exactly what I was looking for!"
> 
> You do know that Dr. Parr is a professor of computer science at the 
> University of San Francisco, don't you? 

Yeah, I saw that.

> So I just assume he knows what 
> he's talking about. I mean, I've never had a professor who didn't.

Well, sure. That's the most basic precondition for a teacher to be 
competent at their job. The problem is that it's a necessary condition, 
not a sufficient one. Other people also have to know what he's talking 
about. If there are complaints lodged about a prof's teaching, it will 
more often be on those grounds, where the guy is just talking at some 
level that goes over all the students' heads.

Anyway, I never suggested that Dr. Parr doesn't know what he's talking 
about. My issue in quoting that excerpt of his writing was whether 
anybody else understands...


> 
>> This gives a feeling for the kind of text this guy writes. I mean,
>> the ANTLR stuff is just full of all kinds of technical terms being
>> used without any definition or (more importantly) example. Left
>> factoring, backtracking, etc.. At various times,  surfing on amazon,
>> I was like one click away from ordering Terence's book on parsing
>> with ANTLR, but I always just remember stuff like this and refrained.
> 
> Well, if you're going to use ANTLR, then you probably need the book, (in 
> my opinion). It's specific to ANTLR 3.0, though.

So you have the book? If so, is it like those webpages? Now that I think 
about it a bit more, I guess it would be hard for it to be quite that 
bad, because, unless it's just self-published, it would have to go 
through some editorial process at the publishing company. They have some 
sanity checks.

> 
> 
>> I can't believe he would be so Machiavellian as to consciously write
>> text like this on his (free) website and hold back all his
>> undestandable writing for the book ...
> 
> Well, if he's a latter-day Machiavelli, he hides it well. (Which, I 
> suppose, is a prerequisite.)
> 
> 
>>>  JavaCC to be a more rational, reasoned effort while ANTLR is a
>>> little more "free and easy."
>> Well, the reason I keep wanting to have another go at ANTLR is that
>> it supports multiple target languages. It can generate not just in
>> java, but C# and Python. Maybe C# as well is done. There is ruby
>> back-end in the works. So it's attractive to have a freemarker
>> grammar written against a tool that can generate those languages.
> 
> Well, you'll need to join the ANTLR mailing list if you want to use it. 
> Same as pretty much anything, these days. There's a lot of list traffic 
> about C#, but I pay it no heed. But my impression is that it's not as 
> well-supported as Java.

I wonder what percentage of people who use an open source tool will 
typically subscribe to the mailing list? There are supposedly 161 
subscribers to this list, though some of them are archiving bots, I 
guess. Of course, some people only follow a list via the newsgroup or a 
web interface. As for people being active, like saying anything, it's 
not like one feels that there are over a hundred people here...

I have to think the 161 subscribers to freemarker-user is a very small 
proportion of the people who use the tool nowadays. OTOH, maybe the docs 
are sufficient for most people and it is a fairly straightforward thing 
to use anyway. A parser generator, no matter how good the docs would 
have more people subscribing to the list, I guess.

> 
> 
>> OTOH, my honest feeling at the moment is that if you have a project
>> that only targets Java and you have no foreseeable need to target
>> another language, you'd have to be a masochist to use ANTLR instead
>> of JavaCC.
> 
> Now he tells me...

I would've told you if you'd asked me. Anyway, that's just my opinion. 
Surely some people would disagree. (I mean, besides Dr. Parr... :-))

> 
> 
>>>  Kind of like the difference between Scala and
>>>  Groovy, though probably not such a strong contrast.)
>> Out of curiosity, which of those too is the rational, reasoned
>> effort? (I haven't really looked at either one that much.)
> 
> ?!? Groovy is a cowboy project and, to say the least, very "scruffy." 

How would you say it compares in this regard to Ruby (or Python)? 
(Assuming you have an opinion...just curious...) Of course, in terms of 
Grails, the Groovy/Ruby comparison seems quite to the point, since 
Grails is more or less based on Rails, isn't it?

I guess the performance must be a lot better than Ruby, with all the 
work that's been put into Java VM technology...

> Scala is a reasoned, theoretically well-founded development project and 
> very "neat."
> 
> Both are general-purpose programming languages that target the JVM and 
> interoperate with existing Java libraries. Scala aims to integrate 
> functional programming with object-oriented programming. Scala's 
> principal is Martin Odersky, one of the developers of Pizza, gcj and 
> early work on generifying Java. (But I don't think he deserves blame 
> for the oddball form of parametric polymorphism we now have in Java.)

Java generics probably aren't genuinely useful enough to justify the 
extra complication to the language AFAICS. Being able to say 
List<String>, say, instead of just List is a bit useful, and it's there, 
so I use it, but I think the ability to do something like that could 
have been hacked in pretty easily, maybe via something like annotations 
or something. I mean, then you have the convenience that you can say.

myString = list.get(n);

instead of

myString = (String) list.get(n);

and AFAICS, that's about all anybody gets out of all of that generics 
generality. A little bit of notational convenience, syntactic sugar. 
That's it. I mean, how many people write their own genericized 
containers?  There's hardly ever that much call for it and it's just too 
complicated anyway, it seems to me.

> 
> Scala requires a much bigger mental shift from Java than Groovy does. 
> I'm using Groovy (for the Grails front-end), but I haven't used Scala,  
> yet.

Every time I try to use a functional language, where I've define some 
little project I have an itch for that would be a good testing ground 
for learning a new programming language, I always start and fairly soon 
give up. I just cannot get my head around functional programming.

And all that FP literature goes on about how powerful it is and I end up 
feeling like screaming at them that a computer language is only powerful 
(I mean, as a practical matter) insofar as it helps people express 
algorithms more easily than they otherwise would. I mean to say, it's 
powerful (or not) in conjunction with human cognition, i.e. people's 
ability to get their heads around it and really use it. If, as a 
practical matter, most people cannot learn to use these things, they are 
not, at least in the general case, practically speaking, "powerful", it 
seems to me. THough, I understand their definition of "powerful" is not 
exactly that.

ANd that's what gets me about all this literature on ANtlr. ALmost every 
other paragraph, TP is telling you how powerful some extremely 
convoluted (AFAICS) way of doing things is. You don't need lexical 
states (though they are on his TODO list) because "semantic predicates" 
are more general and.... (you guessed it)... powerful. Except if that's 
true, it's only true theoretically, because I can't figure out how to do 
anything I want to do that way. I keep trying and it doesn't work. All 
this powerful stuff makes me (and I'm sure, not just me) feel extremely 
powerLESS.

> 
> 
>>>  And while we're talking about parsers, recently I've been feeling
>>> a need for a Lisp-ish s-expression-style grammar for writing FTL
>>> templates. ...
>> I've been consciously trying to make this kind of thing more
>> feasible, so it's easiest probably against the code in the SVN trunk
>> rather than what you have in 2.3 distro.
> 
> OK. After asking, I figured I ought to just dig in, but I checked out 
> tags/2.3.12. I'll get the trunk and look at that.

The stuff I pointed you to will only compile against the stuff in the 
SVN trunk.

> 
> 
>> Actually, this stuff I've been playing with to get FreeMarker to
>> handle Velocity syntax is the best example at the moment for what you
>> want to do.  In fact, you could pretty easily do what you want, I
>> think, using that stuff as a template. See:
> 
> Yeah, I thought that might be a hint / clue to how to get new surface 
> syntaxes for the template specification language.

Yeah, actually, thinking about it a bit more, I think it's drop dead 
simple. At least assuming that you know exactly what you want to do, 
what syntax you want to support, I think you could get it working as a 
one-day project. Basically, you just define your grammar in JavaCC, you 
use the JJTree tool with MULTI=true to generate XXXXNode classes for the 
stuff in your AST, and then, by analogy with the visitor class in 
VelNodeToFTLNode.java, you write yours that visits the nodes in your AST 
and that's about it. ANd then tweaking VelocityBridgeTemplate and 
VelocityBridgeTemplateCache to use your machinery is totally trivial. 
YOu'd have it working in a day, I think.

Cheers, (and Sorry for all the ranting)

JR

> 
> 
>> ...
>>
>> Actually, if you want to work on something like this, I'll give you
>> svn access and you can just put your stuff under the sandbox
>> directory. (I see no reason not to "let 1000 flowers bloom" in said
>> sandbox.  (Or you can store your work somewhere else of course. Just
>> an idea... :-))
> 
> Well, let's hold off on that for the moment. I'm really completely 
> flummoxed by my current work.
> 
> 
>> JR
> 
> 
> -==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==-
> 
> On Wednesday 02 April 2008 10:31, Daniel Dekany wrote:
>> Wednesday, April 2, 2008, 5:23:26 PM, Randall R Schulz wrote:
>>
>> [snip]
>>
>>> And while we're talking about parsers, recently I've been feeling a
>>> need for a Lisp-ish s-expression-style grammar for writing FTL
>>> templates.
>> [snip]
>>
>> Just out of curiosity, how would that look, and why do you feel need
>> for it?
> 
> To the first question, I don't exactly know, at least not yet.
> 
> As to the second:
> 
> I have a variety of problems that, while distinct, can all be variously 
> characterized as "pattern-directed rewritings" of one sort or another.
> 
> My requirements / constraints:
> 
> - The primary content is CLIF (Common Logic Interchange Format; 
> <http://cl.tamu.edu/>). It is a very Lisp-ish notation; not literally 
> Lisp, but it's at least as close to Lisp as Java is to C++.
>  
> - I don't want to proliferate notationals within my system.
> 
> - I need to eventually allow end users to create their own 
> pattern-directed rewrite rules.
> 
> 
> I keep trying to use Scheme macros, but they have limitations that 
> continue to block my attempts. One reason I'm looking at Scheme is that 
> there is a Java-embeddable Scheme implementation called SISC 
> (<http://sisc-scheme.org/>).
> 
> 
> Randall Schulz
> 
> 
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace