Re: JetBrains / IntelliJ IDEA v8 to Support FreeMarker!

On Wed, Apr 2, 2008 at 5:23 PM, Randall R Schulz <[email protected]> wrote:
> Hello, Jonathan,
>
>  On Wednesday 02 April 2008 08:07, Jonathan Revusky wrote:
>  > ...
>
> >
>  > In other matters, Randall, I saw you had posted something on the
>  > ANTLR-interest list recently. It was as a result of googling stuff
>  > about how to do lexical states with ANTLR. I've been taking a run
>  > (another one) at ANTLR recently and... OMG... I'm on the verge of
>  > giving up, man, it's terrible. So difficult...
>  >
>  > Did you ever figure out how do simulate lexical states? Supposedly
>  > you use semantic predicates, which according to TP, is much more
>  > powerful than lexical states, but I can't get it to work. Did you
>  > ever figure it out?
>
>  I'm not sure. I wonder if that goes back to my writing a TPTP parser
>  using ANTLR 2.x or my more recent creation of a CLIF parser using ANTLR
>  3.0. Could you send me a link to the mail archive you found?

http://www.antlr.org:8080/pipermail/antlr-interest/2006-November/018405.html

Actually, it's not that recent. I think I mentally catalogued it as
recent because I was thinking (at 4 a.m.  or something) that this is
2007.

>  All I know for sure is that both these parsers are working. But JavaCC
>  seems to have easy-to-use lexical states, so maybe that's the better
>  choice.

This is my second run at Antlr. The first one was very brief, because
it was when I first got heavily involved in FreeMarker, I looked at
both JavaCC and Antlr. The JavaCC of the time was probably virtually
indistinguishable from the one now, and ANTLR has supposedly evolved
quite a bit, getting to the current 3.x version.  Yet still, the
lexical states, which JavaCC had 7 years ago, are on Terence's TODO
list for Antlr 3.x. OTOH, I'm sure that Antlr has features that JavaCC
lacks. Still, the lexical states, to my mind, seems like something so
basic that I find it mind-boggling that he would have implemented all
kinds of other things without doing that first.


>  (I don't want to start a big flame-fest between partisans, but I find

The problem with ANTLR is not exactly a *lack* of docs. Terence, along
with collaborators, have written a fair bit of verbiage on it.
However, it's a totally disorganized mass, some of it is simply
utterly dense and impossible to understand. Other parts of it are
downright misleading.

Terence also wrote a template engine called StringTemplate. It's a
competitor of this project in principle. It has developed some
following, though frnakly, I doubt their user community is even a
tenth the size of ours. Here is the first paragraph of the
introduction page on StringTemplate.

"Most programs that emit source code or other text output are
unstructured blobs of generation logic interspersed with print
statements. The primary reason is the lack of suitable tools and
formalisms. The proper formalism is that of an output grammar because
you are not generating random characters--you are generating sentences
in an output language. This is analogous to using a grammar to
describe the structure of input sentences. Rather than building a
parser by hand, most programmers will use a parser generator.
Similarly, we need some form of unparser generator to generate text.
The most convenient manifestation of the output grammar is a template
engine such as StringTemplate."

(This is not a parody or a joke. See
http://www.antlr.org/wiki/display/ST/Introduction )

That's the text you get via navigating www.stringtemplate.org->main
documentation->introduction.

To what audience is the above paragraph addressed? A post-graduate
class in computer science? Or, put another way, Is the typical
programmer who is in the market for such a tool going to hit this
introduction and think to himself: "Aha! Suitable formalisms! Great,
that is exactly what I was looking for!"

This gives a feeling for the kind of text this guy writes. I mean, the
ANTLR stuff is just full of all kinds of technical terms being used
without any definition or (more importantly) example. Left factoring,
backtracking, etc.. At various times,  surfing on amazon, I was like
one click away from ordering Terence's book on parsing with ANTLR, but
I always just remember stuff like this and refrained. I can't believe
he would be so machiavellian as to consciously write text like this on
his (free) website and hold back all his undestandable writing for the
book (which you pay for.) this is not, mind you, because I object so
much to cynical, Machiavellian behavior. I thought he was this was the
case, that would mean that the book is likely far more readable than
this other stuff, and maybe I'd buy it.

>  JavaCC to be a more rational, reasoned effort while ANTLR is a little
>  more "free and easy."


Well, the reason I keep wanting to have another go at ANTLR is that it
supports multiple target languages. It can generate not just in java,
but C# and Python. Maybe C# as well is done. There is ruby back-end in
the works. So it's attractive to have a freemarker grammar written
against a tool that can generate those languages.

OTOH, my honest feeling at the moment is that if you have a project
that only targets Java and you have no foreseeable need to target
another language, you'd have to be a masochist to use ANTLR instead of
JavaCC.

>  Kind of like the difference between Scala and
>  Groovy, though probably not such a strong contrast.)

Out of curiosity, which of those too is the rational, reasoned effort?
(I haven't really looked at either one that much.)

>
>  And while we're talking about parsers, recently I've been feeling a need
>  for a Lisp-ish s-expression-style grammar for writing FTL templates.
>  What might be involved in creating an alternate compiler for FreeMarker
>  templates?? Or perhaps I should just write a translator from my
>  (hypothetical) s-expression-based FTL language to the official one?

I've been consciously trying to make this kind of thing more feasible,
so it's easiest probably against the code in the SVN trunk rather than
what you have in 2.3 distro.

Actually, this stuff I've been playing with to get FreeMarker to
handle Velocity syntax is the best example at the moment for what you
want to do.  In fact, you could pretty easily do what you want, I
think, using that stuff as a template. See:

http://freemarker.svn.sourceforge.net/viewvc/freemarker/tools/vel2ftl/src/freemarke/r/tools/vel2ftl

you can check it out via:

svn co https://freemarker.svn.sourceforge.net/svnroot/freemarker/tools/vel2ftl

http://freemarker.svn.sourceforge.net/viewvc/freemarker/tools/vel2ftl/src/freemarker/tools/vel2ftl/VelNodeToFTLNode.java?revision=629&view=markup

Basically this class is just a visitor that walks the Velocity AST (as
generated by the Velocity parser) and generates the appropriate FTL
tree. And actually, come to think of it, the various
org.apache.velocity.runtime.parser.node.Node objects are generated by
JJTree, so if you used JJTree to generate your AST with the JJtree
option of MULTI=true, it is very likely that you could adapt this
example. Even setting the location info is fairly easy so that you get
error messages with locations based on the original in your LISP-like
syntax.

And you could also base your package on the other classes in that
package, VelBridgeTemplate and VelBridgeTemplateCache. Those are
actually very trivial subclasses of Template and TemplateCache. The
other class there is Vel2FTL which is pretty similar to Vel2FTLNode,
except it just generates FTL text. It's less desirable than using
Vel2FTLNOde, because that one transfers the line number info.

Actually, if you want to work on something like this, I'll give you
svn access and you can just put your stuff under the sandbox
directory. (I see no reason not to "let 1000 flowers bloom" in said
sandbox.  (Or you can store your work somewhere else of course. Just
an idea... :-))

JR


>
>
>
>  Randall Schulz
>
>  -------------------------------------------------------------------------
>  Check out the new SourceForge.net Marketplace.
>  It's the best place to buy or sell services for
>  just about anything Open Source.
>  http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
>
>
> _______________________________________________
>  FreeMarker-user mailing list
>  [email protected]
>  https://lists.sourceforge.net/lists/listinfo/freemarker-user
>

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace