Re: FreeMarker, ANTLR & New FreeMarker Template Notations [was: JetBrains / IntelliJ IDEA v8 to Support FreeMarker!]

On Wednesday 02 April 2008 15:41, Jonathan Revusky wrote:
> Randall R Schulz wrote:
> > ...
>
> The problem with what you're doing there, if I understand it right,
> is that you're  massaging the token stream in a certain way between
> the lexing and parsing steps and this is not at all specified in your
> grammar file, right?

(I'm going on my memory from back then.)

It mostly amounts to collecting comment tokens those that precede a 
top-level production get their content attached to the resulting parsed 
object. Since you can have comments anywhere and not just between the 
top-level productions (think methods or, maybe, classes), you have to 
figure out which to discard and which to keep. It wasn't ideal, but it 
worked well enough.

The more I think about it, the less sure I am this is the problem I was 
trying to solve when I asked about lexical states in ANTLR...


> OTOH, I guess that does work since Antlr reads in all the tokens
> before doing any expression parsing. That surprised me, as it strikes
> me as far less flexible than JavaCC in this regard.

It's a distinct problem if you want to be able to parse large inputs. 
The token stream representation isn't too efficient, though you can 
write your own if you need to.


> JavaCC pulls 
> tokens off the token source on an as-needed basis. In JavaCC you can
> even do something like token_source.setState(...) from within the
> expression parser.

I consider that a far more defensible design, personally.


> I use this in at least one point in the FreeMarker 
> parser, for example, when the parser enters a macro invocation with a
> set of named parameters, so the token source enters a different
> (though only slightly different) lexical state. As best I can tell,
> something like this is not even possible in Antlr. Of course, it
> could be that it's a the wrong way to do it, a horrible hack, but
> still, it strikes me as better than putting a separate
> token-massaging step in between lexing and parsing that is not even
> specified in the grammar file!

Well, real-world problems often get messy (and what FreeMarker must do 
is probably one of the more challenging parsing tasks going).

I guess the thing to do is learn about their (ANTLR's) notion of "island 
grammars" which allow you to mingle grammars... Or something like that.


> > ...
>
> Well, OTOH, if one's unstated goal in using open source is as a kind
> of social networking mechanism, it's fine. You get to socialize with
> people, even in paid work time, because ostensibly, it's all about
> learning how to use FOOBAR software, which you need for your work.
> (It's still a rather limited social networking mechanism, since for
> one thing you're not going to meet many women that way...)

I guess in part it depends on what you mean by "meet." It seems 
open-source contribution is much more a European thing. But maybe 
that's just a sampling artifact based on the O-S software I happen to 
use.


> OTOH, if the point of the OSS software is to actually get any work
> done, it's pretty terrible. I guess the very big projects tend to get
> books written about them and you can just buy a book or three.
>
> > ...
> >
> :-) Isn't that just beyond belief though?
> :

I spend a lot of time reading academic papers trying to find concepts I 
can apply to my work. A great deal of it is ... less than what I'd hope 
for.

Sturgeon's law comes frequently to mind...


> > I've written parsers using JavaCC and ANTLR (version 2.7 and 3.0).
> > I guess we can safely say that they each have their strengths and
> > weakness.
>
> THe only comparative strength I can identify in ANTLR vs JavaCC is
> the ability to target multiple languages.

The reason I originally decided to give it a try was the tools 
(ANTLRworks, specifically, which even exists as an IDEA plug-in). But 
I'm not sure it's worth it.


> >> ...
> >
> > You do know that Dr. Parr is a professor of computer science at the
> > University of San Francisco, don't you?
>
> Yeah, I saw that.
>
> > So I just assume he knows what
> > he's talking about. I mean, I've never had a professor who didn't.
>
> Well, sure. That's the most basic precondition for a teacher to be
> competent at their job. ...

My experience (coming once upon a time through a major reseach 
university) is that professors are chosen for their ability to publish 
and bring in research funds and graduate stundents first and their 
ability to teach well second (or third...).


> ...
>
> So you have the book? If so, is it like those webpages? Now that I
> think about it a bit more, I guess it would be hard for it to be
> quite that bad, because, unless it's just self-published, it would
> have to go through some editorial process at the publishing company.
> They have some sanity checks.

Yeah I have it. I have the paper and the PDF (I hate reading off the 
screen, but I like to be able to search the full text, and I have it 
indexed along with all my other on-line publications, which can 
occasionally be handy.)

And I'd say it's better than the Web site. I never really tried to get 
too much out of the Web site. (Or maybe I'm blanking it out of my 
memory. I did write one whole parser and much of another before I even 
got the book...)


> > ...
> >
> > Well, you'll need to join the ANTLR mailing list if you want to use
> > it. Same as pretty much anything, these days. There's a lot of list
> > traffic about C#, but I pay it no heed. But my impression is that
> > it's not as well-supported as Java.
>
> I wonder what percentage of people who use an open source tool will
> typically subscribe to the mailing list? There are supposedly 161
> subscribers to this list, though some of them are archiving bots, I
> guess. Of course, some people only follow a list via the newsgroup or
> a web interface. As for people being active, like saying anything,
> it's not like one feels that there are over a hundred people here...

No matter where you go or what you do, the ratio of reads / readers to 
writes / writers is always high. Even in information systems that's 
true!


> I have to think the 161 subscribers to freemarker-user is a very
> small proportion of the people who use the tool nowadays. OTOH, maybe
> the docs are sufficient for most people and it is a fairly
> straightforward thing to use anyway. A parser generator, no matter
> how good the docs would have more people subscribing to the list, I
> guess.

I don't have a feel for that, actually. I'd have to think there are more 
uses for templating than for parsing.

And I think that FreeMarker is definitely better documented than most 
pieces of software of comparable magnitude, so probably that keeps a 
lot of newbies off your back.


> > ...
>
> >>>  Kind of like the difference between Scala and
> >>>  Groovy, though probably not such a strong contrast.)
> >>
> >> Out of curiosity, which of those too is the rational, reasoned
> >> effort? (I haven't really looked at either one that much.)
> >
> > ?!? Groovy is a cowboy project and, to say the least, very
> > "scruffy."
>
> How would you say it compares in this regard to Ruby (or Python)?
> (Assuming you have an opinion...just curious...) Of course, in terms
> of Grails, the Groovy/Ruby comparison seems quite to the point, since
> Grails is more or less based on Rails, isn't it?

I haven't had more than passing exposure to either Ruby or Python. I 
have at least written some Groovy.


> I guess the performance must be a lot better than Ruby, with all the
> work that's been put into Java VM technology...

Groovy certainly benefits a lot from the JVM optimizations, but they 
squander a great deal of that with their dynamic method dispatching.


> > Scala is a reasoned, theoretically well-founded development project
> > and very "neat."
> >
> > ...
>
> Java generics probably aren't genuinely useful enough to justify the
> extra complication to the language AFAICS. ...

Well, when combined with good static analysis tools (and when they're 
intergrated into your IDE, as they are with IDEA), then they're 
worthwhile, if less than ideal.

And when you're writing code that does boatloads of indexing and 
cross-referencing, it's nice to be warned when you're trying to put the 
wrong thing into a collection or take something out of the wrong (kind 
of) collection.


> ...
>
> and AFAICS, that's about all anybody gets out of all of that generics
> generality. A little bit of notational convenience, syntactic sugar.
> That's it. I mean, how many people write their own genericized
> containers?  There's hardly ever that much call for it and it's just
> too complicated anyway, it seems to me.

I've written a few. A MultiMap<Key, Value> class, a ArrayList-based 
Stack. And looking over my utilities classes, I find a FilterIterator, 
a SkipList and a priority queue based upon the SkipList.

All in all, I happier with Java generics than I was without them.


> > Scala requires a much bigger mental shift from Java than Groovy
> > does. I'm using Groovy (for the Grails front-end), but I haven't
> > used Scala, yet.
>
> Every time I try to use a functional language, where I've define some
> little project I have an itch for that would be a good testing ground
> for learning a new programming language, I always start and fairly
> soon give up. I just cannot get my head around functional
> programming.

Well, Scala doesn't force it on you, so perhaps it's a better starting 
point.


> And all that FP literature goes on about how powerful it is and I end
> up feeling like screaming at them that a computer language is only
> powerful (I mean, as a practical matter) insofar as it helps people
> express algorithms more easily than they otherwise would. ...

Well, the reason people keep at them is that you can prove a lot more 
interesting things about functional programs than you can about the 
semantic balls of string that are conventional imperative, 
side-effect-laden programs. And given the sorry state of software 
reliability, that's nothing to sneeze at.

Not that I disagree with what you say. FP is far from a perfected 
concept.


> ANd that's what gets me about all this literature on ANtlr. ALmost
> every other paragraph, TP is telling you how powerful some extremely
> convoluted (AFAICS) way of doing things is. You don't need lexical
> states (though they are on his TODO list) because "semantic
> predicates" are more general and.... (you guessed it)... powerful.
> Except if that's true, it's only true theoretically, because I can't
> figure out how to do anything I want to do that way. I keep trying
> and it doesn't work. All this powerful stuff makes me (and I'm sure,
> not just me) feel extremely powerLESS.

Well, one thing about ANTLR that's good for programmer productivity is 
LL(k) parsing with all the automatic backtracking. I got my latest 
parser (for CLIF) up and running really fast. (It's a Lisp-y syntax, 
which means there aren't a lot of structural signifiers to guide the 
parse and help disambiguate certain things—just a lot of 
parentheses...) Of course, all that automatically supplied backtracking 
means that error diagnostics are not well localized, but at least when 
the project's priorities shift enough, I can go in and clean that up.


> > ...
>
> >> Actually, this stuff I've been playing with to get FreeMarker to
> >> handle Velocity syntax is the best example at the moment for what
> >> you want to do.  In fact, you could pretty easily do what you
> >> want, I think, using that stuff as a template. See:
> >
> > Yeah, I thought that might be a hint / clue to how to get new
> > surface syntaxes for the template specification language.
>
> Yeah, actually, thinking about it a bit more, I think it's drop dead
> simple. At least assuming that you know exactly what you want to do,
> what syntax you want to support, I think you could get it working as
> a one-day project. Basically, you just define your grammar in JavaCC,
> you use the JJTree tool with MULTI=true to generate XXXXNode classes
> for the stuff in your AST, and then, by analogy with the visitor
> class in VelNodeToFTLNode.java, you write yours that visits the nodes
> in your AST and that's about it. ANd then tweaking
> VelocityBridgeTemplate and VelocityBridgeTemplateCache to use your
> machinery is totally trivial. YOu'd have it working in a day, I
> think.

OK. I may give that a try. Right now I can't get my mind to settle on a 
single approach. Nothing seems even approximately right.


> Cheers, (and Sorry for all the ranting)

Ha! I love ranting!


> JR


Randall Schulz

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
FreeMarker-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/freemarker-user