Re: FreeMarker, ANTLR & New FreeMarker Template Notations [was: JetBrains / IntelliJ IDEA v8 to Support FreeMarker!]

On 2008.04.03., at 10:46, Jonathan Revusky wrote:

> Well, that issue doesn't have much to do with this, because this is
> about a case where of  a 5 megabyte file (no FM instructions in there
> even) that's just getting puled in as plain text tokens, whitespace
> and printable characters and so on. A sequence of plain text is just
> getting continually appended to a StringBuffer and then all 5 megs is
> put in a single TextBlock object and that's the whole parsed template,
> a single plain text node. It's running out of memory as it slurps in
> the tokens and appends them to the StringBuffer. Probably the main
> problem is the llinked list of Token objects that JavaCC is
> maintaining internally.

Yes, the profiling I did confirmed that.

> And that is what is eating up all the RAM.  It
> isn't lookahead per se, because it builds this linked list of Token
> objects as it reads in tokens on an as-needed basis. And then it does
> end up with that whole linked list of tokens sitting in memory.

Yeah, but if it builds them on as-needed basis, how can it end up with  
few millions at once in memory, if it doesn't need them all for the  
lookahead? Or maybe it blindly builds a linked list always, relying on  
your code to not keep a reference to earlier elements of the list when  
you no longer need them?

> That should do it. I haven't tested this though. I know it's easy, but
> I didn't get any sleep last night...

I run a test, and I can now have Mirko's test doc parsed with -Xmx61M,  
which is an improvement over -Xmx192M definitely. Mirko's manual  
parser hack brought it down to -Xmx48M, but I think this is now also  
acceptable. I'll backport this to 2.3.x too so Mirko or whoever else  
can try it out quickly.

> Anyway, I am pretty sure this fixes it and it really has nothing to do
> with our overall grammar or lookaheads or the tricky business of
> chanigng the the lexer state from the parser.

That's a relief :-)

Attila.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace