Re: FreeMarker, ANTLR & New FreeMarker Template Notations [was: JetBrains / IntelliJ IDEA v8 to Support FreeMarker!]

On Thu, Apr 3, 2008 at 11:19 AM, Attila Szegedi <[email protected]> wrote:
>
>  On 2008.04.03., at 10:46, Jonathan Revusky wrote:
>
>  > Well, that issue doesn't have much to do with this, because this is
>  > about a case where of  a 5 megabyte file (no FM instructions in there
>  > even) that's just getting puled in as plain text tokens, whitespace
>  > and printable characters and so on. A sequence of plain text is just
>  > getting continually appended to a StringBuffer and then all 5 megs is
>  > put in a single TextBlock object and that's the whole parsed template,
>  > a single plain text node. It's running out of memory as it slurps in
>  > the tokens and appends them to the StringBuffer. Probably the main
>  > problem is the llinked list of Token objects that JavaCC is
>  > maintaining internally.
>
>  Yes, the profiling I did confirmed that.
>
>
>  > And that is what is eating up all the RAM.  It
>  > isn't lookahead per se, because it builds this linked list of Token
>  > objects as it reads in tokens on an as-needed basis. And then it does
>  > end up with that whole linked list of tokens sitting in memory.
>
>  Yeah, but if it builds them on as-needed basis, how can it end up with
>  few millions at once in memory, if it doesn't need them all for the
>  lookahead?

It's not doing any lookahead really. I mean, you'll see other places
in the grammar, where there is stuff like:

LOOKAHEAD(<WHITESPACE><ID><EQUALS>)...

which says that it should go into that subrule if the next two tokens
popped off the token are whitespace followed by an identifier followed
by equals, just for example. And then if the next 3 tokens are not of
those respective types, it backtracks, and goes into the alternative
subrule, say. Then it needs the current token to have that next field
set, so that it can go forward in the token sequence again.

But none of that kind of thing is going on in this code. All this is
doing is that it just keeps reading in plain text tokens until it hits
something that isn't plain text, like ${ or <@ that would snap it out
of that. And in this case there is no need for the linked list chain,
because no backtracking or retracing ever happens.

The thing is that the use case is bizarre anyway, a million plain text
tokens all in a row like that. The problem isn't so much the amountof
text, but that it is contiguous, unbroken by any FTL instructions. And
then, of course, the reason for it is that I was retaining a reference
to the start token so that I could call textBlock.setLocation(....,
start, end) further down. I could also equally well have just grabbed
the starting line/column from that first token (to use later with the
alternative form of setLocation(....) and then I would have no need to
hang onto a reference to the starting token. But again, the whole
idea, that thinking about a typical template, that you're going to run
out of memory because there are like a million of these plain text
tokens all in a row, one can't be condemned really for failing to
anticipate this case.

> Or maybe it blindly builds a linked list always, relying on
>  your code to not keep a reference to earlier elements of the list when
>  you no longer need them?

Bingo!

Of course, the vast vast majority of input files parsed by a javacc
generated parser is code in some programming language or other and
fits in memory anyway...

>
>
>  > That should do it. I haven't tested this though. I know it's easy, but
>  > I didn't get any sleep last night...
>
>  I run a test, and I can now have Mirko's test doc parsed with -Xmx61M,
>  which is an improvement over -Xmx192M definitely. Mirko's manual
>  parser hack brought it down to -Xmx48M, but I think this is now also
>  acceptable. I'll backport this to 2.3.x too so Mirko or whoever else
>  can try it out quickly.

It still seems like too much memory, but then I remembered the
LineTableBuilder in the Template class. You know, the Template class
keeps a map of all the lines in the template file for location info
and error reporting purposes. So it's keeping an ArrayList of all the
lines in that 5 meg file. Assuming an average of 72 characters to a
line of text, that might be 80,000 lines of text being held in the
table... Of course, it depends on the text you used as a testcase. If
it was all on one long line, then it maybe would require far less
memory and so on.... Also, that TextBlock instance is sitting on 5
megabytes of text which stored as char's requires 10 megs....

I guess one possible optimization wrt the LineTable business would be
to null out the lines that contain only plain text, since we don't
need those to generate error messages, just the ones that contain FTL
code of some sort....

OTOH, how much effort do we put into addressing this use case? It's
not very common...

JR


>
>
>  > Anyway, I am pretty sure this fixes it and it really has nothing to do
>  > with our overall grammar or lookaheads or the tricky business of
>  > chanigng the the lexer state from the parser.
>
>  That's a relief :-)
>
>
>
>  Attila.
>
>  -------------------------------------------------------------------------
>  Check out the new SourceForge.net Marketplace.
>  It's the best place to buy or sell services for
>  just about anything Open Source.
>  http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
>  _______________________________________________
>  FreeMarker-user mailing list
>  [email protected]
>  https://lists.sourceforge.net/lists/listinfo/freemarker-user
>

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace