Re: FreeMarker, ANTLR & New FreeMarker Template Notations [was: JetBrains / IntelliJ IDEA v8 to Support FreeMarker!]

On 2008.04.03., at 11:47, Jonathan Revusky wrote:
> The thing is that the use case is bizarre anyway, a million plain text
> tokens all in a row like that. The problem isn't so much the amountof
> text, but that it is contiguous, unbroken by any FTL instructions.

Well, yes, but our tokenizer will tokenize the stream in plaintext  
blocks regardless... I guess if there were a different lexical state  
for between instructions that'd help a bit, but in reality, I don't  
think we'd need to complicate the parser for this. The effort/benefit  
ratio wouldn't justify it, I think.

> But again, the whole
> idea, that thinking about a typical template, that you're going to run
> out of memory because there are like a million of these plain text
> tokens all in a row, one can't be condemned really for failing to
> anticipate this case.

Of course not.

>
> It still seems like too much memory, but then I remembered the
> LineTableBuilder in the Template class. You know, the Template class
> keeps a map of all the lines in the template file for location info
> and error reporting purposes. So it's keeping an ArrayList of all the
> lines in that 5 meg file. Assuming an average of 72 characters to a
> line of text, that might be 80,000 lines of text being held in the
> table... Of course, it depends on the text you used as a testcase. If
> it was all on one long line, then it maybe would require far less
> memory and so on.... Also, that TextBlock instance is sitting on 5
> megabytes of text which stored as char's requires 10 megs....
>
> I guess one possible optimization wrt the LineTable business would be
> to null out the lines that contain only plain text, since we don't
> need those to generate error messages, just the ones that contain FTL
> code of some sort....

That could maybe be done...

> OTOH, how much effort do we put into addressing this use case? It's
> not very common...

Right. I'd say, we shouldn't put too much effort into it. Your fix  
helped bring down the parsing memory usage significantly. Eliminating  
unneeded elements from the "lines" list would also make sense,  
provided it's reasonably easy to implement. If you look at my comments  
in <http://sourceforge.net/tracker/index.php?func=detail&aid=1851842&group_id=794&atid=100794 
 >, I said there "... found out that after parsing, the template will  
actually take 25.5 MB. Of this, Template.lines will take up 14.26 MB,  
and rootElement and below will take 11.3 MB. Actually, the 11.3 MB  
part is okay, as Java stores each character on 2 bytes, so 5.7 MB  
UTF-8 encoded file does expand to about 11.4 MB. I'm worried about  
"lines" though..."

Attila.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace