Re: Question on TemplateCache synchronization

On Sat, Apr 12, 2008 at 11:45 AM, Attila Szegedi <[email protected]> wrote:
> Well, I probably wouldn't touch TemplateLoader as such -- surprisingly
>  many people have written their own custom loaders. TemplateCache can
>  provide a reader-to-string "slurping" transformation internally.
>  Actually, if I were to write a new loader API, what I'd do primarily
>  is merge the timestamp lookup and retrival, so it's REST friendly,
>  i.e. remote retrieval over HTTP can be implemented with a single GET
>  operation using If-Modified-Since header etc.
>
>  Anyway, aside from that, I like your idea of having all textual state
>  of the template backed up by a single string or char sequence. Problem
>  is, Java APIs are often very inconsistent regarding whether it's more
>  optimal to use String or char[]. I.e. one thing I did back in the 2.0
>  or 2.1 day was I changed the backing store for TextBlock from String
>  to char[], because that way it could invoke Writer.write(char[])
>  directly. I wanted to avoid invoking Writer.write(String) as it will
>  copy the String's characters into a character array and then call
>  write(char[]) so by doing this we avoid copying the string's
>  characters on each write;


Yes, I became aware of that at some point. Anyway, check out the
latest reworking of this.

Now the TextBlock objects don't need to keep their own copy of the
text. See, for example:

Template.writeTextAt(...)

To achieve this cleanly involved a huge rewrite of all that crappy
whitespace trimming stuff I wrote 5 years ago. Using the ASTVisitor
approach, I was able to re-implement it much more cleanly. Still, all
the logic is daunting. THe machinery requires 3 basic steps.
Basically, in the parsing step, the raw character data is chopped up
into 3 kinds of TextBlock node, regular text, opening potentially
ignorable whitespace, and trailing potentially ignorable whitespace.
Then in the post-parse step, the PostParseVisitor builds up a line
info table inside the Template object, that says which lines have trim
instructions and which lines definitely, according to the rules,
output text, and so on. And then there is a separate
WhitespaceAdjuster object that walks the tree and decides which of the
potentially ignorable nodes really are ignorable. And even then there
is another level, where it checks for whitespace nodes that are
ignorable on the basis of, say, being sandwiched between two
assignments or something.

Though currently, there are still 8 unit test failures due to
whitespace deltas. I'm not 100% sure that all 8 failures are really
failures. It may be that the whitespace trimming machinery was
tripping on certain situations and I got fed up and cheated by
changing the unit test so that it validated the implementation. And
maybe now the cleaner implementation is actually doing it correctly,
so it is "failing" to replicate the behavior of the earlier
WS-stripping stuff.

 But anyway, with the WS cleanup and the LineTableBuilder cleanup, I
got one place I wanted to go, which is that TextBlock objects don't
need to store their text.

> here's Writer.write(String str, int off, int
>  len) from JDK 1.5 source code:
>
>      public void write(String str, int off, int len) throws
>  IOException {
>         synchronized (lock) {
>             char cbuf[];
>             if (len <= writeBufferSize) {
>                 if (writeBuffer == null) {
>                     writeBuffer = new char[writeBufferSize];
>                 }
>                 cbuf = writeBuffer;
>             } else {    // Don't permanently allocate very large buffers.
>                 cbuf = new char[len];
>             }
>             str.getChars(off, (off + len), cbuf, 0);
>             write(cbuf, 0, len);
>         }
>      }
>
>  So that'd be a case for storing the Template text content as char[].

That is what I did eventually.


>  However, many other elements will need to use a String one way or the
>  other, so for them doing a new String(char[]) all the time would be
>  wasteful (not to speak of again copying data). A good way to share one
>  underlying char[] is to have one String for the full template source,
>  and obtain all other strings using String.substring() on it. But then
>  we're again back to using Writer.write(String) for a TextBlock. So
>  yeah, it's a surprisingly hard problem to address correctly to have
>  *both* low memory footprint *and* good runtime performance; it's not
>  something that'd be inherently hard, the hardness comes from
>  java.lang.String's (justified) immutability requirements *and* lack of
>  immutable arrays in Java...

I think the current reworking may be about as good as is possible. Do
take a look when you get a chance. There are still odds and ends to
clear up. It really is a huge refactoring.

JR

>
>  Attila.
>
>
>
>  On 2008.04.11., at 19:54, Jonathan Revusky wrote:
>  > BTW,
>  >
>  > I rewrote Template.java yesterday night to be more memory efficient.
>  > Finally, you know, what I did, is I just read all the template into a
>  > String and create a map of lines by running through that string and
>  > using String.substring(..) calls. That way, since String objects are
>  > immutable, the substring returned just uses a region of the character
>  > array in the main TemplateText string.
>  >
>  > It doesn't create a whole bunch of new character arrays like the old
>  > LineTableBuilder hack I wrote back when I implemented the error
>  > reporting stuff.
>  >
>  > The thing is that, finally, I wonder why we bother passing around
>  > Reader
>  > objects and so on in this Template API. I think it might just be
>  > simpler
>  > to use String just about everywhere we use Reader. Or we could use
>  > CharSequence for generality maybe.
>  >
>  > I mean, if we're going to keep all this text in memory anyway, why not
>  > just slurp it all into a String anyway and, where the TemplateLoader
>  > vends a Reader, just have it pass back a String and be done with it. I
>  > think the whole thing might just end up being a lot simpler and even
>  > more efficient that way.
>  >
>  > JR
>
>
>
>
>
>
> -------------------------------------------------------------------------
>  This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>  Don't miss this year's exciting event. There's still time to save $100.
>  Use priority code J8TL2D2.
>  http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
>  _______________________________________________
>  FreeMarker-user mailing list
>  [email protected]
>  https://lists.sourceforge.net/lists/listinfo/freemarker-user
>

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone