Re: Continue Eclipse Freemarker DLTK?

Wednesday, July 24, 2013, 11:21:58 AM, Angelo zerr wrote:

> Hi Daniel,
>
> It's a little difficult for me to give you a good answer because it
> was long time that I have developped DLTK Freemarker. 
> DLTK provides an ASTNode interface that you must implement. The
> goal is to build a DLTK ASTNode from a FM content.
>
> FM 3.0 provided an ASTVistor that I have implemented to build a
> DLTK AST . See
> https://github.com/angelozerr/Freemarker-Eclipse-DLTK/blob/master/org.eclipse.dltk.freemarker.core/src/org/eclipse/dltk/freemarker/internal/core/parser/visitors/DLTKFreemarkerASTVisitor.java

(In terms of information it's the same as if you just walk the AST.)

> I had managed only if block (see
> https://github.com/angelozerr/Freemarker-Eclipse-DLTK/blob/master/org.eclipse.dltk.freemarker.core/src/org/eclipse/dltk/freemarker/internal/core/parser/visitors/DLTKIfBlock.java)
> and assign directives (see
> https://github.com/angelozerr/Freemarker-Eclipse-DLTK/blob/master/org.eclipse.dltk.freemarker.core/src/org/eclipse/dltk/freemarker/internal/core/parser/ast/expressions/Assignment.java).

So DTLK gets its AST, but it's almost empty, if I see it well. Why
DLTK requires that AST, what can it do with it if what information I
provide? So that's what we should figure out. I don't necessarily mean
right now, I guess as the development goes on, it will turn out. And
then I hope it will be a good representation of what's needed in the
FreeMarker Tools API in general.

> To build a DLTK ASTNode we have 2 choices : 
>
> 1) Load the FM Template and loop for each structure to build the
> DLTK AST Node (it was the solution that I used with FM 3.0
> ASTVistor). But it seems you don't want expose the AST structure of FM (why?)

Basically, because the AST is a too low-level stuff.

1) I don't want to promise backward compatibility on AST-level... it's
an internal implementation detail. FM already has a quite quirky API
regarding (randomly...) exposing internals as public API-s, which
binds my hands if I want to fix things. (Luckily people aren't looking
at the API much, they mostly just write templates.) (The AST isn't
particularly intuitive either, if you consider #if VS #if-#else, or
the way #attempt-#recover is represented. Or even assignments.)

2) Also, as I said, I think if a 3rd party app digs the AST to extract
something useful out of it... it will probably show that they don't
know FTL much.

3) Then, if FTL improves (and if it doesn't, it's dead so then the
whole thing doesn't mater, so let's assume it will), those tools will
often break. It's like putting even more backward compatibility burden
on FM. Like if you add a new directive that can write variables,
that's backward compatible now, but if people try find out what
variables are assigned where, or pretty much anything for that you
need a detailed AST, then it's not BC anymore. Surely it "only" breaks
tools, not apps, still if it can done better... If they want to know
what variables assigned, etc., they should ask the FreeMarker Tools
API, not digging the AST. It will tell that with the positions. They
don't have to deal with details like multiple assignments per #assign
VS multiple #assign-s with single assigments, out-variables of
directive calls, argument variable declarations VS #local assignments,
etc.

> 2) Create a DLTK AST directly by the Javacc.
>
> The 2) is more faster than 1) (because it avoids to create a FM
> structure before).

(If we want to improve speed, I think right now the most important
thing would be to rewrite the lexter. I don't believe it has to be
that slow as it's now, and that would be also important for
`?interpret` (or `?eval` maybe). The performance loss on creating two
AST-s is certainly nothing compared to that.)


-- 
Thanks,
 Daniel Dekany


> My idea is to change the FTL.jj
>
> to use a factory which creates template element. So instead to do
> that in the FMParser :
>
> -------------------------------------------------------
> BlockAssignment ba = new BlockAssignment(block, varName, scope, nsExp);
> -------------------------------------------------------
>
> Use a factory :
>
> -------------------------------------------------------
> IBlockAssignment ba = factory.newBlockAssignment(block, varName, scope, nsExp);
> -------------------------------------------------------
>
> IBlockAssignment is an interface. By using a factory gives you the
> capability to use our own structure. For Freemarker, teh factory
> will create IIBlockAssignment with existing IBlockAssignment
> For DLTK, factory will create IIBlockAssignment with
> DLTKIBlockAssignment which will implement DLTK ASTNode.
>
> What do you think about that?

If we will need a such detailed AST then that's probably the good
approach. But as I have written earlier, that's something that should
be avoided, if it can be. I'm not 100% sure it can be avoided, but I
hope it can be.

FM has a quite detailed *internal* AST traversal API since 2.3.20.
It's a nasty one, tries to minimize runtime impact etc., but it's not
for the public so I thought it's OK. So if we could explore what
information is *actually* needed for the DTLK plugin, then I could
implement an API that gives you just that information through a nicer
public API. Maybe in the form of a tree, through the visitor pattern,
for some queries. I don't want to pass in AST nodes, only the
information needed. As you move along, and you realize new things are
needed, the capabilities of this tools API would be extended, and the
point is, it will belong to the FreeMarker project, not to the DLTK
project. So if I add a new feature, it's my responsibility to bring it
up to date. What do you think about this approach?

> Regards Angelo
>
>
> 2013/7/23 Daniel Dekany <[email protected]>
> Tuesday, July 23, 2013, 11:59:55 AM, Angelo zerr wrote:
>
>> Hi Daniel, Greg,
>>
>> @Daniel : perhaps a visitor pattern could be a clean mean to avoid exposing the Template AST?
>
> Maybe, but I think the important question is if what information
> should be exposed (with visitor pattern or otherwise). Since you have
> experience with developing IDE-s and editors, maybe you already know
> that. Consider that some information (like, what variables are used in
> a template or in a block of a template) should be exposes through an
> API rather than left for the IDE to extract. Or at least, that's what
> I think would be the best (see last mail). So, what information is
> enough for the tree stuff? Are statements (TemplateElement-s) enough,
> or do we need the expression nodes too? What information is needed
> about the statements?
>
> --
> Thanks,
>  Daniel Dekany
>
>> @Greg : I'm waiting for your study about Freemarker DLTK to restart it (switch to the FM 2.3)
>>
>> Regards Angelo
>>
>>
>> 2013/7/22 Greg Amerson <[email protected]>
>> Hey guys,
>>
>> I hope to be able to take a look at Angelo's DLTK based plugins
>> this week and then we can start discussing our next steps of our collaboration on them.
>>
>>
>> On Sun, Jul 21, 2013 at 8:41 PM, Angelo zerr <[email protected]> wrote:
>> Hi Daniel,
>>
>> I"m waiting for Greg answer if he likes DLTK plugins and if he wishes to contribute it.
>> I don't want to develop this plugin alone (it's a big task and I
>> know if we are several, our motivation will be kept).
>>
>> If Greg is motivated, the first step is to switch to the FM 2.3
>> (today dltk plugin uses FM 3 based on freecc).
>> I have created
>> https://github.com/angelozerr/Freemarker-Eclipse-DLTK/tree/master/freemarker.provisionnal
>> which contains java classes usefull for debugger and ide with FM.
>> It should be cool if FM contains thoses classes (once dltk plugin will switch to FM 2.3)
>>
>> Regards Angelo
>>
>>
>> 2013/7/13 Daniel Dekany <[email protected]>
>> Hi,
>>
>> Anything with this since then? Last time we were there that it should
>> be checked if what changes are needed in the current branch of
>> FreeMarker for it become usable with the DLTK plugin. I know, I could
>> check that too... it's that I have too much on my plate regarding
>> FreeMarker already. Lot of core stuff, and I don't think that anybody
>> can solve them in the foreseeable future but me. OTOH there are
>> several other critical things for FreeMarker that's much easier to
>> jump into, and they also need some specific experience that I happen
>> not to posses (much...):
>>
>> - Better Eclipse IDE. (I have used the JBoss Tools stuff nowadays, and
>>   it's not in a good shape at all.)
>>
>> - Spring integration should be checked by someone. Framework authors
>>   often just throw in FM, together with Velocity, because it's the
>>   standard checklist thing to support template engines, and end of
>>   story. So from the viewpoint of the user it will simply mean that
>>   FreeMarker sucks.
>>
>> - Struts integration too.
>>
>> - Better JSP support. The biggest issue with the current one is that it's
>>   part of the FreemarkerServlet. People often want to call FreeMarker
>>   by other means. FreemarkerServlet is rather limiting and ugly anyway...
>>
>> Anyway... my current priorities are (in the order I plan to do them):
>>
>> - Fixing overloaded method selection and numerical conversions on method
>>   calls. This is in progress.
>>
>> - Allow more configurability with the String-based settings API (as
>>   this is what most frameworks use in their own configuration XML-s).
>>   Without this the overloaded method fixes can't be activated...
>>
>> - Adding stepping to the debug API
>>
>> - Adding #p and some other advancement in formatting.
>>
>> But meanwhile if something blocks the DLTK plugin development, I will
>> try to act quickly.
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
>> Tuesday, June 11, 2013, 2:11:41 PM, Angelo zerr wrote:
>>
>>> Hi Greg, Daniel,
>>>
>>>> Do you think you could rehost your DLTK plugins onto github?  That will make it easier to collaborate.
>>> Sure!
>>>
>>> I have pushed Freemarker DLTK on GitHub
>>> https://github.com/angelozerr/Freemarker-Eclipse-DLTK
>>> It works with last version of DLTK 4.0 (current is 5.0).
>>>
>>> I have written a short dev-guide
>>> https://github.com/angelozerr/Freemarker-Eclipse-DLTK/wiki/Developer-Guide which explains how to launch it.
>>> For the features, please read for the moment
>>> https://freemarker.svn.sourceforge.net/svnroot/freemarker/sandbox/org.eclipse.dltk.freemarker/docs/
>>>
>>> I add you as colaborator in my GitHub project. So I think you can push it.
>>> @Daniel, if you are interested, I can add you too.
>>>
>>>> Then I can install DLTK dependencies in my environment and try to import them in.
>>> No needs to install DLTK with classic mean. Just set the Target
>>> Platform (see explanation at
>>> https://github.com/angelozerr/Freemarker-Eclipse-DLTK/wiki/Developer-Guide)
>>> I like this idea about using Target Platform, because you can
>>> switch to another version of DLTK without installing it in your Eclipse.
>>>
>>>> Which part of the features of the current DLTK editor require FM3.0 and which can work on 2.3?
>>> My work doesn't work with FM 2.3 because FM3.0 changes package name
>>> for instance. But I think it will be easy to switch to 2.3
>>> the only thing is to create some patches because debugger should be improved.
>>>
>>>> If you are interested I could create a GitHub project in my GitHub https://github.com/angelozerr
>>>
>>> That's good, I prefer that over sf.net SVN.
>>>
>>> Ok, it's done.  https://github.com/angelozerr/Freemarker-Eclipse-DLTK
>>>
>>>
>>> The FM 3 branch you are talking about (the sf.net SVN trunk) is
>>> "officially" dead for a long time, and it wont ever yield a release.
>>> So in theory, you had to build on what's on GitHub (use the "2.3-gae"
>>> branch now). Of course, 2.3 has virtually no public AST API, so you
>>> can't use that either. (And I guess some other things will be missing
>>> too. Can you tell me what?) So for now you will have to work against
>>> FM 3.
>>>
>>> As of what will it work on then when it's released... The AST stuff
>>> changes will break BC too much for a 2.3 (I guess), so it will have to
>>> be FM 2.4. But if it's 2.4.0, then of course I will want to put in
>>> some other minor not-strictly-BC changes too (not to mention new
>>> features that somehow justifies in the eyes of the average uses the
>>> 2nd version increase). I will be careful not to be too ambitious with
>>> it, as I don't want to impede the release of the plugin, or undermine
>>> the trust on the release of the new 2.4 branch. So as you see, there
>>> are some difficulties, but if you are going to work on the plugin, I
>>> will try very hard to clear its way.
>>>
>>> I will try to migrate to 2.3 to see if they are a lot of modification to do.
>>>
>>>> 3) DBGP Debugger
>>>>
>>>> Existing code of Debugger should be modified to support several
>>>> implementation of Debugger (RMI, DBGP, etc). So I think it's the
>>>> first step if we decide to develop Freemarker DLTK.
>>>
>>> (I see the discussion about that with Greg Amerson has been started .)
>>>
>>>> Regards Angelo
>>> Regards Angelo
>>>
>>> --
>>> Thanks,
>>>  Daniel Dekany


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk