Re: SrcML project

On Mon, Dec 20, 2004 at 03:37:00PM +0000, Martin Klang wrote:
> >As SrcML is more of a framework there are several modules included
> >and I suggest you check them all out from CVS.
> 
> I found I had to download jars antlr.jar and dom4j-full.jar to build.
> Also I got compilation errors compiling some of the modules, eg 
> parser-shared (after compiling config, api and util successfully).
> I'll find the top-level build.xml file and try that.

The neccessary jar files are mentioned as dependencies on the installation
page. I know it's still a bit of work to get it all up and running. This
is mainly due to the project still being in its infancy. You'll also have
to manually take care of the path issues (i.e. to include all those jars
in your CLASSPATH). The srcml-full.jar is a mean to deal with this issue,
so if you don't succeed in getting SrcML working from source drop me a note
and I'll wrap up a srcml-full.jar of the current CVS state for you.

> Framework for application development, similar to an IDE? Or framework 
> for code manipulation?

Something of both. We have toyed with the idea of writing an Eclipse 
plugin to do the conversion to SrcML, so that the remaining tools can 
operate on it (again maybe Eclipse plugins of standalone 
applications/scripts). But as I will try to explain later SrcML will
also provide a mean to improve the development process.
In fact I've come to think that most of these understanding problems
might occur from SrcML being used as the framework name as well as
the XML format name. However this might still change in the future.

> I believe that the data format, whether you wish to call it 
> intermediary or not, is really important (understatement).
> It's the basis that all other tools and utilities have to operate on, 
> be they visualisers, analysers or processors. That's why it must 
> contain the right information at the right level of abstraction.

Indeed. However for SrcML this format is intertwined with the API for
working on it. So the right level of abstraction can be made
available through the API as well.

> C switch statements allow loop unrolling in rather bizarre ways (as for 
> example in Duff's device), afaik this is not possible in Java.
> (googling - found it: 
> http://java.sun.com/docs/books/jls/first_edition/html/14.doc.html#35518 
> )
> The difference is more syntactic than semantic, but important 
> nonetheless.

Thanks for the pointer. We'll have to look at that in more detail and
keep it in mind. However as far as syntactic difference goes we should
be able to handle that in the parser and create a semantically equivalent
representation of that code. Although it might be a bit tricky in this
case of course.

> The 'infoset' of MLML is:
> - Types (classes)

As not all languages are typed you'd have to exclude some languages, or
MLML isn't a pure subset of the languages again?

> - Variables and variable references (fields)

What is 'Variables'? In SrcML we have Variablegroups, Variable and VarUse.
VarUse relates to what you call variable references and Variable is used
in declarations/definitions. We also have the Variablegroups to distinguish
int x,y;
from
int x;
int y;
as there is an intentional difference between the two versions. Although
this is - similar to comments - more of an esoteric issue, as it is of
course impossible to tell whether the intention to group the variables
was indeed present at the time of writing.

> - Functions and function calls (static and dynamic, oo and procedural) 
> (methods)
> - Operators and operations (expressions)
> - instructions and statements (as per above - defined language specific 
> statements)

I don't quite see the difference between expressions, instructions and
statements. If you mean something like 'a = 2+2;' being the statement and
2+2 being the expression, then this is indeed a high level abstraction
which allows for a lot of things to be represented. However for SrcML
we'd like to be able to dissect expressions further leading to this:
assignment
 |-lvalue
 |  \-a
 \-rvalue
    \-expression
       \-operation +
          |-operand1
          |  \-2
          \-operand2
             \-2

I can see how this might be containing too many details for some cases,
but then the neccessary abstraction layer can be provided through the
API.

> Auto-generated or transformed code is not a problem, unless you're 
> intending to transform the program back to source format and expect it 
> to still be readable and intelligible. Is that a design goal of SrcML?

Yes it is indeed :). What you're holding in your hand (aka harddisk) with
the current CVS version is already including a ./bin/view.py which converts
SrcML documents back into source code which is properly indented and
readable. It is no replica of the original code and the presentation is
still lacking at some points (lines getting too long, too many annoying
{..} and some (..) brackets, etc..). When you convert source code to SrcML
and run it through the view platform to create source code again you kind
of get a code beautifier as a side product.
I'm afraid the documentation of the ViewPlatform is not yet written, but
the general idea is to be able to create different visualizations of the
source code by simply using the different plugins. This ranges from the
creation of plain compileable source (no formatting needed to save time)
over a readable version of the source to a complete UML diagram of class
hierarchies of a project.
A more drastical approach to this is the use in IDEs. Our vision includes
code being stored on SrcML level, but the same code being shown totally
different when viewed by different developers due to their IDE settings.

Wilson's paper mentioned in my last mail is describing a rather 
long-termed view off this. Here's a small excerpt of his summary on
dividing the model (SrcML) from the view (IDE, ..):

=====
Using XML to separate software models from software views will bring 
four benefits:

   1. It will make languages more extensible. Programs stored as XML 
will be easier to process than ones stored as collections of arbitrary 
ASCII tokens. In particular, programmers will be able to apply XSL and 
other tools to them---tools which will be as familiar to tomorrow's 
programmers as regular expressions are to today's.
   2. It will simplify the construction of active libraries. Programmers 
will be able to mark up their code to indicate which sections are 
intended for which tools. This will allow them to store code and meta-code
side by side, in the same notation.
   3. Programmers will be able to embed arbitrary content in their code,
including as mathematics (using MathML), class diagrams (using SVG), and 
all the meta-data that Computer-Aided Software Engineering (CASE) tools 
require. Donald Knuth's dream of "literate programming" may therefore 
finally be realized [Knuth].
   4. Programmers may finally stop arguing about where curly braces 
should go, since they will be able to customize their views of software 
without modifying the underlying model. For example, a programmer could 
easily choose to view the code fragment shown above as this:
 if (record.age < threshold)    /* Only replace below threshold */
 {
     record.release();
 }
or even this:
 ;;; Only replace below threshold
 (if (< (record 'age) threshold)
     (record 'release))
without altering the underlying representation.
=====

As for the first point I think it is rather obvious and of course
shared among SrcML and MLML as well as most other such projects.
I personally see points 2 and 3 as almost equal. Marking sections
as relevant for certain tools comes down to embedding the content
for those tools in my oppinion. We have however not yet gone into
this direction with SrcML. We'll see next semester if some of the
students working on SrcML show an interest in working on this kind
of embedding.

The fourth point above is exactly what we achive through our
ViewPlatform. Changing the view from the first to the second
version is just a matter of selecting the other plugin and could
be integrated into an IDE and customized to the developer. 
The remaining challenge however is not to present this view, 
but to be able to re-parse it or in some other way reflect changes
made to it to the original SrcML representation. This becomes
especially evident when thinking of something like a class hierarchy
view which can modify inheritance relations and such. While it may
be rather easy to reflect those changes through meta-programming
it is significantly harder to do so on arbitrary formatted source
code. But then again there are already projects out there which
try to create parsers directly from the grammar. New views could
then be accompanied by their grammars.. but of course this is
mostly speculation and the way there is still a long and steep one.

With kind regards,
-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

Present to inform, not to impress; if you inform, you will impress.
(Frederick P Brooks)