SrcML project

Hello,

I'm writing on behalf of the SrcML developers (www.srcml.de).
Apparently our project was mentioned on this list earlier and
Martin contacted us off-list and requested us to discuss the
issues in this mail on the list for a broader audience. So
without further ado:

On Fri, Dec 17, 2004 at 01:03:34AM +0000, Martin Klang wrote:
> Okay, I've checked out all the CVS modules. Now what do I do?!
> Are there any examples of SrcML code available?

We don't yet have any relevant real-life applications using SrcML
and therefore the examples and introductions are still lacking.
As SrcML is more of a framework there are several modules included
and I suggest you check them all out from CVS. On the webpage
describing the installation you can find an additional build.xml
file which can be used - as described there - to simply build the
whole framework with a single ant command.

As we are basing SrcML on Java you can then use the generated .jar
files in your own java programs. For convenience we provide a few
commandline scripts based on jython which perform the most common
operations. Those scripts can be found in the srcml-bin/ module.
Some example usages for the parser and sgrep are given here:
http://bach.informatik.uni-ulm.de/srcml/wiki/index.php/SrcMLWiki:Modules#bin
(sorry for the long URL)
There is currently no documentation for the view.py script as its
rather new.
You can read about the details on our webpage, but here's the short
version:

- take a java file (only <=1.3 supported so far. 1.4 should work too,
  1.5 won't work yet)
- run the parser on it. simplest way: 
  $ ./bin/parse.py myclass.java > myclass.srcml
  This generates a SrcML representation of the given code.
- You can now query the code using sgrep with any XPath expression, f.ex:
  $ ./bin/sgrep.py '//variable[@name="i"]' myclass.srcml
  This is not really supposed to be as useful as the standard grep tool
  and the output is therefore not really readable. You get no output and
  a non-zero exit code if such a node is not found, otherwise you just
  get the toString() conversion of the node found.
- Finally as SrcML is more of a background format and we expect tools to
  work on it, the .srcml file may have been modified. 
  However as I pointed out in an earlier mail to Martin we believe that
  developers do not want to work directly on the XML representation of
  their source code. In fact we have created a ViewPlatform which allows
  the conversion of SrcML to just about any format. This includes normal
  (compileable) text representations as well as control flow diagrams,
  class interaction visualizations, etc.
  The supplied view.py file is given a plugin which will determine how the
  converted output will actually look like. Right now we only have 
  implemented the conversion from SrcML back to normal Java source code.
  You can achieve that conversion like this:
  $ ./bin/view.py myclass.srcml > myclass_again.java
  As you will come to realize there are differences between the original
  and the recreated version. However as with o:XML and MLML we also only
  want to achieve semantical equivalence here. (Our biggest problem are
  comments which are rather hard to properly relate to the part of code
  they cover)

> I've experimented with direct translation from o:XML to Java, which  
> sort of works (experimental, incomplete!) but is not ideal.
> Instead, the key tool for o:XML is the compiler/interpretor (ObjectBox)  
> which executes o:XML programs in a Java environment.

>From what we've seen in the programming guide for o:XML it appears as
if o:XML is highly based on XSL with f.ex procedures being used instead
of the XSL templates. Of course this might be a bit blunt, but from that
introduction we can not yet see the big difference.

> MLML would provide a platform for o:XML to, eg, Java transformations,  
> as well as other things. In that context MLML is simply an intermediary  
> code format. Much like SrcML, no?

SrcML is essentially a framework. It does have an intermediary code
format specified in our DTD. Though it is not really intermediate, but
rather an underlying format. The data is kept as XML and worked with it,
but it is not visible to the user/developer most of the times. It's also
not intermediate, because we have tools working on SrcML and producing
SrcML as output again.

> MLML was meant to be quite minimal in the constructs that it provides.  
> To provide a language superset would require not only that all  
> constructs of each accepted language would have to be represented, but  
> also that the differences between languages somehow be clear. Eg a  
> 'switch' statement will mean slightly different things in say Java and  
> C, which would somehow have to be taken into consideration.

I don't really see a difference between the switch statement in Java
and C. Do you mind explaining this point a little further?

> Instead I thought to provide a minimal subset, including basic  
> conditionals and iterations. All other constructs and syntactic sugar  
> can be translated to a more minimal form, which would preserve semantic  
> identity. However, this would mean moving away from representing the  
> source code towards a more assembly-like format, which is against the  
> design principles of MLML.

Indeed we went through the same thought process for SrcML. However we
came to realize that this minimal subset would in fact be very limited.
You have to take into account, that not even objects, methods or
procedures are shared among all languages. Not even variable declarations
would be fond in this subset.
Our next thought was to go into the direction you already proposed as
well:

> As a possible solution that I've recently considered, the specific  
> constructs of any language could be represented in a generic way, eg:
> <mlml:statement name="java:switch">
>    <mlml:param name="arg"/>
>     ...
> </mlml:statement>
> This generic form would be useful not only because it can represent any  
> language construct of any language, but also because it provides a  
> language for reasoning about differences, and similarities, between  
> specific programming languages and their constructs.
> The drawback is that the form is so general, it might make it difficult  
> to express common content in a common way.

The drawback turned out to be too heavy for our goals. One of the main
goals of the SrcML framework is to allow developers to create tools which
can work on different languages, without (in the most idealistic case)
even knowing what language it is working with. This however turns out to
be impossible when every single construct is only available in a language
dependent namespace.
If you're interested in this approach you can take a look at the 
languages module of SrcML which provides a means of dynamically gather
information about a language at runtime (i.e. your java program can 
actually ask the SrcML framework, whether the program it is just working
with does support OOP or multiple inheritance, etc and perform the
neccessary operations then)

> >This implies you want translations from functional to procedural, which
> >could provide quite a bunch of problems when considering semantical
> >equivalence? (say lazy-evaluation, definition of infinite data  
> >structures,..)
> 
> It's still possible - lazy evaluation of expressions in language A  
> could be translated to language B, even if B doesn't have native  
> support for lazy operators, by using conditionals.
> Likewise a statically typed language can do dynamic type resolution and  
> function dispatch by 'adding in' the code to look types up at runtime,  
> and call the appropriate function based on runtime type information. Or  
> dynamic code can be statically bound, by static type analysis.
> MLML is just the abstract code format, it doesn't provide a processing  
> model, type system or any of the specifics necessary to actual execute  
> code.

This is kind of true. However here is another main difference between
o:XML/MLML and SrcML. We do intend to work with our result documents, but
when transforming a program from functional to procedural by such means
we will end up with an unreadable mostly auto-generated version of the
original code which does not really reflect the intentions of the original
code. Especially when you think about classes and transforming into
procedural languages you will lose all the OOD ideas. This is a prime
reason why we do not plan on having any such conversion process at all.

> MLML doesn't in itself aim to provide transformation methods or tools,  
> it's simply a code format that may be used for that purpose.
> o:XML is indeed meant to be hand-edited, MLML is not. o:XML is for  
> people like myself, who think that writing XML by hand is okay!

As written above SrcML is more of a framework and the code representation
within this framework is definitely not supposed to be hand-edited. Very
apparently our ideas of what is okay and what is not differ extremly when
it comes to editing source code in XML format ;)

> Good to see that there's work going on in this area.

There is in fact quite a lot of work going on in this area. Here's a small
collection of interesting links:

Another project called srcML from the Kent edu (our projects differ in the
goals a lot more than with o:XML/MLML):
http://www.sdml.info/projects/srcml/

An interesting paper with proposals on how code development will change:
http://www.third-bit.com/~gvwilson/xmlprog.html

Finally of course I'll have to mention our page too :)
http://www.srcml.de

If you read this far: Congratulations. It's over now.

With kind regards,
-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

User: n. The word computer professionals use when they mean "idiot."
(Dave Barry)