RE: Request for comments: CONS specification
Søren Mou Jakobsen <[email protected]> Sun, 30 May 2004 21:22:36 +0200
| Newsgroups | gmane.comp.gnu.cons.general |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format.
--===============2140672739==
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0004_01C4468C.3DA62180"
This is a multi-part message in MIME format.
------=_NextPart_000_0004_01C4468C.3DA62180
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi all
=20
I have used cons in various employments through the last four years.
Developing a new extensible version of cons is a good idea. In fact that =
has
been proposed several times on this mailing list, but no project has =
taken
of yet. I=92m willing to pitch in to get things moving.
=20
I have recently implemented an alternative to cons called qs (named so
because it=92s quick to type). Qs was developed to counter some of the
deficiencies discussed recently, and we=92re currently migrating to this =
at
our current workplace. We might be able to use some of the ideas and
possibly code behind qs to get things accelerated. Currently qs is =
strictly
a core build engine. It has a very simply interface to build up a =
dependency
graph and then make a number of targets up-to-date. It doesn=92t know =
anything
about file types, how to build certain things (like TeX file), etc. This =
is
something that could be built on top of qs.=20
=20
>From an architectural point of view, I believe it makes sense to think =
of
the new CONS as at least two layers: A core build engine which should be =
as
simple and optimized as possible and modules on top of the build engine
which contains the =93expert=94 systems which knows how to build =
different
targets. The expert modules will then build up the dependency graph =
though a
simple interface into the core build engine. What I=92m proposing is to
migrate qs into the new cons as the core build engine. This =
architectural
division will also ensure that users will always be able to bypass the =
upper
layer to do whatever they want.
=20
The current version of qs is about 1100 lines of perl. It=92s well =
documented,
object oriented and supports parallel builds (with a caveat). The basic
interface to qs is very simple. It got two functions: Qs::node which =
defines
a new node in the build graph and Qs::edge which add one or more
dependencies between nodes and a command to make the source nodes from =
the
target nodes.
=20
Qs supports the plug-in of different types of nodes. Each inherits from =
an
abstract Node class. This means that the dependency graph can depend on
regular file nodes, files in cvs, perl references, rows in a database, =
etc.
Edges are also plug-in objects which inherits from an abstract Edge =
class.
Obviously the most used edge class is the Cmd edge which executes a =
program
in the shell. Also supported could be edges which build by calling a
reference to a perl sub function, edges which execute sql queries or =
runs
heavy-duty processing on build servers, etc.
=20
Qs support parallel building, but it doesn=92t work at the moment =
because it
turned out that mutex protection of perl objects is currently =
unsupported.
The documentation says object mutexes are work in progress. There are
properly also some alternatives to implementing this. The main point is =
that
qs were designed with parallelization in mind.
=20
In a previous mail the problem of not knowing the exact dependencies in =
a
build tree was discussed in relation to building TeX documents. I=92ve =
had
major problems with this particular problem. I work at a company which
created application for interactive television. The build process is =
usually
very complicated and can contain up to ten build steps from sources to =
final
target. We generate a lot of C code and headers during the build =
process.
Those files can=92t be dependency scanned when a project is built from
scratch. Also when a project is built the second time the dependency =
scan is
sometimes wrong because the previous versions of C and header files are
scanned. It was suggested in a mail to add the ability to modify the
dependency graph during the build process. I believe this is a good
solution. Qs supports this by a late-dependency-scan callback function. =
This
example of a C file compilation illustrates how it works:
=20
Qs::edge FileNode(=91foo=92), FileNode(=91foo.c=92), =91gcc =85=92; # =
foo is built from
foo.c
Qs::scan FileNode(=91foo=92), \&c_dependency_scan; # asks qs to call
c_dependecy_scan right before it=92s about to build foo
=20
sub c_dependency_scan
{
my $source =3D shift;
my $target =3D shift
=20
my @dependencies =3D =85 do actual dependency scan of source =85
=20
# modify the dependency graph during the build process! The =
dependencies
will be made up-to-date before foo is built
Qs::edge $target, @dependencies;
}
=20
As a proof of concept for qs I made also made some node and edges =
classes
which provides a back wards compatibility with cons. This means that =
good
old construct files were parsed and a dependency graph was build up =
using
the qs interface. It might be interesting to complete this so that the =
new
cons is actually contains an export builder which gives back wards
compatibility with old project. This could ease the transition from old =
cons
to new cons.
=20
Another area I would like to improve is the debugging of the dependency
graph. This can be hard with cons if the dependency graph is very large =
and
complex. One thing qs supports now is to output the dependency graph as
HTML. All sources are targets are hyperlinked so that the user can =
easily
click through dependencies. It would also be possible to use the perl =
module
which outputs graph to see a visual presentation of (subsets of) the
dependency graph.=20
=20
It was also suggested earlier to add native support for build =
clustering.
That would be a very cool feature.
=20
Kind regards,
S=F8ren Mou Jakobsen
=20
=20
=20
=20
=20
------=_NextPart_000_0004_01C4468C.3DA62180
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Diso-8859-1">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
name=3D"place"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
name=3D"State"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:Arial;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3DEN-US link=3Dblue vlink=3Dpurple>
<div class=3DSection1>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Hi all<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>I have used cons in various employments through the =
last four
years. Developing a new extensible version of cons is a good idea. In =
fact that
has been proposed several times on this mailing list, but no project has =
taken
of yet. I’m willing to pitch in to get things =
moving.<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>I have recently implemented an alternative to cons =
called qs
(named so because it’s <font color=3Dnavy><span =
style=3D'color:navy'>quick</span></font>
to <font color=3Dnavy><span style=3D'color:navy'>type</span></font>). Qs =
was
developed to counter some of the deficiencies discussed recently, and
we’re currently migrating to this at our current workplace. We =
might be
able to use some of the ideas and possibly code behind qs to get things
accelerated. Currently qs is strictly a core build engine. It has a very =
simply
interface to build up a dependency graph and then make a number of =
targets
up-to-date. It doesn’t know anything about file types, how to =
build certain
things (like <st1:place w:st=3D"on"><st1:State =
w:st=3D"on">TeX</st1:State></st1:place>
file), etc. This is something that could be built on top of qs. =
<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>From an architectural point of view, I believe it =
makes
sense to think of the new CONS as at least two layers: A core build =
engine
which should be as simple and optimized as possible and modules on top =
of the
build engine which contains the “expert” systems which knows =
how to
build different targets. The expert modules will then build up the =
dependency
graph though a simple interface into the core build engine. What =
I’m proposing
is to migrate qs into the new cons as the core build engine. This =
architectural
division will also ensure that users will always be able to bypass the =
upper
layer to do whatever they want.<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>The current version of qs is about 1100 lines of =
perl.
It’s well documented, object oriented and supports parallel builds =
(with
a caveat). The basic interface to qs is very simple. It got two =
functions:
Qs::node which defines a new node in the build graph and Qs::edge which =
add one
or more dependencies between nodes and a command to make the source =
nodes from
the target nodes.<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Qs supports the plug-in of different types of nodes. =
Each inherits
from an abstract Node class. This means that the dependency graph can =
<font
color=3Dnavy><span style=3D'color:navy'>depend on</span></font> regular =
file nodes,
files in cvs, perl references, rows in a database, etc. Edges are also =
plug-in
objects which inherits from an abstract Edge class. Obviously the most =
used
edge class is the Cmd edge which executes a program in the shell. Also
supported could be edges which build by calling a reference to a perl =
sub
function, edges which execute sql queries or runs heavy-duty processing =
on
build servers, etc.<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Qs support parallel building, but =
it
doesn’t work at the moment because it turned out that mutex =
protection of
perl objects is currently unsupported. The documentation says object =
mutexes
are work in progress. There are properly also some alternatives to =
implementing
this. The main point is that qs were designed with parallelization in =
mind.<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>In a previous mail the problem of =
not
knowing the exact dependencies in a build tree was discussed in relation =
to
building <st1:State w:st=3D"on"><st1:place =
w:st=3D"on">TeX</st1:place></st1:State>
documents. I’ve had major problems with this particular problem. I =
work
at a company which created application for interactive television. The =
build
process is usually very complicated and can contain up to ten build =
steps from
sources to final target. We generate a lot of C code and headers during =
the
build process. Those files can’t be dependency scanned when a =
project is
built from scratch. Also when a project is built the second time the =
dependency
scan is sometimes wrong because the previous versions of C and header =
files are
scanned. It was suggested in a mail to add the ability to modify the =
dependency
graph during the build process. I believe this is a good solution. Qs =
supports
this by a late-dependency-scan callback function. This example of a C =
file
compilation illustrates how it works:<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Qs::edge =
FileNode(‘foo’),
FileNode(‘foo.c’), ‘gcc …’; # foo is built =
from
foo.c<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Qs::scan =
FileNode(‘foo’),
\&c_dependency_scan; # asks qs to call c_dependecy_scan right before =
it’s
about to build foo<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>sub =
c_dependency_scan<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>{<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>=A0 my $source =3D =
shift;<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>=A0 my $target =3D =
shift<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>=A0 my @dependencies =3D … do =
actual
dependency scan of source …<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>=A0 # modify the dependency graph =
during the
build process! The dependencies will be made up-to-date before foo is =
built<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>=A0 Qs::edge $target, =
@dependencies;<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>}<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>As a proof of concept for qs I made =
also
made some node and edges classes which provides a back wards =
compatibility with
cons. This means that good old construct files were parsed and a =
dependency
graph was build up using the qs interface. It might be interesting to =
complete
this so that the new cons is actually contains an export builder which =
gives
back wards compatibility with old project. This could ease the =
transition from
old cons to new cons.<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'>Another area I would like to =
improve is
the debugging of the dependency graph. This can be hard with cons if the
dependency graph is very large and complex. One thing qs supports now is =
to
output the dependency graph as HTML. All sources are targets are =
hyperlinked so
that the user can easily click through dependencies. It would also be =
possible
to use the perl module which outputs graph to see a visual presentation =
of
(subsets of) the dependency graph. <o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>It was also suggested earlier to add native support =
for
build clustering. That would be a very cool =
feature.<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>Kind regards,<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'>S=F8ren Mou Jakobsen<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
</div>
</body>
</html>
------=_NextPart_000_0004_01C4468C.3DA62180--
--===============2140672739==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
[email protected]
http://lists.gnu.org/mailman/listinfo/cons-discuss
Cons URL: http://www.dsmit.com/cons/
--===============2140672739==--