Re: Cross-dependencies and output folders and more

On Tue, 2003-11-04 at 04:19, Bram Moolenaar wrote:
> Dobes Vandermeer wrote:
> 
> Phew, lots of questions.  Most of them don't have a clear "this is how
> you do it" answer.  There are many ways to do something, and which one
> works best depends on your situation.

Thanks for your patience!

> > I want to build my system, which is structured like this:
> > 
> > Libs/expat - Build a lib file
> > Libs/tomcrypt - Build a lib file
> > Libs/Utility - Build a lib file
> > Libs/ProtocolHandler - depends on Libs/Utility, Libs/tomcrypt,
> > Libs/expat
> > windows/ShellExtension - depends on Libs/ProtocolHandler and the others
> > by proxy
> 
> How do these dependencies work exactly?  If it's a dependency on the
> generated library itself it could be simple.  This also depends on
> whether you want to be able to build each library separately, or just
> build everything from the top.  Being able to build each part separately
> is more complicated.

Seeing that chart again I realize its a little dumb -- really the
dependency is only from the binaries to the libraries.  Currently I have
a dependency for the library folders on the libraries they use because
there are test programs inside the library folders which use the other
libs.  Although some of the header files in the libraries are generated,
currently this isn't integrated into the build system, so there isn't
really a dependency there.

> > The output binaries used for testing have to simulate the installation
> > tree because the binaries use their own location to find their
> > configuration & the jar files.  The installer builders we are using are
> > somewhat restrictive about where they will get their files from, so we
> > want the install folders to be specially laid out.
> 
> That sounds complicated.  If runtime files matter I mostly prefer to do
> testing after installing into a temp directory, so that it works exactly
> like a real install.  That means you also test the install procedure
> (not forgetting a file or putting it in the wrong place).

I think you described what I wanted to -- creating a full install tree
"somewhere" as part of the build.  I just don't want to have to type
"aap install" to get this effect.  I'd also want to be able to build
specific targets by name, and have to output appear in this same temp
install folder.

> > We currently use Jam, and its "grist" feature was worked out very well,
> > because we can have multiple targets with the same filename but a
> > different "grist".  
> > How is this done in aap?  
> > Would I use :publish or :install?  And if so, can I have this done even
> > when not running 'aap install'?
> 
> What does "grist" mean?  I looked at the Jam docs but it's not clear to
> me how it works.  In Aap you can have real targets, which are identified
> by the file name, and Virtual targets, which can be anything.  I don't
> see how you can have multiple targets with the same file name, they
> would overwrite each other.  Or do you mean targets in different
> directories or with different variants?

Sorry, I think I've gotten myself a little confused because of these
jam-isms.  Of course you could specify different targets by using a
different path instead of using a different "grist".

I'll explain grist and what makes it useful, then you might have a peek
into my twisted view: file targets can have a prefix of the form <grist>
-- examples might be <installed>libz.a <built>libz.a, etc.  This allows
you to distinguish between "different" files with the same name.  This
is then combined with jam's LOCATE and SEARCH variables.  When jam
expands variables in shell commands, it deletes the grist and inserts
the full path to the file, based on either LOCATE or SEARCH for that
target.  So, if you have to copy files around you give each copy its own
"grist".  This feature is used for many purposes, such as distinguishing
two source files with the same name, distinguishing multiple copies of
the same file, and finding header files (using SEARCH).  Its handy
because your rules can more easily predict the "grist" of a file than
its path, and you can bind the actual locations late.

So, it never occurred to me until just now I could use the path to the
target instead of a grist.  I don't really know whether grist is really
worth it or not.

> If it's about making sure that files in the right place for testing then
> the simplest solution is often to simply copy the files with ":cp".
> Doing this with dependencies can be uneccessary complicated.

Yeah... I suppose you locate the source file using $BDIR and the output
file using some variable of your choice?

Could I make a rule like ":cptobin" that would find the file and copy it
to the correct "bin" folder?

> > How should I handle the cross-tree dependencies?  We I use:
> > 
> > Libs/ProtocolHandler/main.app:
> > :include {once} ../expat
> > :include {once} ../tomcrypt
> > 
> > Or maybe:
> > :child ../expat
> > :child ../tomcrypt
> 
> For dependencies you should use ":child".  But this is normally used from
> the parent recipe in the top directory.  Apparently you want to build
> the ProtocolHandler by itself, and make sure the libraries it depends on
> are build first.  

Exactly.

> Although ":child ../expat" will work, this will cause
> trouble when building everything.  Hmm, apparently Aap doesn't handle
> this yet.  It would require a bit of Python:

OK.. well thats something for you to chew on I suppose!

> > In our jam system, we have some top level files, Jamrules and
> > Jamrules.local, that allow us to specify useful variables and build
> > actions.
> > 
> > In aap, you seem to be recommending that we put:
> > 
> > include ../../common.aap
> > 
> > Since we know every file will have exactly the same line, couldn't we
> > imply insert that line?  In jam, you say:
> > 
> > SubDir TOP Libs expat ;
> > 
> > And it reverse-engineers the location of Jamrules and loads it.
> 
> You can use $TOP, but that won't work if you only build a sub-project.
> 
> > SCons is even better -- it searches upwards for the top build script for
> > you, and loads it.  
> 
> There have been ideas to add something like this to Aap.  It hasn't been
> implemented yet.  A disadvanage of SCons is that it's too easy to end up
> using the wrong build script without a warning.

Yeah, that does seem like a risk -- at list with the jam way you can
predict which tree you'll load from.

> > Does aap have any kind of automatic tree detection?  It would be great
> > if I could omit not only the :include ../../common.app, but also any
> > lines for cross-tree dependencies (other than a line saying that this
> > executable links with that library, or this library uses that library so
> > link that one if you link this one).
> 
> Currently it only works if you build from the toplevel.  I agree it
> would be nice to add support for building sub-projects and still use the
> dependencies that were defined at the toplevel.

Perhaps instead of running aap I could write a script that hunts upwards
for the top-level main.aap, and runs aap there with a computed target
based on the path?  What if I specified a specific target when building
in the subdirectory?

> > Some of my source code is generated as part of the build process, and
> > the generator is also built in the build tree.   How do I tell aap how
> > to run my tool to generate those source files?  Is it ":action build
> > <whatever> ?"
> 
> Just specify the dependencies, e.g. (sketchy):
> 
> 	file.c : file.c.in  generator
> 		:sys generator <$source >$target
> 
> 	generator : generator.c
> 		:do build $source
> 
> Aap will build "generator" before invoking it, when necessary.

OK, thats very cool.

> > With Jam, I can specify new rules, like "MakeNSISInstaller".  I can
> > apply this rule to build multiple different installers with different
> > input and output files.  Can I create an equivalent ":makensisinstaller
> > xxx : yyy" in aas?  Or how is this meant to be done?
> 
> It depends on what kind of rule you want to make.  You can use a ":rule"
> if you want to match the file name with a pattern.  But you probably
> want to use ":produce".  Alternatively, you can use actions, since they
> are selected on file type.

So:

:filetype
        declare nsis_installer

:produce nsis_installer server-$version.exe : server.nsi

:action build nsis_installer
	:sys makensis $source etc.

is this the way this should be done?  That seems OK to me.

> > BTW, these might make good FAQ entries (even if they are in the
> > documentation); I'm probably not the only one.
> > 
> > I'd like to use aap for my project, but I'm not a Python programmer
> > (yet) so I can't really hack it up to add these features if they aren't
> > there.
> 
> Your project sounds like a quite complicated one with specific demands.

Well, it doesn't seem that complicated -- really the main trouble is
that the lib files are built in a different folder from the binaries
that use them, and its not a sub-folder (because there are multiple
binaries).  We managed to modify jam, at least, so that it allowed us to
build in a folder and have it rebuild any outdated libs used by binaries
in that folder.  Maybe thats too much -- I suppose I could try and live
with having to build at the toplevel to update the libraries... or
something.

> You probably run into a few things that have not been implemented yet.
> First thing is then to find out what you actually wanted to do before
> thinking of possible solutions.

Well, we already have a jam-based build system, so I'm hoping to clear
out the few annoying quirks of jam, which are:

- Doesnt rebuild when the build commands change
- Doesnt support using checksums
- Doesnt support caching (pretty minor, though)
- Uses the native shell to run commands, so you get command-line-length
problems, backslash vs forward slash problems (aggravated by cygwin use,
where you sometimes want both).
- Doesn't include platform-independent builtin commands (e.g. move,
copy, zip, unzip, see aap command list) so you have to use
platform-specific ones, with platform-specific flags.
- Has lots of little quirks, like:
    - jam clean also cleans the sibling folders you depend on
    - sometimes you want to build a specific target, but that target has
a grist that is too annoying to type in.
- Doesn't build java with dependencies (but nobody else does, either)
 
So far it seems like aap and SCons are the best -- but SCons' syntax is
way too ugly and verbose.  Its starting to look like aap still has some
maturing to do in the area of building large projects but its feature
list is pretty cool.  I'm curious to experiment with this stuff where it
downloads or cvs checkouts other projects and builds them so far ctags
worked for me but subversion didn't.  Nevertheless it would be neat to
be building my own project and have it connect to sourceforge and
download new versions of the libs I use!





-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive?  Does it
help you create better code?   SHARE THE LOVE, and help us help
YOU!  Click Here: http://sourceforge.net/donate/