testing and landing the new packaging system (bug 20640)

If you are on the "to" list of this post, I am requesting your specific 
help in some way. If you are "cc"ed, you should at least be aware of the 
changes that are coming. I am also cross-posting to various newgroups. 
Please post substantive replies to npm.xpinstall, and specific 
fixes/patches to bug 20640.

After my posting last month about the making the packaging system 
cross-platform, and additional discussions with Dan Veditz and others, I 
have created a mozilla branch tag and done significant work on the 
packaging system. This work is now mostly completed and ready for 
review. The branch tag is PACKAGING_20030906_BRANCH, and it is a 
mini-branch (only files changed from the CVS trunk have been tagged). 
There is a matching PACKAGING_20030906_BASE rolling base tag so that I 
can merge trunk changes onto the branch as necessary.

The new packaging system is designed to entirely replace the embedding 
and packaging manifests that currently live in embedding/config and 
xpinstall/packager. The new system allows packages to responsd more 
effectively to configuration options, and gives Makefiles complete 
control over packages.

The master-list of packages that can be built is 
build/package/packages.list  It contains a set of dependencies that 
allow, for example, the GRE XPI package to include NSPR, XPCOM, gecko, 
and the install.js file in one package.

During the "export" phase of building, each Makefile has the opportunity 
to export one or more package files (I have given them the extension 
.pkg) by setting the PACKAGE_FILE variable. Each package file is then 
preprocessed by Hixie's perl preprocessor. Various Makefile variables 
are automatically defined for the preprocessor, and others can be 
defined as necessary by setting PACKAGE_VARS or PACKAGE_DEFINES. See 
config/rules.mk (the realpackages: target). Each package file is 
exported to dist/packages/$(MODULE) for use later.

Each package file has the format:
[package-name package-name]
dist/bin/file.to.package
dist/include/file.to.package
!specialcommand arguments
!xpt dist/bin/components/necko_cookie.xpt

[other-package]
more-files

Lines beginning with ! are "command lines" that have special meaning... 
for example, all the !xpt files can be merged together at packaging 
time. The following commands have been defined:
!xpt !optional !exec !preprocess !touch

After the tree has been built, the "packaging system" can parse the 
files in dist/packages and build packages of various kinds. For example, 
it can take the "gre-xpi" package and ZIP it up into an XPI. The 
packaging system can also make gzipped and bzipped tarballs. I also 
*hope* that it can create macintosh DMG files, but see below.

Once I completed this work, I reworked the perl scripts that create the 
installers on windows, unix, and OS2. I haven't touched 
xpinstall/packager/mac because it appears to be unused.

I have tested (reasonably thoroughly) the win32 and linux tarballs and 
installers created by my branch. They appear to work as expected. In 
addition, I have reduced the size of the win32 installer by 1.2 Meg, by 
using the zip -9 flag. NOTE: I have only tested "ordinary" 
configurations such as those that mozilla.org releases. I do intend to 
support special configuration options such as static builds, etc, in the 
future, but my main goal was to duplicate existing functionality and 
fill the rest in later.

***
SO, here's what I need:
***

Review by Dan Veditz. The changes are so pervasive that I haven't even 
tried to make a patch, it would be too long for bugzilla anyway. 
Instead, can I just get you to review the changes on the branch and 
using bonsai: 
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=PACKAGING_20030906_BRANCH&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=all&mindate=&maxdate=&cvsroot=%2Fcvsroot

Review or at least MOA by bryner and/or leaf. The changes to the build 
system are significant but don't have real side-effects.

I have landed various extraneous patches on this branch for testing 
purposes: these bugs/patches are posted separately:
bug 216366
bug 219233
bug 200091

The results from this system ought to match the results from the old 
packaging system fairly closely: but I have made the following changes:
Moved jpeg.dll from xpfe-browser into embedding/GRE (controlled by the 
--with-system-
jpeg flag, of course)
Moved xome XPT files into the embedding/GRE packages, to match the 
components that ar
e shipped there.
Moved wallet from the GRE into the xpfe-browser package and embedding 
packages.
moved nsAxSecurityPolicy.js to embedding/gre if it is being built
The packaging system no longer supports flat chrome, only jar-packaged 
chrome. This m
ay be re-added at a future point, by integrating jar.mn and the 
packaging scripts.


I have not tested either OS/2 or macintosh, and I can guarantee that 
there will need to be platform-specific tweaking. I made my best guess 
at the OS/2 perl scripts to build the installer, but I'm not even sure 
they compile. mkaply has said he'll look at the OS/2 stuff, and I need 
to find someone who is willing to build this branch on mac and tell me 
what doesn't work.

The testing is two-part. The first part is making sure that the packages 
contain the correct files. This is best accomplished
(from objdir/build/package)
make dump-packages PACKAGE_FLAGS=-w
In particular watch out for "warning: missing file" and the "unpackaged 
files" list at the end.

The second part is actually creating the tarballs/installers and making 
sure that they run correctly. To stage all of seamonkey to a test directory:
(from objdir/build/package)
make stage-seamonkeyall-enUS STAGE_DIR=/path/outside-of-tree

Leaf, could you let me know how you want to control this tarball process 
for nightly builds and releases? The makefile in build/package is very 
flexible, and I think I can make it suit your every need with not much 
hackery.

The following items are NOT YET COMPLETE on the packaging branch:
The MFCEmbed installer scripts. This shouldn't take me long, I'm 
essentially copying what I did in the other installer dirs.
Static builds in embedding/browser/gtk/src are broken. I have a plan, 
it's just not done yet.

I am working on documentation... there are many interacting pieces of 
this puzzle and I need to provide a good overview. I will check it in 
somewhere on the mozilla website when I get a chance.

--BDS