Re: Making NetBeans' Mercurial More Manageable

On 11/02/2011 01:00 PM, Tom Wheeler wrote:
> splitting the single repository (e.g. hg.netbeans.org/main)
> into multiple repositories, each of which might carry only the things
> relevant to a particular product (e.g. just the platform vs. all of
> the IDE).

Say, one repo per cluster. (*) The key question here is how to manage changes across repos. There are two possible directions:

1. Make each repo buildable in isolation given certain versions of binaries from upstream, and creating its own binary deliverable ready to run or be used for building 
downstream. This presupposes some Maven-like system of publishing and consuming binary artifacts - by actually converting sources to Maven format; or by using the 
<autoupdate> task (as in 7.1's $suite/nbproject/platform.xml) and also creating some tool for uploading NBMs (such as [1]) after some kind of build, plus some details 
like test-to-test dependencies. One big problem would be developing coordinated changes between modules that might span repos, such as API reviews; to test such changes, 
you would need to temporarily use some "snapshot" dependencies between the repos, but then roll them back and commit real changes in stages (#198781 [2] gives a flavor). 
Another problem would be changing numerous practices by quality and release engineers that all currently assume a monolithic source repository built at once and 
identified by one SCM revision. Benefits would include faster source scanning (**), especially for developers working on downstream repos; and being ready to accept 
features developed on a separate schedule possibly by teams using different infrastructure, which is currently very hard.

2. More conservatively, keep the current build system nearly intact, using Mercurial submodules for each repo, plus a special submodule for nbbuild, a top-level repo to 
synchronize them all. If you cloned the top repo, you would get the full source tree (submodules are not optional); but you could manually clone nbbuild as well as 
upstream repos like Platform in the same locations (without using the submodule feature) and still be able to do smaller builds. From such a small checkout you could push 
changes to individual repos but not "publish" them (that would require an action by someone with the full tree, which might also be more work for most developers). API 
reviews and the like would be somewhat more cumbersome since you would need to maintain multiple parallel patches, but probably easier than in scenario #1. The Hudson 
plugin would need to be fixed to fully support submodules. QE/RE processes would not be much affected, since an official build would still be identified by a changeset 
(that of the top synchronization module), though tools linking to code (e.g. bug update after daily builds) would have to be updated. The main objection to this scenario 
is that it would still be a significant disruption, yet has no chance of providing the long-term benefits that #1 could.

Discussion of any change like these is hypothetical for now, because there would be a substantial engineering cost that would directly compete with feature development 
and bug fixing for at least one release cycle, so management would have to be convinced that this is a top priority. I am guessing it would have to be justifiable based 
on benefit to paid (i.e. full-time) product development, since there is no guarantee that production-ready work from volunteers would suddenly take off.

An easy action would be to publish a read-only Subversion mirror from which you could check out and update a subset of sources. This might be enough for casual/infrequent 
contributors who are fine with running 'svn diff' and uploading to a bug report. They could not directly commit anything, creating diffs including file moves does not 
really work, and iterative reviews of patches is cumbersome.

> shallow clones, which
> only pull a relevant portion of history by default and bring the rest
> in as needed.

While this is a longstanding Mercurial wishlist item, it has not yet come to pass.

> the "Largefiles Extension".  As I
> understand it, this could mean that NetBeans' custom solution to
> binary file storage might not be necessary in the future.

Yes, I know about the Largefiles extension, and it could potentially supersede ExternalBinaries - assuming our hg.netbeans.org server, any mirrors such as BitBucket, and 
all developers are using 2.0+. It does not offer any major improvement over what we do now, but some convenience: easier to upload external binaries (hg add -b rather 
than the current web upload), and easier to download (hg up rather than running an Ant build, though the download-selected-extbins run by default is better because it 
omits bins from experimental modules). We would still be wary of anything resembling HgExternalBinaries, the original solution (a Hg extension), which turned out to be 
more trouble than it was worth: obscure corner cases of merging or patching could make Hg confused and abort, leaving the developer in an uncomfortable place. Presumably 
Largefiles would be more thoroughly tested for such scenarios, but I would not even consider it until it has had plenty of time to stabilize.

(*) Under the assumption that cluster boundaries are highly stable, so modules rarely need to be migrated; and that people would in practice want to retrieve sources at 
cluster boundaries rather than, for example, all Maven-related modules (currently in java, enterprise, apisupport, and profiler clusters). No clearly superior partition 
of modules is available at the moment. (The "plugins" shown in Plugin Manager depend on individual modules but these dependencies may overlap, so it is not a partition.)

(**) Scanning classes in a JAR is rather faster than scanning the source tree it came from. More importantly, a versioned JAR will rarely change, and requires only a 
single stat to check its timestamp, as opposed to a slow tree traversal for a timestamp check on a source tree.

[1] http://wiki.netbeans.org/AggregatingUC#Upload_process
[2] https://netbeans.org/bugzilla/show_bug.cgi?id=198781