Re: Developing frameworks and applications in tandem and managing dependencies

Below is a long thinking-out-loud kind of a discussion. tl;dr: only use
svn externals definitions to point at immutable release tags or at
pegged revision numbers.

On 2012-07-31 19:23, Anna-Jayne Metcalfe wrote:
>> > Are you using Subversion as your version control system?
> I'm afraid not. The scheme we use should work with any SCC that supports
> shared files or projects, though.

Subversion has a feature called svn externals which might be like what
you refer to as 'shared projects' but I'm not sure... (What SCC system
are you using btw?) You can use an svn externals definition to link in
directories from projects in other Subversion repositories (or in the
same repository).

The problem with svn externals is this. Say I'm developing app-a and I
have it in my svn repository at repo/app-a/trunk. I can checkout trunk
to a working copy directory say wc/app-a. I might then have some
libraries that app-a uses and I'll put them in a subdirectory
wc/app-a/lib. I can add an svn externals property on the wc/app-a/lib
directory that says to use repo/myframework/trunk and to put that in the
wc/app-a/lib/myframework directory. Then I'll be working on the 'trunk'
of app-a and the 'trunk' of myframework. I think 'trunk' is analogous to
what you referred to as your 'Dev' branch.

This is a great set up at first because it gives me exactly what I want.
The ability to edit files in wc/app-a for the purposes of developing my
app's functionality and also the ability to directly edit the files in
wc/app-a/lib/myframwork. In both cases edits to any of those files would
be checked back in to the 'trunk' (Dev branch) of the respective projects.

The problem happens when I create a 'tag' (what I think you are calling
a 'Release' branch) of app-a after I do a release of it. What happens
then is that a snapshot of repo/app-a/trunk gets made in
repo/app-a/tags/version/0.1.0. So if I want the definitive code for
version 0.1.0 I can look in the svn repository at
repo/app-a/tags/version/0.1.0. After I've done such a release I might
make repo/app-a/trunk then be the 0.2 version which hasn't yet been
released. The problem is that when I tagged v0.1.0 in the repository the
svn externals definition for the library that was being used by app-a
0.1.0 still has repo/app-a/tags/version/0.1.0/lib/myframework pointing
at the trunk of myframework at repo/myframework/trunk.

So if at some point in the future I want the code for the app-a 0.1.0
release I might try to checkout repo/app-a/tags/version/0.1.0 into a
working copy somewhere, say wc/app-a-0.1.0. That will give me the
application code for app-a as it was at version 0.1.0 but the svn
externals definition on repo/app-a/tags/version/0.1.0/lib will still
have an svn externals property saying that the relevant version of the
myframework code is at repo/myframework/trunk. So I'll get the released
code of app-a and the latest myframework code, not the myframework code
that was used when app-a 0.1.0 was released.

It would seem that a viable solution to the problem would be that prior
to releasing app-a the myframework library was released first. Releasing
myframework would mean that the development branch of myframework in
repo/myframework/trunk got tagged as a release in
repo/myframework/tags/version/0.1.0 and then after that the svn
externals definition on repo/app-a/trunk/lib got updated to point at
repo/myframework/tags/version/0.1.0 rather than repo/myframework/trunk.

Then if I was to checkout the release of app-a v0.1.0 to wc/app-a-0.1.0
the wc/app-a-0.1.0/lib/myframework directory would contain not the
latest version of the myframework code but the code as it was for
myframework v0.1.0.

The above just discusses 'tags' and not branches, and with branches
things get more complicated still. Say for instance I released v0.1.0 of
app-a and then v0.2.0 of app-a which is not backwards compatible with
v0.1.0. Some of my customers are running v0.1.0 and some v0.2.0. Then I
discover a security vulnerability (hey, could happen) that affects both
v0.1.0 and v0.2.0 of app-a. Let's say I'm still offering support for
both the 0.1 and 0.2 versions, so I need to create v0.1.1 and v0.2.1
that patch the security vulnerability. I then discover that in addition
to needing 'tags' to snapshot my codebase at a particular point in time
I also need 'branches' in which I can continue development of minor
versions. So I might have repo/app-a/branches/0.1 and
repo/app-a/branches/0.2 while v0.3 is being developed in
repo/app-a/trunk. There remains the question of what to do with the svn
externals definition in repo/app-a/branches/0.1/lib for the myframework
library. Should I point that externals definition to the trunk of
myframework (repo/myframework/trunk), to a minor version branch of
myframework (repo/myframework/branch/0.1), or to a release tag of
myframework (repo/myframework/tags/version/0.1.0)?

It all starts to seem to get too hard and a copy and paste approach
seems like it might be the best bet. As a yard stick I look around at
what other successful projects do and it seems to me that most people
take the practical approach of just copying and pasting in a particular
version of a framework or library. Then if they need new functionality
from the framework or library they just overwrite their copied and
pasted version with an updated version of the framework or library.

So having had this discussion (and thanks to those who have played along
while I thought out loud) I think I've decided what will work for me.

I will develop a library/framework called myframework and keep it in my
svn repository in repo/myframework. As part of the myframework project
will be unit tests which run over various features and a reference
application which utilise the features of the application. So I'll be
able to run the reference application and develop framework features
with respect to the reference application. When I'm happy that the
development version of myframework in repo/trunk/myframework is release
worthy I will do a "release" of the software. A "release" means that
repo/myframework/trunk gets branched to repo/myframework/branches/0.1
(where backwards compatible fixes can be done if necessary in
maintenance mode) and gets tagged to repo/myframework/tags/0.1.0 (where
an immutable copy of the entire library is kept forever.

I will then develop any number of applications that use myframework. For
example repo/app-a and repo/app-b. Because app-a and app-b depend on
myframework they will have svn externals definitions in their lib
subdirectories so that both repo/app-a/lib/myframework and
repo/app-b/lib/myframework are references to some immutable release tag
of myframework, for instance repo/myframework/tags/version/0.1.0.

Then if an application I'm developing calls for a new feature of
myframework to be developed I will develop the feature along with unit
tests and reference implementation in repo/myframework/trunk. Then when
I'm happy that the new feature is functional I will do a new release of
myframework say to repo/myframework/tags/version/0.2.0 and then I will
update the svn externals properties in app-a or app-b as necessary to
point to the latest development tag.

In short what I think I've decided having had this discussion is that
when using svn externals definitions I will always either peg a revision
number or set the externals to an immutable tag.

This means that when I'm working on app-a and I find that I'm missing a
feature that I would like in myframework I can't just edit the source
code in wc/app-a/lib/myframework because that code is an immutable
snapshot of a particular version of myframework. Rather I will have to
do the development in wc/myframework, then release myframework, then
update the svn externals definition for wc/app-a/lib/myframework to
point at the latest useful release.

This is a little more labour intensive than I would have liked, but I
think it's a reasonable and robust enough process to pursue. Such a
process would certainly be an improvement on the disaster I have at the
moment wherein all of my svn externals definitions point to trunk and my
software spends more of its time broken than functional.