Release process (was Re: DarwinPorts 1.3.1)

Le 2 août 06 à 11:48, Landon Fuller a écrit :

> If the release process isn't ferreting out bugs, or is not agile  
> enough to incorporate fixes quickly enough, do you have a  
> suggestion for improving the situation?
> Telling users to switch to CVS for the latest changes negates the  
> purpose of doing releases at all, which was to shield users from  
> unstable code in HEAD.

The recent fiascos of releases 1.3 and 1.3.1 show indeed that there  
is something wrong with the release process.
But it is not (only) a matter of the timing of release. It's mostly a  
matter of testing.

And this problem dates back to the introduction of selfupdate.

In an ideal selfupdate world, you have two classes of users:
- selfupdate users, who just get new versions when a new release  
comes out
- CVS users, who are ideally limited to the developers since all  
users should use selfupdate.

This is flawed because there isn't enough testing of base/. The very  
nature of Tcl requires a lot of testing (compared to compiled  
language, I'm not even talking about ML languages here) and the very  
nature of DP makes testing even more important. And in an ideal  
selfupdate world, testing of base/ is made by developers.

Developers:
- are savvy enough to test base/ and solve problems themselves when  
they encounter a problem which is strictly related to their  
installation. The problem should be avoided in the first place  
because users don't always realize the problem comes from their  
installation.
- use more recent OS versions and more recent hardware than users.  
darwintrace.c has been broken on 10.3 for a couple of week, no one  
noticed until jmpp tried to build the dmg for the 1.3 release.
- believe in DP principles. For example, they are believers that we  
should not have anything in /usr/local. The recent 1.3.1 fiasco shows  
that it's not the case from our user base. Another example: instead  
of using an installer for some software, such as i-packages for TeX,  
developers, unlike users, will develop the missing ports to have  
everything in /opt/local.
- might do things that work on their box but not on a selfupdate  
install. For example, the ${worksrcpath} is much larger with  
selfupdate/rsync and hdiutil doesn't like it. I've been bitten by  
this one for XeTeX because I committed a port that worked perfectly  
well on my box but failed on every selfupdate installs.
- want to take advantage of base/ new features in HEAD without always  
realizing that they're not available in the release version. There is  
no way for them to commit ports that use HEAD features and avoid  
failures on selfupdate-based installs.

The last point is pretty much important. How can we test a new  
feature that requires a change in a Portfile making it incompatible  
with the selfupdate version?

In a nutshell, developers are really bad testers because they're not  
representative of our user base, the one who actually faces problem.  
Consequently, when the release time comes, the code isn't ready and  
polished enough. And we encounter fiascos like the 1.3 series.

You're asking for solutions, as if there was nothing better than  
selfupdate.
I strongly disagree. There is an alternative. Actually, besides Paul  
Beard's suggestion, I can see three easy solutions that scale well  
with the size of the user base and the developer base.

Solution #1 is a release candidate or beta version.
This is the less appealing one, but what James and Juan came up with.  
I think if it's just a CVS branch or tag, it won't just work because  
no one who actually doesn't use HEAD will test it. I don't rule it  
out if it's a dmg that is advertised exactly everywhere the full  
release is advertised. Of course, the selfupdate feature of this one  
should be disabled to avoid users to revert to the previous release  
when they install the release candidate.

Solution #2 is encouraging users who hook up here to use CVS. This is  
the best short-term solution. This enlarges the testing user base and  
specifically targets people who will report any problem here. The  
bugs can then be quickly fixed.

Solution #3 is reverting to pure CVS. We should never have migrated  
to rsync+selfupdate in the first place. Non-savvy users don't update  
their installation when something changes in base/, so they're  
protected from unstable code in HEAD (if any, please let me remind  
you that heavy-development work happens on branches), and savvy do.  
The only drawback is the load and, considering the end of opendarwin,  
I think we should not revert to CVS now unless we find a host ready  
to cope with the traffic/box load.

Paul
-- 
Ministre ultraplénipotentiaire en disponibilité.
Mobile. Sans baignoire fixe.
http://www.kallisys.com/
http://www-poleia.lip6.fr/~guyot/