Re: Release process (was Re: DarwinPorts 1.3.1)

Le 2 août 06 à 15:16, Jordan K. Hubbard a écrit :

>
> On Aug 1, 2006, at 9:55 PM, Paul Guyot wrote:
>
>> The recent fiascos of releases 1.3 and 1.3.1 show indeed that  
>> there is something wrong with the release process.
>> But it is not (only) a matter of the timing of release. It's  
>> mostly a matter of testing. [ ... ]
>
> And here, I agree with you 100%.  Where I think you go off in the  
> wrong direction is with the rest of the points you make.
>
> I'm going to sound like a broken record here, but the real problem  
> is that the DarwinPorts project simply doesn't eat its own  
> dogfood.  It relies on the user community to do that for it, and no  
> number of release candidates or changes to how the bits are  
> actually released (CVS, etc) will change that one bit.  Yes,  
> developers make terrible testers, but for a collection of 3400+  
> ports coupled with 40,000 lines of Tcl and C code supporting it,  
> users make even worse testers because there's no way they can  
> consistently cover all of that ground even once, much less on a  
> regular basis, and the end result is a total hit-and-miss journey  
> through the collection as users stumble across broken ports more or  
> less at random (or test the really popular ports and ignore the rest).
>
> DarwinPorts will never be more than an intellectual curiosity until  
> it evolves the infrastructure for clean-room building each and  
> every port on a regular basis, with known combinations of "base"  
> bits, and resolving (or at least reporting) all of the errors  
> encountered.   DarwinPorts is not a paint program or a word  
> processor, and you can't get comprehensive testing of it just by  
> releasing it to the user community and letting them bang on it for  
> a while.  It's something which requires an entirely different level  
> of testing (and has never had).

There is a little misunderstanding here.
I mentioned testing base/, not testing in general. The 1.3.x fiascos  
were caused by the lack of testing of HEAD base/ code.

Testing base/ can partically be done automatically with unit-like  
tests such as the ones in base/tests/. I personally check they all  
pass before doing any commit and try to create more tests to improve  
this. I wish other developers of base/ would do the same. However,  
considering all the possible installations, we cannot fully test  
base/ automatically. base/ is like a word processor as you say.  
Testing all ports will not mean a full test of base/. For example,  
three of the six tests test properties that cannot (or should not) be  
found in real portfiles: test of using two environment variables in  
extra_env, test of violations from a port caught by the trace mode  
and test of the xcodeversion.

Nevertheless, I wholeheartedly agree with your point about testing  
the ports: we need a real infrastructure. In case you didn't notice,  
the recent changes to the trace option are exactly intended for that  
purpose. With the -t option, you no longer need to use the union disk  
image trick you (?) came up with when you tried to automate the  
process. Ports cannot write outside the sandbox at run time. If -t is  
always used, /opt/local only contain files copied during the  
activation phase. Combined with the automatic detection of  
dependencies (still option -t), the (almost) full test of all ports  
can be done in a much simpler and less expensive way. There are a  
couple of additional structural changes to get this working  
completely (for example to drop the bin: and the lib: dependencies in  
favor of the port: dependency, or include a conflict key to portfiles  
as you mentioned recently), but we've never been that close to the  
goal software-wise (which is as much I can do as a base/ committer).

Paul
-- 
Ministre ultraplénipotentiaire en disponibilité.
Mobile. Sans baignoire fixe.
http://www.kallisys.com/
http://www-poleia.lip6.fr/~guyot/