Re: Re: Known working sets II [was: Eggification redux]

Dieter Maurer wrote:
> Martijn Faassen wrote at 2007-9-25 19:57 +0200:
>> ...
>> If you choose for flexibility first, people will need to think about
>> versions all the time.
> 
> I follow Tres argumentation: somehow the Linux distributors
> have this problem mostly solved:

While I don't dispute we should look at package management systems, they 
don't have *our* problems exactly solved. I have 20 different buildouts 
installed, which may or may use the same pool of eggs, and may use 
different versions of the same package. I am the one who wants to have 
the final say in what versions of packages. I want to use. A linux 
distributor needs to have one working set of packages, instead.

>   Standard distributions come with a set of known working components
>   and quite weak dependancy declarations.
> 
>   When I install additional components, I will be told about
>   potential conflicts (based on the weak dependancies) and
>   asked what to do (install nevertheless, upgrade more things to
>   get dependancies right or abort).
> 
>   Usually, I do not have to worry about versions -- only
>   occationally (when I see conflict lists) or even more rarely,
>   when something breaks even though there has not been a conflict.

A linux distribution, unless you're on debian unstable, has quite a 
strict control over what packages enter a common pool. They do not 
release a newer version of a library unless they know the software that 
depends on it either works or can be upgraded to it.

We have a situation where we have developers, not maintainers, uploading 
new versions of packages. There will be no integrated testing done for 
all software built on all packages in the cheeseshop. Again, I can see 
similarities, but I don't believe linux distributions have *exactly* our 
problems solved. Our buildouts are used as development environments, not 
  only deployment environments.

> We currently made bad experiences with weak dependancies.
> 
> I see several reasons for this:
> 
>   *  not yet ready distributions (insufficiently tested,
>      alpha quality) have been uploaded to PyPI
> 
>      We are now aware that we must not do things like this

Better diligence here would help. It would help me most as a framework 
developer developing the framework - I can upgrade to newer versions of 
my dependencies without so much stuff breaking.

>   *  installation tools have prefered newer versions over
>      older ones, even when the newer versions were development/alpha
>      while the older ones have been stable
> 
>      The tools meanwhile have changed and stick preferably to
>      stable versions

Sticking to stable versions helps, until a new stable version is 
released. Then all the old stuff suddenly starts using the *new* stable 
version, and probably break.
> 
>   *  The installation tools work incrementally with dependancies
>      rather than based on a global dependancy graph.
> 
>      This is not yet changed.

> Maybe, our bad experience are drastically reduced when the above
> reasons are taken care of -- even with weak dependancies?

I used to believe that, but after seeing Grok 0.10 break once or twice 
*every* week for the last month, I don't believe it anymore. We're 
talking about the same release, breaking over and over again as 
something goes wrong with some egg somewhere. I want these dependencies 
pinned down hard. That said, I believe there are ways to solve these 
problems without hardcoding them in install_requires. I hope we can have 
the benefits of weak dependencies while having the safetly of hardcoded 
ones. See here:

http://faassen.n--tree.net/blog/view/weblog/2007/09/26/0

Regards,

Martijn