Re: RFC: Known working sets

Tres Seaver wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Philipp von Weitershausen wrote:
>   
>> On 4 Sep 2007, at 01:21 , Tres Seaver wrote:
>>     
>>> Philipp von Weitershausen wrote:
>>>       
>>>> Wichert Akkerman wrote:
>>>>         
>>>>>> The only problem is that distributing grok-0.11.cfg is a bit  
>>>>>> tedious.
>>>>>> How about if buildout could get it from the web?
>>>>>>             
>>>>> How about making it available from an egg, through a hook in egg- 
>>>>> info
>>>>> perhaps?
>>>>>           
>>>> This is a very good point. So good in fact that I thought of it  
>>>> myself
>>>> :) I was already writing the email when your message came in.
>>>>
>>>>
>>>> Martijn and I discussed the known working set problem over IRC this
>>>> afternoon. He brought up a few good points which suggest that the
>>>> version data could be associated with the egg:
>>>>
>>>> The approach that I described in my original posting (and which  
>>>> actually
>>>> works *today*, as I found out!) has some disadvantages. For instance,
>>>> the discoverability and release mechanism of the known working set
>>>> configuration file differs a lot from our normal packages.  
>>>> Releasing a
>>>> package is a well-known routine by now. How and where would we  
>>>> release
>>>> the working set configuration file? SVN?
>>>>         
>>> Why not just release a meta-egg with hard-wired dependencies, plus the
>>> scripts required to set up and run the application?  If the other
>>> components are as relaxed as possible about their dependencies,  
>>> then it
>>> shouldn't matter that the meta-egg nails them down:  it isn't going to
>>> be possilbe to re-use the meta-egg anyway (and I for one wouldn't want
>>> to -- I'd be creating a new environment to run it in).
>>>       
>> I think the use case of being able to override a particular version  
>> of a package is valid. Let's say I'm on Zope 3.5.0 (meta egg or  
>> whatever) and I'd like to get the newest ZODB because it has a really  
>> cool feature. Sure, the warranty is void now, but I'd still like to  
>> be able to do it *without* having to reproduce all the version  
>> dependencies that the meta egg specifies.
>>
>>     
>>>> Another problem are dependencies and how we'd like to depend on other
>>>> working sets. Let's say we made a 'Zope' working set that  
>>>> contained the
>>>> stable versions of the zope.* packages. It would make sense for  
>>>> grok to
>>>> depend on this information. And packages using grok should depend on
>>>> that. It'll be complicated to maintain such a chain in separate text
>>>> files, especially across version updates. It only seems natural to  
>>>> use
>>>> the egg dependency mechanism for this.
>>>>         
>>> Depending on another WS would basically Just Work (TM) if we used egg
>>> dependencies.
>>>       
>> It would, but setuptools doesn't allow us to specify "overrides" in  
>> case of version conflicts. Perhaps zc.buildout could be taught how.  
>> Then I suppose I could settle for using '==' in a meta egg's setup.py.
>>
>>     
>>>> When EGG-INFO is written, a custom writer will then take this
>>>> information and generate 'known_working_versions.txt' or whatever in
>>>> EGG-INFO. The only problem is that this custom writer needs to be
>>>> installed when setup.py is called to create egg, in other words to
>>>> generate the right kind of eggs you'd need to have something  
>>>> installed
>>>> first. Not sure if this is better than maintaining .egg-info  
>>>> directories
>>>> in SVN...
>>>>         
>>> I guess I don't get the usecase for maintaining "known good"
>>> dependencies outside of setup.py (for the "meta" egg).
>>>       
>> I don't mind maintaining them in setup.py, but the "install_requires"  
>> mechanism is insufficient. Sooner or later you're going to end up  
>> with a version conflict.
>>     
>
> This is definitely a case where looking at how the Linux distros
> packaging works is instructive:  if you want to modify a package's
> dependencies (your overried, in this case), then you need to roll a new
> package.  At that point, perhaps we need a tool which introspects a
> "base" egg's dependencies and creates a new set, overriding only what is
> different:  in this case, the new egg would not depend on the old (in
> fact, it should *conflict with* or *replace* it).
>   
Linux distros take an approach that does not fit in the python world 
though: their meta-set is a release with its own package database. In 
other words every distribution/meta-set has its own PyPI instance database.

Wichert.

-- 
Wichert Akkerman <[email protected]>   It is simple to make things.
http://www.wiggy.net/                  It is hard to make things simple.