Re: [poposal for preparing a pep] sharing package installations between virtualenvs

Have you tried using conda envs? Conda avoids duplication by unpacking all packages to a shared directory and then hard-linking them into environments.

I'm not saying that should preclude doing something similar for virtualenvs, but it's probably worth looking at existing solutions before embarking on a change of this magnitude.

On Mon, Oct 8, 2018, at 8:42 AM, RonnyPfannschmidt wrote:
> hi everyone,
> 
> i just did a little spring cleaning, and while i cleaned up my
> ~/Projects folder - the vast amount of that was virtualenvs with roughly
> the same packages installed - each about 500 mb for like 2-5 mb of code
> 
> this seems excessively wasteful - my impression is that ~98% of the
> files seem to be duplicate.
> 
> since the impending eol of python2 and the multi python version
> capabilities of python3 compiled code caches, i'd like to create a
> proposal for a shared folder structure
> 
> 
> $LIB/python/shared-packages/ should be a root folder in which unpacked
> wheels would reside
> 
> at a initial iteration this solution would suffer from the expanding
> python path issue since each package would have a toplevel sys.path entry.
> 
> on order to later on speed this up a simple api can be provided that
> makes python aware of the metatadata locations and the map
> toplevel/namespace names directly to import locations
> 
> 
> for example like
> 
> shared-packages.pth
> 
>     import sys;sys.path.insert(0,
> "$LIB/python/shared-packages/shared_packages_support-1.0-py3.whl/");
> import
> shared_packages_support;sys.path.pop(0);shared_packages_support.enable()
> 
> shared_packages.json
> 
>     {
> 
>       "pytest": "pytest-3.8.8-py2.py3.whl";
> 
>     }
> 
> 
> in the aftermath, when shared packages support and fast import/metadata
> for shared packages where implemented, the sizes of most virtualenvs
> would shrink significantly
> 
> 
> -- Ronny
> --
> Distutils-SIG mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at 
> https://mail.python.org/mm3/archives/list/[email protected]/message/JWX7VFJRLWVIIBP4NZ5MEPRVCYTV4ZJT/
--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/mm3/archives/list/[email protected]/message/XAZPBTKGEWVDLE3PJITY4JQ3YGK3NL7G/