Re: [poposal for preparing a pep] sharing package installations between virtualenvs

Hi Freddy, Thomas

thanks for the well cautioned notes.

I'm vaguely aware that both, nix and conda, as "meta package managers"
have a effective way of linking correct sets of packages including the
tracing.
however both tools never made it into a consistent set of tools that my
peers, coworkers, co-maintainers and myself use.

as such i want to see the capabilities happen more at the python level.

wrt tracing whats from where, as a starting point i'd only packages
wheels from pypi (or a custom designated index server) as target for the
linking
I am well aware that this constraint precludes certain things that nix
and conda would do, but the initial isolation from binary dependencies
is critical on order to avoid layering that rabbit-hole
in at a greater expense. I'd love to see a followup in tooling that
would allow to map available/required feature sets from build time
dependencies to eventually elevate the issue,
but such process will take quite a while because of lack of available
contributors

One of my initial main goals for a minimal viable solution is to trim
down the massive duplication of available source wheels and manylinux wheels
more nuanced builds should stay on a classical layout/setup on a per
package basis while the details are fleshed out,

i certainly would like to see a path towards linking packages build on
tools like nix and conda more effectively
(in particular removing the need for deep layered and nested symlink
trees which i perceived in nix would be fabulous)

-- Ronny

Am 08.10.18 um 10:20 schrieb Freddy Rietdijk:
> What Conda and Nix do is basically having a store with packages, and
> then indeed linking them to form environments. While this works, it is
> important that one is very careful tracing how the packages are built,
> because not doing so can easily cause trouble in case of modules with
> extension types.
>
> On Mon, Oct 8, 2018 at 10:14 AM, Thomas Kluyver <[email protected]
> <mailto:[email protected]>> wrote:
>
>     Have you tried using conda envs? Conda avoids duplication by
>     unpacking all packages to a shared directory and then hard-linking
>     them into environments.
>
>     I'm not saying that should preclude doing something similar for
>     virtualenvs, but it's probably worth looking at existing solutions
>     before embarking on a change of this magnitude.
>
>     On Mon, Oct 8, 2018, at 8:42 AM, RonnyPfannschmidt wrote:
>     > hi everyone,
>     >
>     > i just did a little spring cleaning, and while i cleaned up my
>     > ~/Projects folder - the vast amount of that was virtualenvs with
>     roughly
>     > the same packages installed - each about 500 mb for like 2-5 mb
>     of code
>     >
>     > this seems excessively wasteful - my impression is that ~98% of the
>     > files seem to be duplicate.
>     >
>     > since the impending eol of python2 and the multi python version
>     > capabilities of python3 compiled code caches, i'd like to create a
>     > proposal for a shared folder structure
>     >
>     >
>     > $LIB/python/shared-packages/ should be a root folder in which
>     unpacked
>     > wheels would reside
>     >
>     > at a initial iteration this solution would suffer from the expanding
>     > python path issue since each package would have a toplevel
>     sys.path entry.
>     >
>     > on order to later on speed this up a simple api can be provided that
>     > makes python aware of the metatadata locations and the map
>     > toplevel/namespace names directly to import locations
>     >
>     >
>     > for example like
>     >
>     > shared-packages.pth
>     >
>     >     import sys;sys.path.insert(0,
>     > "$LIB/python/shared-packages/shared_packages_support-1.0-py3.whl/");
>     > import
>     >
>     shared_packages_support;sys.path.pop(0);shared_packages_support.enable()
>     >
>     > shared_packages.json
>     >
>     >     {
>     >
>     >       "pytest": "pytest-3.8.8-py2.py3.whl";
>     >
>     >     }
>     >
>     >
>     > in the aftermath, when shared packages support and fast
>     import/metadata
>     > for shared packages where implemented, the sizes of most virtualenvs
>     > would shrink significantly
>     >
>     >
>     > -- Ronny
>     > --
>     > Distutils-SIG mailing list -- [email protected]
>     <mailto:[email protected]>
>     > To unsubscribe send an email to [email protected]
>     <mailto:[email protected]>
>     >
>     https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>     <https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/>
>     > Message archived at
>     >
>     https://mail.python.org/mm3/archives/list/[email protected]/message/JWX7VFJRLWVIIBP4NZ5MEPRVCYTV4ZJT/
>     <https://mail.python.org/mm3/archives/list/[email protected]/message/JWX7VFJRLWVIIBP4NZ5MEPRVCYTV4ZJT/>
>     --
>     Distutils-SIG mailing list -- [email protected]
>     <mailto:[email protected]>
>     To unsubscribe send an email to [email protected]
>     <mailto:[email protected]>
>     https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>     <https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/>
>     Message archived at
>     https://mail.python.org/mm3/archives/list/[email protected]/message/XAZPBTKGEWVDLE3PJITY4JQ3YGK3NL7G/
>     <https://mail.python.org/mm3/archives/list/[email protected]/message/XAZPBTKGEWVDLE3PJITY4JQ3YGK3NL7G/>
>
>
>
> --
> Distutils-SIG mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/archives/list/[email protected]/message/3L24OT5FOHPBNTQHJYRNKR4HHDQLRD4X/

--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/mm3/archives/list/[email protected]/message/TYUKA44NU3VLS7X4RQOZXRUILU25RZ5Q/