Re: Three clarification questions about PEP 425 and PyPy3

On Mon, 27 Aug 2018 at 19:05 Nathaniel Smith <[email protected]> wrote:

> I think the answer to all of these questions is "well, no-one's ever
> really looked that closely".
>

I figured, but I just needed someone to verify that hunch was correct. :)


>
> There's a theory behind the tags; they're supposed to be a reasonably
> expressive language for talking about Python dialect compatibility,
> Python C ABI compatibility, and platform ABI compatibility,
> respectively. But in practice so far only a small fixed set of tag
> combinations actually gets used, so there's plenty of room for weird
> stuff to accumulate in the corners where no-one looks.
>

Yep. There's also the side-effect that the pep425tags code that the various
tools have embedded makes some very broad assumptions that all interpreters
follow a tagging/versioning style similar to what is used for CPython,
hence the odd PyPy3 results where the "3" part of the interpreter _name_
gets used as if it's a major version.


>
> I've never been able to figure out a use case for the interpreter tags
> in the first field ("cp36", "pp3", etc).


To help answer that, here are some statistics on the number of projects
that have a certain wheel tag based on download counts on 2018-08-24.

   - cp36
   - %cp36-none-any.whl: 7 (example
      <https://pypi.org/project/google-python-cloud-debugger/2.8/#files>)
      - %cp36-none-%.whl: 70 (example
      <https://pypi.org/project/qucumber/#files>)
      - cp36-none-%.whl but not cp36-none-any.whl: 65 (example
      <https://pypi.org/project/numpy/1.15.1/#files>that Nathaniel knows
      very well ;)
   - cp3
   - %cp3-none-any.whl: 2 (example <https://pypi.org/project/kinto/#files>)
      - %cp3-none-%.whl: 3 (example
      <https://pypi.org/project/billiard/3.3.0.23/#files>)
      - cp3-%: 3 (same as cp3-none-%)
   - py36
   - py36-none-% but not py36-none-any: 2 (example
      <https://pypi.org/project/pytrack-analysis/0.0.3/#files>)
   - py3
      - py3-none-% but not py3-none-any: 142 (example
      <https://pypi.org/project/mypy/0.511/#files>)
      - pp3
      - %pp3-%: 1 (example <https://pypi.org/project/cliquet/3.1.1/#files>)
   - pp360
      - %pp360-%: 6 (example <https://pypi.org/project/Pillow/5.2.0/#files>)

To put this into perspective, there are currently 150,410 projects on PyPI.
Plus a lot of those odd examples for broadly stated interpreter versions
consistently come from the kinto and cliquet projects (although not for
their latest releases).

In the end I think you can view the interpreter tag as representing a
namespace for the ABI tag. Otherwise it technically isn't necessary as you
could just use either an interpreter version or Python version as a form of
ABI tag and drop the interpreter tag.


> IIUC, the theory is that
> they're supposed to mean "the Python code in this package is not
> actually portable Python, but uses an interpreter-specific dialect of
> the language". (This is very different from the second field, where
> tags like "cp36m" tell you about the required C ABI -- that one is
> obviously super useful.) I guess if you had a package that like,
> absolutely depended on GC being refcount-based, you could use a cp3
> tag to indicate that, or if you had a pure-Python, python 2 package,
> that required 'dict' to be ordered, maybe that's 'pp2-none-any'? But
> this never seems to actually happen in practice. It seems like an idea
> that sounded plausible early on, and then never got fleshed out or
> revisited.
>

Yeah, and it actually isn't expressive _enough_ to be self-contained to
cover all use-cases for these tags because the "Python version" tag --
which I consider the interpreter tag -- doesn't necessarily cover Python
version compatibility for the interpreter that's been specified. This
becomes a need when you want to figure out what wheel is the best fit for
Python for a certain interpreter (i.e. if I was a cloud provider and said
what Python was supported by tag triple it actually wouldn't be enough to
download appropriate wheels without also knowing the Python version as a
side-channel bit of information). Not a huge deal, but something I noticed.


>
> The distutils folks have never sat down to seriously think about
> non-CPython implementations, where the language version and the
> implementation version are separate things.
>

Guess what I've started doing? ;)


>
> The pypy folks have never sat down to seriously think about API/ABI
> stability. Generally at the Python dialect level they try to match a
> given version of (C)Python, and at the ABI level every new release is
> a new ABI.
>
> My guess is you shouldn't spend too much effort on trying to slavishly
> reproduce pip's logic, and that if you wanted to go clean up pip's
> logic (and maybe extract it into a reusable library?) then the devs
> would be perfectly happy that someone was doing it...
>

That's exactly what I'm in the process of doing. :) My goal is to have a
library that tools will drop their internal copies of pep425tags for so
there's a standardized PEP 425 implementation. I just wanted to make sure
that before I write any more code that I knew what needed to be handled for
backwards-compatibility versus what is a historical accident or was a guess
at what the future might need when the PEP was written.

Anyway, I will give this a think and try to come up with a reasonable
algorithm for generating the sequence of supported tags based on a specific
tag and Python version and then code that up into a library (at least I
will definitely have something to work on at the dev sprints :) .

-Brett


>
> -n
>
> On Mon, Aug 27, 2018 at 6:28 PM, Brett Cannon <[email protected]> wrote:
> > And to help in getting a reply, here is the trimmed-down results for
> CPython
> > 3.7 to compare against:
> >
> >     [('cp37', 'cp37m', 'macosx_10_13_x86_64'),
> >     …
> >      ('cp37', 'abi3', 'macosx_10_13_x86_64'),
> >     …
> >       ('cp37', 'none', 'macosx_10_13_x86_64'),
> >     …
> >      ('cp36', 'abi3', 'macosx_10_13_x86_64'),
> >     …
> >       ('cp35', 'abi3', 'macosx_10_13_x86_64'),
> >     …
> >       ('cp34', 'abi3', 'macosx_10_13_x86_64'),
> >     …
> >      ('cp33', 'abi3', 'macosx_10_13_x86_64'),
> >     …
> >      ('cp32', 'abi3', 'macosx_10_13_x86_64'),
> >     …
> >      ('py3', 'none', 'macosx_10_13_x86_64'),
> >     …
> >      ('cp37', 'none', 'any'),
> >      ('cp3', 'none', 'any'),
> >      ('py37', 'none', 'any'),
> >      ('py3', 'none', 'any'),
> >      ('py36', 'none', 'any'),
> >      ('py35', 'none', 'any'),
> >      ('py34', 'none', 'any'),
> >      ('py33', 'none', 'any'),
> >      ('py32', 'none', 'any'),
> >      ('py31', 'none', 'any'),
> >  ('py30', 'none', 'any')]
> >
> > So, it re-iterate the questions:
> >
> > What is ('pp3', 'none', 'any') supposed to represent for PyPy3? Since the
> > version of the interpreter is PyPy3 6.0 the lack of major version number
> > seems like a bug more than a purposeful interpreter version (and there's
> > only a single project -- cliquet -- that has a wheel that's compatible
> with
> > that tag triple and it's not even for their latest release).
> > Why does CPython have (*, 'none', 'any') from the version of the
> interpreter
> > down to Python 3.0 plus generically Python 3 while PyPy3 only gets
> generic
> > Python 3?
> > Why isn't (*, 'none', platform) listed from Python 3.7 to 3.0 for either
> > CPython or PyPy3? I understand not iterating through all versions when an
> > ABI is involved (without knowing exactly which versions are compatible
> like
> > abi3), but this triple seems safe to iterate through as a fallback just
> as
> > much as (*, 'none', 'any'). Maybe because it's too ambiguous to know how
> > important such a fallback would be between e.g. ('py36', 'none',
> > 'macosx_10_13_x86_64') and ('py37', 'none', 'any'), and so why bother
> when
> > the older version triples are there just for a safety net to have at
> least
> > some chance of a match?
> > I still think ('py360', 'none', 'any') is a bug. ;)
> >
> >
> > P.S.: The ('py3', 'none', 'macosx_10_13_x86_64') triple being between
> e.g.
> > ('pp360', 'none', 'macosx_10_13_x86_64') and ('pp360', 'none', 'any') is
> > really messing with my head and making the code to generate supported
> > triples a bit less elegant. ;)
> >
> > On Sat, 25 Aug 2018 at 15:03 Brett Cannon <[email protected]> wrote:
> >>
> >> I noticed that for PyPy3, the tag triples considered compatible were
> >> (roughly; trimmed out the long list of macOS versions):
> >>
> >>     [('pp360', 'pypy3_60', 'macosx_10_13_x86_64'),
> >>      ('pp360', 'none', 'macosx_10_13_x86_64'),
> >>       ('py3', 'none', 'macosx_10_13_x86_64'),
> >>      ('pp360', 'none', 'any'),
> >>      ('pp3', 'none', 'any'),
> >>      ('py360', 'none', 'any'),
> >>      ('py3', 'none', 'any')]
> >>
> >> Now the first question I have is about ('pp3', 'none', 'any'). Is this
> >> meant to be a generic thing for any interpreter of the interpreter
> >> implementation and major version, or is this special to CPython and
> PyPy3?
> >>
> >> Question two is why isn't there a ('py35', 'none', 'any') or ('py34',
> >> 'none', 'any') and older to py30 after py3 like there is for CPython?
> Seems
> >> like if they are just source then they should be compatible as much as
> >> CPython.
> >>
> >> Question three is why isn't there a ('py35', 'none',
> >> 'macosx_10_13_x86_64') for PyPy3 or CPython 3.7? I can't figure out
> what a
> >> Python- and platform-specific wheel but agnostic to API wouldn't ever
> work?
> >>
> >> And I'm assuming ('py360', 'none', 'any'), isn't legitimate since that
> >> makes no sense. ;)
> >
> >
> > --
> > Distutils-SIG mailing list -- [email protected]
> > To unsubscribe send an email to [email protected]
> > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> > Message archived at
> >
> https://mail.python.org/mm3/archives/list/[email protected]/message/D7H47XWWDMOIUWODELQU27TGTC5KL6WY/
> >
>
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>

--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/mm3/archives/list/[email protected]/message/3K3NND564LI6HWF736QD4CTALE5UB6KU/