Re: Three clarification questions about PEP 425 and PyPy3

On Thu, Aug 30, 2018 at 6:52 PM, Brett Cannon <[email protected]> wrote:
>
>
> On Thu, 30 Aug 2018 at 11:21 Nathaniel Smith <[email protected]> wrote:
>>
>> If we're going to rethink this,
>
>
> Well, I didn't want to "rethink" so much as "fill in". :)
>
>>
>> then I would really like to move away from assigning special meaning to
>> specific combinations of tags. The thing where if you use cp36m as you
>> Python ABI tag then you're forced to use cp36 as your python dialect tag
>> doesn't make sense. And the thing where if you have a wheel with an
>> extension module tagged as cp36-abi3, then that works fine on 3.7,
>
>
> I wouldn't expect that to if the stable ABI was expanded in Python 3.7 and
> you used the expanded part.

In my example, you have a wheel using the 3.6 ABI, running on 3.7.
That's case that's supposed to work :-).

>>
>> but if you *remove* the extension module then it *stops* working on 3.7?
>> That's just bizarre...
>
>
> I don't quite follow what you mean by "remove the extension module then it
> stops". What stops by removing the extension module(s)?

If I'm reading your proposal right, it says that when running on
Python 3.7, pip should be willing to install wheels tagged cp36-abi3,
but not wheels tagged cp36-none. But conceptually, if you take the
extension modules out of a cp36-abi3 wheel, then you're left with a
cp36-none wheel. So this is weird. Anywhere we're willing to install a
cp36-abi3 wheel, we should also be willing to install a cp36-none
wheel.

>>
>>
>> So my suggestions:
>>
>> * Make the 3 tag categories totally independent. Compute a separate set
>> for each, and then take the full cross product.
>
>
> I think Paul was hinting at this as part of his "wildcard" idea (and I
> honestly thought of this initially as well as it greatly simplifies things).
> So what would cp36-cp36m-plat expand to?

I don't know what it means to "expand" a wheel tag. Are you punning
the tag as also describing a Python installation ("CPython 3.6, with a
certain soabi tag and platform"), and then asking to find all the
wheel tags that could go in that installation? You can't actually do
this in general – for example, determining whether a target
interpreter is compatible with manylinux wheels requires running some
special sniffing code on that interpreter.

> cp36: cp3N where N is any positive digit(s)? And then toss in py3 and py3N?
> Prefer exact, then generic '3', then older, and finally newer?

Given that we don't have any real use cases for cpXY and pyXY, I'd
rather not expand the options – just preserve what we do now... so for
CPython 3.6, I'd say py3, py3N for N <= 6, cp3N for N <= 6, and I
don't really care about the exact order – some sort of
more-specific-before-less-specific makes sense, and whatever we do now
is probably fine. If someone goes wild and starts distributing py35
and py36 wheels for the same package (via a "36to35" tool, I guess?)
then that's probably what you want? But I don't imagine this will ever
be an important use case. We tried it with 2to3, and everyone decided
they'd rather write in the subset language for a decade instead.

> cp36m: that, abi3, and then 'none'? Do we care if someone has some crazy ABI
> that no one understands like 'b' (and if so should it only be applied to
> 'py' interpreter versions which break your nice cross product simplicity)?
> plat: depends on platform.

Do you mean, what happens if we find ourselves running on an
interpreter we don't recognize (not cpython/pypy/jython/...), and that
interpreter returns something wacky from
sysconfig.get_config_var("SOABI")? I think there's a reasonable
argument that in that case we should only accept 'none' as the tag.
(And then maybe whoever invented this new interpreter gets the job of
adding sysconfig.get_supported_abi_tags() as a standard stdlib feature
:-).)

> And the match preference goes to platform, interpreter version, and then
> ABI? I think that would work in terms of ignoring C ABIs that have very
> little chance of linking while being the broadest in terms of accepting a
> wheel that has some semblance of a chance of running.

I don't think the preference order matters that much as long as its
well-defined. The only cases where it would affect things are like...

Suppose in the future we add support for platform tags based on
different ISA levels, like x86_64_avx2 vs x86_64_sse3 vs x86_64. Then
you could find yourself in a situation where examplepkg v1.2.3 has the
following wheels available:

cp36-cp36m-manylinux1_x86_64
cp36-abi3-manylinux1_x86_64_avx2

Our environment is CPython 3.6, running on a manylinux1-compatible OS
and we do have AVX2 support available, so either of these wheels could
work. The first wheel has a "better" ABI (cp36m > abi3), but a "worse"
platform (x86_64 < x86_64_avx2). So which one should we pick?

I have zero intuition here. Whoever compiled these wheels is clearly
perverse, and they should stop doing silly things like this.

> And how would that apply to PyPy3? Same ranging on 'pp36N` and drop the
> 'abi3' insertion?
>
>>
>>
>> * Since the stable ABI actually changes over time, we should define new
>> tags abi35, abi36, etc. that mean "requires the stable ABI as defined by
>> this version of cpython or higher", instead of relying on abi3 + a dialect
>> tag. (Imagine if pypy started implementing the stable ABI –we'd have to
>> start allowing cpXY tags to match PyPy.)
>>
>> * Plan to move away from the pyXY and cpXY tags over time; they're
>> confusing and not useful. Of course this will have to be a gradual process,
>> but if pip stops requiring them now then in a year or two we could make
>> setuptools stop generating then.
>
>
> What would you do then about preferred match order for pure Python wheels?
> E.g. how do you preferably match against Python 3.7 wheels over 3.6 when
> running a Python 3.7 interpreter? Or are you suggesting equivalent ABI tags
> to make up for the pyXY tags and the interpreter tag simply gets ignored?

I don't understand your questions, sorry :-(.

I'm just saying: we should make the stable-ABI tags self-contained, so
that 'py3-abi36' would be sufficient to express "uses the stable abi,
as it existed on python 3.6". This would match the same set of
interpreters that that 'cp36-abi3' is currently used to target. And
also, we should try to gradually decrease the usage of pyXY and cpXY
tags (e.g. eventually setuptools should stop generating them by
default).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/mm3/archives/list/[email protected]/message/WFPUFDKMOFESIKSYFDYN7YYAQBCDJG67/