Re: How have you found compiling in OpenSSL and other libraries into your relocatable builds?

I think there are two different topics here:

- macOS, windows: PyPy is not doing a good job with the native 
installers. We should be following CPython and provide real packages, 
not a tarball. This is issue 3697 [0] for macOS and issue 2894 [5] for 
windows.


- linux: (aarch64, x86_64, i686): we create a portable build of PyPy. It 
starts in a docker file based on manylinux2014 [1] (note the various 
install scripts to get updated versions of some libraries). There is an 
additional wrinkle in that some of the python modules we ship are based 
on cffi c-extensions (like _ssl, _gdbm, lzma). For these we have an 
additional build step to download and build the relevant support library 
on every build [6]. Then when packaging, we use  ldd to check 
dependencies [2], set RPATH appropriately, and bundle those into the 
tarball [4]. We make sure that not only libpypy3-c.so is checked with 
ldd, but also the c-extensions [3]. There is a tradeoff between putting 
libraries in the Docker file and rebuilding the buildbots or building 
them each time, and openssl is actually on both build paths (probably 
this could be simplified). CI caching could probably be used to lessen 
the burden, we have a different build/test workflow from CPython where 
we typically build only once a day.


[0] https://foss.heptapod.net/pypy/pypy/-/issues/3697

[1] https://foss.heptapod.net/pypy/buildbot/-/tree/branch/default/docker

[2] 
https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/tool/release/make_portable.py#L39

[3] 
https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/tool/release/make_portable.py#L119

[4] 
https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/tool/release/package.py#L292

[5] https://foss.heptapod.net/pypy/pypy/-/issues/2894

[6] 
https://foss.heptapod.net/pypy/pypy/-/blob/branch/py3.8/lib_pypy/pypy_tools/build_cffi_imports.py#L65


As for complaints about OpenSSL versions: we release 2-3 times a year so 
can respond "quickly" if security concerns arise. My impression, based 
on the little feedback we get, is that shipping portable 
manylinux2014-based builds solves more end user problems than it causes. 
Sophisticated end users can always rebuild to suit their purposes.


Matti


On 1/4/22 01:47, matt--- via pypy-dev wrote:
> I think you have to bundle on macOS - Apple ships a really old openssl.
>
> I think cpython on macOS does this already? Might be good to be consistently bundling the same thing in Linus as well.
>
> M
>
> --
> Matt Billenstein
> [email protected]
>
>> On Mar 31, 2022, at 3:36 PM, Brett Cannon <[email protected]> wrote:
>>
>> I noticed that https://www.pypy.org/download.html says that "the linux binaries ship an OpenSSL library". Have you found that to be an issue in any way? Have users been accepting of it?
>>
>> I ask as I'm contemplating trying to get relocatable builds for CPython on Linux (to start), and I suspect the OpenSSL question is going to be one of the bigger/stickier discussion points and having some real-world experience/information on the topic would probably help.
>> _______________________________________________
>> pypy-dev mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> https://mail.python.org/mailman3/lists/pypy-dev.python.org/
>> Member address: [email protected]
_______________________________________________
pypy-dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/pypy-dev.python.org/
Member address: [email protected]