Re: [Pythonmac-SIG] py2app apps codesigned with "--options runtime", i.e. any notorizable app, fails to launch with ctypes error (was Re: ctypes MemoryError when attempting to launch notarized app)
Ronald Oussoren via Pyobjc-dev <[email protected]> Sun, 28 Oct 2018 21:48:13 +0100
| Newsgroups | gmane.comp.python.pyobjc.devel,gmane.comp.python.apple |
|---|---|
| Message-ID | <[email protected]> |
--===============9015513363968423747== Content-type: multipart/alternative; boundary="Apple-Mail=_132F19C7-ACA1-4B4B-8065-CCF47C74FBA6" --Apple-Mail=_132F19C7-ACA1-4B4B-8065-CCF47C74FBA6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 28 Oct 2018, at 19:47, Glyph <[email protected]> wrote: >=20 >=20 >=20 >> On Oct 28, 2018, at 11:20 AM, Glyph <[email protected] = <mailto:[email protected]>> wrote: >>=20 >>=20 >>=20 >>> On Oct 28, 2018, at 2:27 AM, Ronald Oussoren <[email protected] = <mailto:[email protected]>> wrote: >>=20 >>>>=20 >>>> Curiously, this is the same traceback that comes from = https://forum.kodi.tv/showthread.php?tid=3D329171 = <https://forum.kodi.tv/showthread.php?tid=3D329171>, which suggests it's = something fundamental to strict shared-library sandboxing that ctypes = trips over when trying to initialize itself. >>>>=20 >>>> Does anyone have experience with this, or ideas about what to do? >>>=20 >>> I=E2=80=99m afraid not. I currently get away with not signing apps = at all, although properly supporting signing is on my way too long wish = list for py2app. =20 >>=20 >> The ability to distribute unsigned apps is not-so-slowly going away; = even the ability to distribute non-notarized apps has a very limited = shelf-life at this point. So this ought to be an alarming development = for everyone - having Python apps effectively banned from macOS = distribution is a big potential problem :-\. >>=20 >> The good news here is that aside from having to write a little for = loop in shell (shown below) getting the app codesigned previously was = easy, and my app *did* pass notarization, so nothing that py2app is = doing is breaking things on apple's end. It's just a matter of a ctypes = bug. >=20 > On that note: more good news. While I haven't round-tripped through = notarization again yet, this is a bit less dire than it first appeared. = If I prevent the import of ctypes with an `import sys; = sys.modules['ctypes'] =3D None`, and add a 'sed' script to my build = process to prevent _setup_ctypes from running in __boot__, then the app = launches again. >=20 > Apparently my app doesn't actually need ctypes. Good to hear that.=20 >=20 > The problem seems to be that Twisted includes a ctypes import; = modulegraph sees this and thinks there is a hard dependency, and inserts = the ctypes setup blob into __boot__. However, this is a conditional = import, and it's for Windows support anyway. Hmm=E2=80=A6. I wonder what=E2=80=99s the best way forward here. I could = add on option to disable ctypes support, but that is a kludge. A weak = importing hook (something like the never withdrawn PEP 369) could = execute this code only when actually needed, but I have no idea how hard = it would be to implement this. >=20 > (There also seem to be problems with cffi-using libraries, but not = other shared objects, so maybe this is a bug in libffi; however, these = don't interfere with py2app itself starting up.) Interesting=E2=80=A6 I haven=E2=80=99t had complaints about PyObjC yet, = and that also uses libffi. =20 I wonder what the =E2=80=9Chardened runtime=E2=80=9D option actually = does and enforces. In 3.7 the line in ctypes/__init__.py that causes = the exception is a call that creates a dummy C function, and likely = triggers the first allocation for storing a libffi closure which could = be something the hardened runtime doesn=E2=80=99t like (being writeable = + executable memory).=20 P.S. I just noticed that the traceback in your initial message doesn=E2=80= =99t include the actual exception, just the traceback.=20 Ronald --Apple-Mail=_132F19C7-ACA1-4B4B-8065-CCF47C74FBA6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><br = class=3D""><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On 28 Oct 2018, at 19:47, Glyph <<a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><meta = http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8" = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; line-break: after-white-space;" class=3D""><br class=3D""><div = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Oct 28, 2018, at 11:20 AM, Glyph <<a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><meta = http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8" = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; line-break: after-white-space;" class=3D""><br class=3D""><div = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Oct 28, 2018, at 2:27 AM, Ronald Oussoren <<a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> = wrote:</div></blockquote></div></div></div></blockquote><blockquote = type=3D"cite" class=3D""><div class=3D""><div style=3D"word-wrap: = break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" = class=3D""><div class=3D""><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><blockquote type=3D"cite" style=3D"font-family:= Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><br = class=3D"">Curiously, this is the same traceback that comes from<span = class=3D"Apple-converted-space"> </span><a = href=3D"https://forum.kodi.tv/showthread.php?tid=3D329171" = class=3D"">https://forum.kodi.tv/showthread.php?tid=3D329171</a>, which = suggests it's something fundamental to strict shared-library sandboxing = that ctypes trips over when trying to initialize itself.<br class=3D""><br= class=3D"">Does anyone have experience with this, or ideas about what = to do?<br class=3D""></blockquote><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""><span style=3D"caret-color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;" = class=3D"">I=E2=80=99m afraid not. I currently get away with not signing = apps at all, although properly supporting signing is on my way too long = wish list for py2app. </span><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""></div></blockquote><div class=3D""><br = class=3D""></div><div class=3D"">The ability to distribute unsigned apps = is not-so-slowly going away; even the ability to distribute = non-notarized apps has a very limited shelf-life at this point. So = this ought to be an alarming development for everyone - having Python = apps effectively banned from macOS distribution is a big potential = problem :-\.</div><div class=3D""><br class=3D""></div><div class=3D"">The= good news here is that aside from having to write a little for loop in = shell (shown below) getting the app codesigned previously was easy, and = my app *did* pass notarization, so nothing that py2app is doing is = breaking things on apple's end. It's just a matter of a ctypes = bug.</div></div></div></div></blockquote><br class=3D""></div><div = class=3D"">On that note: more good news. While I haven't = round-tripped through notarization again yet, this is a bit less dire = than it first appeared. If I prevent the import of ctypes with an = `import sys; sys.modules['ctypes'] =3D None`, and add a 'sed' script to = my build process to prevent _setup_ctypes from running in __boot__, then = the app launches again.</div><div class=3D""><br class=3D""></div><div = class=3D"">Apparently my app doesn't actually need = ctypes.</div></div></div></blockquote><div><br class=3D""></div>Good to = hear that. </div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><div = class=3D""><br class=3D""></div><div class=3D"">The problem seems to be = that Twisted includes a ctypes import; modulegraph sees this and thinks = there is a hard dependency, and inserts the ctypes setup blob into = __boot__. However, this is a conditional import, and it's for = Windows support anyway.</div></div></div></blockquote><div><br = class=3D""></div>Hmm=E2=80=A6. I wonder what=E2=80=99s the best way = forward here. I could add on option to disable ctypes support, but that = is a kludge. A weak importing hook (something like the never = withdrawn PEP 369) could execute this code only when actually needed, = but I have no idea how hard it would be to implement this.</div><div><br = class=3D""></div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><div = class=3D""><br class=3D""></div><div class=3D"">(There also seem to be = problems with cffi-using libraries, but not other shared objects, so = maybe this is a bug in libffi; however, these don't interfere with = py2app itself starting up.)</div></div></div></blockquote><div><br = class=3D""></div>Interesting=E2=80=A6 I haven=E2=80=99t had = complaints about PyObjC yet, and that also uses libffi. = </div><div><br class=3D""></div><div>I wonder what the =E2=80=9Chard= ened runtime=E2=80=9D option actually does and enforces. In 3.7 = the line in ctypes/__init__.py that causes the exception is a call that = creates a dummy C function, and likely triggers the first allocation for = storing a libffi closure which could be something the hardened runtime = doesn=E2=80=99t like (being writeable + executable = memory). </div><div><br class=3D""></div><div>P.S. I just noticed = that the traceback in your initial message doesn=E2=80=99t include the = actual exception, just the traceback. </div><div><br = class=3D""></div><div><br class=3D""></div><div>Ronald</div><div><br = class=3D""></div></body></html>= --Apple-Mail=_132F19C7-ACA1-4B4B-8065-CCF47C74FBA6-- --===============9015513363968423747== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============9015513363968423747== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Pyobjc-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/pyobjc-dev --===============9015513363968423747==--