py2app apps codesigned with "--options runtime", i.e. any notorizable app, fails to launch with ctypes error (was Re: ctypes MemoryError when attempting to launch notarized app)
Glyph <[email protected]> Sun, 28 Oct 2018 11:20:47 -0700
| Newsgroups | gmane.comp.python.pyobjc.devel,gmane.comp.python.apple |
|---|---|
| Message-ID | <[email protected]> |
--===============1477917853548813147== Content-Type: multipart/alternative; boundary="Apple-Mail=_5F6E07AE-BA51-4835-A855-4427F3964D45" --Apple-Mail=_5F6E07AE-BA51-4835-A855-4427F3964D45 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Oct 28, 2018, at 2:27 AM, Ronald Oussoren <[email protected]> = wrote: >=20 >=20 >=20 >> On 28 Oct 2018, at 06:27, Glyph <[email protected]> wrote: >>=20 >> I adjusted my code-signing to use the new, stricter requirements = imposed by app notarization. I managed to get it successfully notarized, = but the app is now non-functional as a result: at startup, I get: >>=20 >> Traceback (most recent call last): >> File "my.app/Contents/Resources/__boot__.py", line 93, in <module> >> _setup_ctypes() >> File "my.app/Contents/Resources/__boot__.py", line 86, in = _setup_ctypes >> from ctypes.macholib import dyld >> File "<frozen importlib._bootstrap>", line 971, in _find_and_load >> File "<frozen importlib._bootstrap>", line 955, in = _find_and_load_unlocked >> File "<frozen importlib._bootstrap>", line 656, in _load_unlocked >> File "<frozen importlib._bootstrap>", line 626, in = _load_backward_compatible >> File "ctypes/__init__.pyc", line 538, in <module> >> File "ctypes/__init__.pyc", line 273, in _reset_cache >>=20 >> (If anyone wants to follow along in the traceback, this is using = python.org 3.6.6.) >=20 > On what version of macOS? I expect 10.14 because that=E2=80=99s the = only release that actually knows about notarization, but enabling this = feature might also affect how the app is signed. ProductName: Mac OS X ProductVersion: 10.14 BuildVersion: 18A391 >> This happens before any of my code even runs, so I can't just try to = avoid ctypes. >=20 > You could patch the __boot__.py file before signing to see if that = helps. Although this should cause problems further on, the call to = _setup_ctypes should only be created when some code in your app bundle = has a dependency on ctypes.=20 I'll give that a shot. >>=20 >> Curiously, this is the same traceback that comes from = https://forum.kodi.tv/showthread.php?tid=3D329171 = <https://forum.kodi.tv/showthread.php?tid=3D329171>, which suggests it's = something fundamental to strict shared-library sandboxing that ctypes = trips over when trying to initialize itself. >>=20 >> Does anyone have experience with this, or ideas about what to do? >=20 > I=E2=80=99m afraid not. I currently get away with not signing apps at = all, although properly supporting signing is on my way too long wish = list for py2app. =20 The ability to distribute unsigned apps is not-so-slowly going away; = even the ability to distribute non-notarized apps has a very limited = shelf-life at this point. So this ought to be an alarming development = for everyone - having Python apps effectively banned from macOS = distribution is a big potential problem :-\. The good news here is that aside from having to write a little for loop = in shell (shown below) getting the app codesigned previously was easy, = and my app *did* pass notarization, so nothing that py2app is doing is = breaking things on apple's end. It's just a matter of a ctypes bug. As I see it, there's 2 problems here: py2app's __boot__.py should fail more gracefully if initializing ctypes = doesn't work, since not everybody needs ctypes. Shall I file this on = the tracker? ctypes itself should address whatever eldritch hideousness is causing = this; in addition to the windows security layer stuff I found, = grsecurity TPE causes the same traceback: = https://bugs.python.org/issue28429 > With some luck there=E2=80=99s some entitlement or code signing option = that causes this problem. What is the output of "codesign --display = --verbose=3D4=E2=80=9D for the application? Both with and without = notarisation?=20 Sorry, my original message was not clear. App notarization itself is = not the problem, it's the "stricter requirements" that I ambiguously = referenced. The requirement in question is the '--options runtime' flag = passed to 'codesign'. So you can just codesign an app (even with an = ad-hoc identity, you technically could do this without even having a = valid cert, although the way one generates one of those escapes me) with = the 'runtime' option, you can reproduce this. So if I sign my app like this: #!/bin/bash find "${NAME}.app" -iname '*.so' -or -iname '*.dylib' | while read libfile; do codesign --sign "${IDENTITY}" \ --deep "${libfile}" \ --force \ --options runtime; done; codesign --sign "${IDENTITY}" \ --deep "${NAME}.app" \ --force \ --options runtime; and then run it as "./${NAME}.app/Contents/MacOS/${NAME}". I = immediately get the traceback given above. -glyph= --Apple-Mail=_5F6E07AE-BA51-4835-A855-4427F3964D45 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><br = class=3D""><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Oct 28, 2018, at 2:27 AM, Ronald Oussoren <<a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;" class=3D""><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;" class=3D""><blockquote type=3D"cite" style=3D"font-family: = Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D"">On 28 = Oct 2018, at 06:27, Glyph <<a href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:<br class=3D""><br = class=3D"">I adjusted my code-signing to use the new, stricter = requirements imposed by app notarization. I managed to get it = successfully notarized, but the app is now non-functional as a result: = at startup, I get:<br class=3D""><br class=3D"">Traceback (most recent = call last):<br class=3D""> File = "my.app/Contents/Resources/__boot__.py", line 93, in <module><br = class=3D""> _setup_ctypes()<br class=3D""> File = "my.app/Contents/Resources/__boot__.py", line 86, in _setup_ctypes<br = class=3D""> from ctypes.macholib import dyld<br = class=3D""> File "<frozen importlib._bootstrap>", line 971, = in _find_and_load<br class=3D""> File "<frozen = importlib._bootstrap>", line 955, in _find_and_load_unlocked<br = class=3D""> File "<frozen importlib._bootstrap>", line 656, = in _load_unlocked<br class=3D""> File "<frozen = importlib._bootstrap>", line 626, in _load_backward_compatible<br = class=3D""> File "ctypes/__init__.pyc", line 538, in = <module><br class=3D""> File "ctypes/__init__.pyc", line 273, = in _reset_cache<br class=3D""><br class=3D"">(If anyone wants to follow = along in the traceback, this is using <a href=3D"http://python.org" = class=3D"">python.org</a> 3.6.6.)<br class=3D""></blockquote><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;" class=3D""><span style=3D"caret-color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;" class=3D"">On what = version of macOS? I expect 10.14 because that=E2=80=99s the only release = that actually knows about notarization, but enabling this feature might = also affect how the app is signed.</span></div></blockquote><div><br = class=3D""></div><div><div>ProductName:<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>Mac OS = X</div><div>ProductVersion:<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> = </span>10.14</div><div>BuildVersion:<span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>18A391</div></div><br = class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D""><blockquote type=3D"cite" style=3D"font-family: = Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D"">This = happens before any of my code even runs, so I can't just try to avoid = ctypes.<br class=3D""></blockquote><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""><span style=3D"caret-color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;" = class=3D"">You could patch the __boot__.py file before signing to see if = that helps. Although this should cause problems further on, the call to = _setup_ctypes should only be created when some code in your app bundle = has a dependency on ctypes.<span = class=3D"Apple-converted-space"> </span></span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;" class=3D""></div></blockquote><div><br class=3D""></div><div>I'll = give that a shot.</div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><blockquote type=3D"cite" style=3D"font-family:= Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><br = class=3D"">Curiously, this is the same traceback that comes from<span = class=3D"Apple-converted-space"> </span><a = href=3D"https://forum.kodi.tv/showthread.php?tid=3D329171" = class=3D"">https://forum.kodi.tv/showthread.php?tid=3D329171</a>, which = suggests it's something fundamental to strict shared-library sandboxing = that ctypes trips over when trying to initialize itself.<br class=3D""><br= class=3D"">Does anyone have experience with this, or ideas about what = to do?<br class=3D""></blockquote><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""><span style=3D"caret-color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;" = class=3D"">I=E2=80=99m afraid not. I currently get away with not signing = apps at all, although properly supporting signing is on my way too long = wish list for py2app. </span><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""></div></blockquote><div><br = class=3D""></div><div>The ability to distribute unsigned apps is = not-so-slowly going away; even the ability to distribute non-notarized = apps has a very limited shelf-life at this point. So this ought to = be an alarming development for everyone - having Python apps effectively = banned from macOS distribution is a big potential problem = :-\.</div><div><br class=3D""></div><div>The good news here is that = aside from having to write a little for loop in shell (shown below) = getting the app codesigned previously was easy, and my app *did* pass = notarization, so nothing that py2app is doing is breaking things on = apple's end. It's just a matter of a ctypes bug.</div><div><br = class=3D""></div><div>As I see it, there's 2 problems = here:</div><div><br class=3D""></div><div><ol class=3D"MailOutline"><li = class=3D"">py2app's __boot__.py should fail more gracefully if = initializing ctypes doesn't work, since not everybody needs ctypes. = Shall I file this on the tracker?</li><li class=3D"">ctypes itself = should address whatever eldritch hideousness is causing this; in = addition to the windows security layer stuff I found, grsecurity TPE = causes the same traceback: <a href=3D"https://bugs.python.org/issue28429" = class=3D"">https://bugs.python.org/issue28429</a></li></ol></div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;" class=3D"">With some = luck there=E2=80=99s some entitlement or code signing option that causes = this problem. What is the output of "codesign --display = --verbose=3D4=E2=80=9D for the application? Both with and without = notarisation?<span class=3D"Apple-converted-space"> </span></span><br= style=3D"caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;" class=3D""></div></blockquote></div><br class=3D""><div = class=3D"">Sorry, my original message was not clear. App = notarization itself is not the problem, it's the "stricter requirements" = that I ambiguously referenced. The requirement in question is the = '--options runtime' flag passed to 'codesign'. So you can just = codesign an app (even with an ad-hoc identity, you technically could do = this without even having a valid cert, although the way one generates = one of those escapes me) with the 'runtime' option, you can reproduce = this.</div><div class=3D""><br class=3D""></div><div class=3D"">So if I = sign my app like this:</div><div class=3D""><br = class=3D""></div><blockquote style=3D"margin: 0 0 0 40px; border: none; = padding: 0px;" class=3D""><div class=3D"">#!/bin/bash</div><div = class=3D"">find "${NAME}.app" -iname '*.so' -or -iname '*.dylib' = |</div><div class=3D""> while read libfile; do</div><div = class=3D""> codesign --sign = "${IDENTITY}" \</div><div class=3D""> = --deep "${libfile}" \</div><div = class=3D""> = --force \</div><div class=3D""> = --options runtime;</div><div = class=3D""> done;</div><div class=3D""><br = class=3D""></div><div class=3D"">codesign --sign "${IDENTITY}" = \</div><div class=3D""> --deep = "${NAME}.app" \</div><div class=3D""> = --force \</div><div class=3D""> = --options runtime;</div></blockquote><div class=3D""><br = class=3D""></div><div class=3D"">and then run it as = "./${NAME}.app/Contents/MacOS/${NAME}". I immediately get the = traceback given above.</div><div class=3D""><br class=3D""></div><div = class=3D"">-glyph</div></body></html>= --Apple-Mail=_5F6E07AE-BA51-4835-A855-4427F3964D45-- --===============1477917853548813147== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============1477917853548813147== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Pyobjc-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/pyobjc-dev --===============1477917853548813147==--