Re: [Pythonmac-SIG] py2app apps codesigned with "--options runtime", i.e. any notorizable app, fails to launch with ctypes error (was Re: ctypes MemoryError when attempting to launch notarized app)
Glyph <[email protected]> Sun, 28 Oct 2018 14:57:38 -0700
| Newsgroups | gmane.comp.python.pyobjc.devel,gmane.comp.python.apple |
|---|---|
| Message-ID | <[email protected]> |
--===============3998945443032513487== Content-Type: multipart/alternative; boundary=Apple-Mail-FDD2C097-28C5-443E-98E6-6696B4965234 Content-Transfer-Encoding: 7bit --Apple-Mail-FDD2C097-28C5-443E-98E6-6696B4965234 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable > On Oct 28, 2018, at 1:48 PM, Ronald Oussoren <[email protected]> wrot= e: >=20 >=20 >=20 >> On 28 Oct 2018, at 19:47, Glyph <[email protected]> wrote: >>=20 >>=20 >>=20 >>> On Oct 28, 2018, at 11:20 AM, Glyph <[email protected]> wrote: >>>=20 >>>=20 >>>=20 >>>> On Oct 28, 2018, at 2:27 AM, Ronald Oussoren <[email protected]> w= rote: >>>=20 >>>>>=20 >>>>> Curiously, this is the same traceback that comes from https://forum.ko= di.tv/showthread.php?tid=3D329171, which suggests it's something fundamental= to strict shared-library sandboxing that ctypes trips over when trying to i= nitialize itself. >>>>>=20 >>>>> Does anyone have experience with this, or ideas about what to do? >>>>=20 >>>> I=E2=80=99m afraid not. I currently get away with not signing apps at a= ll, although properly supporting signing is on my way too long wish list for= py2app. =20 >>>=20 >>> The ability to distribute unsigned apps is not-so-slowly going away; eve= n the ability to distribute non-notarized apps has a very limited shelf-life= at this point. So this ought to be an alarming development for everyone - h= aving Python apps effectively banned from macOS distribution is a big potent= ial problem :-\. >>>=20 >>> The good news here is that aside from having to write a little for loop i= n shell (shown below) getting the app codesigned previously was easy, and my= app *did* pass notarization, so nothing that py2app is doing is breaking th= ings on apple's end. It's just a matter of a ctypes bug. >>=20 >> On that note: more good news. While I haven't round-tripped through nota= rization again yet, this is a bit less dire than it first appeared. If I pr= event the import of ctypes with an `import sys; sys.modules['ctypes'] =3D No= ne`, and add a 'sed' script to my build process to prevent _setup_ctypes fro= m running in __boot__, then the app launches again. >>=20 >> Apparently my app doesn't actually need ctypes. >=20 > Good to hear that.=20 >=20 >>=20 >> The problem seems to be that Twisted includes a ctypes import; modulegrap= h sees this and thinks there is a hard dependency, and inserts the ctypes se= tup blob into __boot__. However, this is a conditional import, and it's for= Windows support anyway. >=20 > Hmm=E2=80=A6. I wonder what=E2=80=99s the best way forward here. I could a= dd on option to disable ctypes support, but that is a kludge. A weak import= ing hook (something like the never withdrawn PEP 369) could execute this cod= e only when actually needed, but I have no idea how hard it would be to impl= ement this. >=20 >=20 >>=20 >> (There also seem to be problems with cffi-using libraries, but not other s= hared objects, so maybe this is a bug in libffi; however, these don't interf= ere with py2app itself starting up.) >=20 > Interesting=E2=80=A6 I haven=E2=80=99t had complaints about PyObjC yet, a= nd that also uses libffi. =20 >=20 > I wonder what the =E2=80=9Chardened runtime=E2=80=9D option actually does a= nd enforces. In 3.7 the line in ctypes/__init__.py that causes the excepti= on is a call that creates a dummy C function, and likely triggers the first a= llocation for storing a libffi closure which could be something the hardened= runtime doesn=E2=80=99t like (being writeable + executable memory).=20 Interesting. Perhaps what I want is simply https://developer.apple.com/docum= entation/security/com_apple_security_cs_allow-unsigned-executable-memory the= n? Any chance you know how to jam that into a `codesign` command line someh= ow? :-) > P.S. I just noticed that the traceback in your initial message doesn=E2=80= =99t include the actual exception, just the traceback.=20 Oh; it=E2=80=99s =E2=80=9CMemoryError=E2=80=9D, no exception message. > Ronald >=20 --Apple-Mail-FDD2C097-28C5-443E-98E6-6696B4965234 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D= utf-8"></head><body dir=3D"auto"><div dir=3D"ltr"></div><div dir=3D"ltr"><br= ></div><div dir=3D"ltr"><br>On Oct 28, 2018, at 1:48 PM, Ronald Oussoren <= ;<a href=3D"mailto:[email protected]">[email protected]</a>> wr= ote:<br><br></div><blockquote type=3D"cite"><div dir=3D"ltr"><meta http-equi= v=3D"Content-Type" content=3D"text/html; charset=3Dutf-8"><br class=3D""><di= v><br class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On 28 O= ct 2018, at 19:47, Glyph <<a href=3D"mailto:[email protected]" clas= s=3D"">[email protected]</a>> wrote:</div><br class=3D"Apple-interc= hange-newline"><div class=3D""><meta http-equiv=3D"Content-Type" content=3D"= text/html; charset=3Dutf-8" class=3D""><div style=3D"word-wrap: break-word; -= webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><br clas= s=3D""><div class=3D""><br class=3D""><blockquote type=3D"cite" class=3D""><= div class=3D"">On Oct 28, 2018, at 11:20 AM, Glyph <<a href=3D"mailto:gly= [email protected]" class=3D"">[email protected]</a>> wrote:</div= ><br class=3D"Apple-interchange-newline"><div class=3D""><meta http-equiv=3D= "Content-Type" content=3D"text/html; charset=3Dutf-8" class=3D""><div style=3D= "word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-sp= ace;" class=3D""><br class=3D""><div class=3D""><br class=3D""><blockquote t= ype=3D"cite" class=3D""><div class=3D"">On Oct 28, 2018, at 2:27 AM, Ronald O= ussoren <<a href=3D"mailto:[email protected]" class=3D"">ronaldousso= [email protected]</a>> wrote:</div></blockquote></div></div></div></blockquote>= <blockquote type=3D"cite" class=3D""><div class=3D""><div style=3D"word-wrap= : break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" clas= s=3D""><div class=3D""><br class=3D""><blockquote type=3D"cite" class=3D""><= div class=3D""><blockquote type=3D"cite" style=3D"font-family: Menlo-Regular= ; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weigh= t: normal; letter-spacing: normal; orphans: auto; text-align: start; text-in= dent: 0px; text-transform: none; white-space: normal; widows: auto; word-spa= cing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; t= ext-decoration: none;" class=3D""><br class=3D"">Curiously, this is the same= traceback that comes from<span class=3D"Apple-converted-space"> </span= ><a href=3D"https://forum.kodi.tv/showthread.php?tid=3D329171" class=3D"">ht= tps://forum.kodi.tv/showthread.php?tid=3D329171</a>, which suggests it's som= ething fundamental to strict shared-library sandboxing that ctypes trips ove= r when trying to initialize itself.<br class=3D""><br class=3D"">Does anyone= have experience with this, or ideas about what to do?<br class=3D""></block= quote><br style=3D"caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; fo= nt-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: n= ormal; letter-spacing: normal; text-align: start; text-indent: 0px; text-tra= nsform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-wi= dth: 0px; text-decoration: none;" class=3D""><span style=3D"caret-color: rgb= (0, 0, 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; f= ont-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-= align: start; text-indent: 0px; text-transform: none; white-space: normal; w= ord-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; flo= at: none; display: inline !important;" class=3D"">I=E2=80=99m afraid not. I c= urrently get away with not signing apps at all, although properly supporting= signing is on my way too long wish list for py2app. </span><br style=3D= "caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 12px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spac= ing: normal; text-align: start; text-indent: 0px; text-transform: none; whit= e-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-dec= oration: none;" class=3D""></div></blockquote><div class=3D""><br class=3D""= ></div><div class=3D"">The ability to distribute unsigned apps is not-so-slo= wly going away; even the ability to distribute non-notarized apps has a very= limited shelf-life at this point. So this ought to be an alarming dev= elopment for everyone - having Python apps effectively banned from macOS dis= tribution is a big potential problem :-\.</div><div class=3D""><br class=3D"= "></div><div class=3D"">The good news here is that aside from having to writ= e a little for loop in shell (shown below) getting the app codesigned previo= usly was easy, and my app *did* pass notarization, so nothing that py2app is= doing is breaking things on apple's end. It's just a matter of a ctyp= es bug.</div></div></div></div></blockquote><br class=3D""></div><div class=3D= "">On that note: more good news. While I haven't round-tripped through= notarization again yet, this is a bit less dire than it first appeared. &nb= sp;If I prevent the import of ctypes with an `import sys; sys.modules['ctype= s'] =3D None`, and add a 'sed' script to my build process to prevent _setup_= ctypes from running in __boot__, then the app launches again.</div><div clas= s=3D""><br class=3D""></div><div class=3D"">Apparently my app doesn't actual= ly need ctypes.</div></div></div></blockquote><div><br class=3D""></div>Good= to hear that. </div><div><br class=3D""><blockquote type=3D"cite" clas= s=3D""><div class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mod= e: space; line-break: after-white-space;" class=3D""><div class=3D""><br cla= ss=3D""></div><div class=3D"">The problem seems to be that Twisted includes a= ctypes import; modulegraph sees this and thinks there is a hard dependency,= and inserts the ctypes setup blob into __boot__. However, this is a c= onditional import, and it's for Windows support anyway.</div></div></div></b= lockquote><div><br class=3D""></div>Hmm=E2=80=A6. I wonder what=E2=80=99s th= e best way forward here. I could add on option to disable ctypes support, bu= t that is a kludge. A weak importing hook (something like the never wi= thdrawn PEP 369) could execute this code only when actually needed, but I ha= ve no idea how hard it would be to implement this.</div><div><br class=3D"">= </div><div><br class=3D""><blockquote type=3D"cite" class=3D""><div class=3D= ""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; line-break= : after-white-space;" class=3D""><div class=3D""><br class=3D""></div><div c= lass=3D"">(There also seem to be problems with cffi-using libraries, but not= other shared objects, so maybe this is a bug in libffi; however, these don'= t interfere with py2app itself starting up.)</div></div></div></blockquote><= div><br class=3D""></div>Interesting=E2=80=A6 I haven=E2=80=99t had co= mplaints about PyObjC yet, and that also uses libffi. </div><div><br c= lass=3D""></div><div>I wonder what the =E2=80=9Chardened runtime=E2=80=9D op= tion actually does and enforces. In 3.7 the line in ctypes/__init__.p= y that causes the exception is a call that creates a dummy C function, and l= ikely triggers the first allocation for storing a libffi closure which could= be something the hardened runtime doesn=E2=80=99t like (being writeable + e= xecutable memory). </div></div></blockquote><div><br></div><div>Interes= ting. Perhaps what I want is simply <a href=3D"https://developer.apple.= com/documentation/security/com_apple_security_cs_allow-unsigned-executable-m= emory">https://developer.apple.com/documentation/security/com_apple_security= _cs_allow-unsigned-executable-memory</a> then? Any chance you kno= w how to jam that into a `codesign` command line somehow? :-)</div><br><bloc= kquote type=3D"cite"><div dir=3D"ltr"><div>P.S. I just noticed that the trac= eback in your initial message doesn=E2=80=99t include the actual exception, j= ust the traceback. </div></div></blockquote><div><br></div><div>Oh; it=E2= =80=99s =E2=80=9CMemoryError=E2=80=9D, no exception message.</div><br><block= quote type=3D"cite"><div dir=3D"ltr"><div>Ronald</div><div><br class=3D""></= div></div></blockquote></body></html>= --Apple-Mail-FDD2C097-28C5-443E-98E6-6696B4965234-- --===============3998945443032513487== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============3998945443032513487== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Pyobjc-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/pyobjc-dev --===============3998945443032513487==--