Re: Deferring native compilation finalization until idle

On Sat, Jul 25, 2026 at 1:37 PM Stéphane Marks <[email protected]> wrote:

> On Sat, Jul 25, 2026 at 1:11 PM Stéphane Marks <[email protected]>
> wrote:
>
>> On Sat, Jul 25, 2026 at 12:29 PM Stéphane Marks <[email protected]>
>> wrote:
>>
>>> On Sat, Jul 25, 2026 at 12:22 PM Aaron Jensen <[email protected]>
>>> wrote:
>>>
>>>> On Sat, Jul 25, 2026 at 9:18 AM, Stéphane Marks <[email protected]>
>>>> wrote:
>>>>
>>>>> On Sat, Jul 25, 2026 at 12:14 PM Aaron Jensen <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> On Sat, Jul 25, 2026 at 9:04 AM, Stéphane Marks <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> FWIW, my eln-cache this morning, starting fresh from Thursday, has
>>>>>>> 146 files with seldom used packages uncompiled, of course.  Rebuilding that
>>>>>>> cache was entirely asynchronous, perhaps except for the trampolines.  I
>>>>>>> have not tried setting native-comp-enable-subr-trampolines to nil in
>>>>>>> early-init.el to see if this bypasses the delays that you're seeing.  Have
>>>>>>> you tried that?
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> Setting native-comp-enable-subr-trampolines to nil does not
>>>>>> noticeably reduce the stutter for me.
>>>>>>
>>>>>> Ending up with only 146 files likely means that even if you were to
>>>>>> not have one of the workarounds in place, you would not notice as much
>>>>>> stutter as one who had more.
>>>>>>
>>>>>> My guess is that if you were to try my config, or someone else's that
>>>>>> notices the issue and not disable macOS's security checks, you too would
>>>>>> notice the stutter.
>>>>>>
>>>>>> I work with others that use macOS. I can survey them to see how many
>>>>>> of them have issues if that would be convincing to anyone. However, I think
>>>>>> the facts are pretty well established and ultimately it's a decision as to
>>>>>> whether or not the maintainers want to account for the particularities of
>>>>>> macOS's bundle validation process by loading the bundle within the async
>>>>>> worker (and take on the additional maintenance burden).
>>>>>>
>>>>>> Personally, I have my workaround, so I'm "fine". But I'm not a fan of
>>>>>> macOS users that have have sufficient packages having an experience that
>>>>>> leaves them with the opinion that Emacs has poor startup performance, or
>>>>>> native compilation makes things worse, etc. Yes, those are
>>>>>> narrowly-informed opinions, but I'm sure many users would not take the time
>>>>>> to compare their startup performance to other OS's and/or to find this
>>>>>> thread to better understand the depths of the problem. Instead, they'll
>>>>>> just have a bad taste and deal with it every time they install a new
>>>>>> version of Emacs.
>>>>>>
>>>>>
>>>>> From first principles, what package is waiting to be built in the
>>>>> eln-cache that Emacs is waiting for that prevents input?  AFAIK, the byte
>>>>> code is first loaded and available to run and only then replaced on-the-fly
>>>>> when native code is produced without error to replace it in the symbol's
>>>>> function cell.
>>>>>
>>>>
>>>> No, this is not what the problem is. That's the thing that I'm saying
>>>> has been pretty well established in this thread, though it seems not well
>>>> enough.
>>>>
>>>> The problem is that macOS, upon first *loading* the bundle in the main
>>>> Emacs process will undergo a security check (the one disabled by the
>>>> entitlements call you posted). This is a synchronous, blocking operation.
>>>> MacOS does not allow the process to load the bundle until it has verified
>>>> it, so the load is blocking. During this time, Emacs cannot receive input
>>>> because the main thread is completely blocked by the operating system. As
>>>> soon as you chain together hundreds of these macOS-initiated blocking
>>>> operations (every single bundle being loaded) you have a very poor,
>>>> stuttery, slow experience.
>>>>
>>>
>>> Okay so it is just the macOS binary scanner then and like any virus
>>> scanner interferes at the OS level.  A binary has to either opt out (as
>>> PROBLEMS document a method to do and the jimeh build does), or has to
>>> otherwise be whitelisted.  Every binary on macOS undergoes the same thing
>>> without being opted out so this isn't an Emacs issue.  I agree that Emacs
>>> loads "DLLs" and those can amplify the OS-imposed delay.  But only when
>>> they're loaded into memory.  Someone with a jillion packages surely doesn't
>>> need them all at once and can benefit from a carefully-crafted init.el with
>>> deferred package loading.
>>>
>>
>> FWIW, I did investigate whether we could enable DYLD environment variable
>> (a different codesign entitlement) to see if we could allow the Emacs
>> loader to temporarily disable library validation when loading a file from
>> the eln cache.  I also took a casual look at the source code here
>> https://github.com/apple-oss-distributions/dyld I failed to find a way
>> to create a finer-grained approach.  Perhaps an enterprising person will
>> dig something up.  For now, it seems it's all or nothing.
>>
>
> One thing I did not look at that someone may want to look into is whether
> it is possible to codesign the files in the eln cache using the same
> credentials/opt outs as the Emacs bundle (or ad-hoc signing) to avoid
> library validation.  If that works, perhaps a hook could be added to the
> native compiler to allow some post processing of each file to codesign it
> as a one-time cost.
>

One more avenue of investigation could be to leverage the JIT exception as
documented here
https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.security.cs.allow-jit?language=objc
and
Emacs on macOS would have to do some more work to mmap an eln file onto RAM
pages flagged with execution permissions.