Re: Deferring native compilation finalization until idle

On Sat, Jul 25, 2026 at 1:11 PM Stéphane Marks <[email protected]> wrote:

> On Sat, Jul 25, 2026 at 12:29 PM Stéphane Marks <[email protected]>
> wrote:
>
>> On Sat, Jul 25, 2026 at 12:22 PM Aaron Jensen <[email protected]>
>> wrote:
>>
>>> On Sat, Jul 25, 2026 at 9:18 AM, Stéphane Marks <[email protected]>
>>> wrote:
>>>
>>>> On Sat, Jul 25, 2026 at 12:14 PM Aaron Jensen <[email protected]>
>>>> wrote:
>>>>
>>>>> On Sat, Jul 25, 2026 at 9:04 AM, Stéphane Marks <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> FWIW, my eln-cache this morning, starting fresh from Thursday, has
>>>>>> 146 files with seldom used packages uncompiled, of course.  Rebuilding that
>>>>>> cache was entirely asynchronous, perhaps except for the trampolines.  I
>>>>>> have not tried setting native-comp-enable-subr-trampolines to nil in
>>>>>> early-init.el to see if this bypasses the delays that you're seeing.  Have
>>>>>> you tried that?
>>>>>>
>>>>>>>
>>>>>>>
>>>>> Setting native-comp-enable-subr-trampolines to nil does not noticeably
>>>>> reduce the stutter for me.
>>>>>
>>>>> Ending up with only 146 files likely means that even if you were to
>>>>> not have one of the workarounds in place, you would not notice as much
>>>>> stutter as one who had more.
>>>>>
>>>>> My guess is that if you were to try my config, or someone else's that
>>>>> notices the issue and not disable macOS's security checks, you too would
>>>>> notice the stutter.
>>>>>
>>>>> I work with others that use macOS. I can survey them to see how many
>>>>> of them have issues if that would be convincing to anyone. However, I think
>>>>> the facts are pretty well established and ultimately it's a decision as to
>>>>> whether or not the maintainers want to account for the particularities of
>>>>> macOS's bundle validation process by loading the bundle within the async
>>>>> worker (and take on the additional maintenance burden).
>>>>>
>>>>> Personally, I have my workaround, so I'm "fine". But I'm not a fan of
>>>>> macOS users that have have sufficient packages having an experience that
>>>>> leaves them with the opinion that Emacs has poor startup performance, or
>>>>> native compilation makes things worse, etc. Yes, those are
>>>>> narrowly-informed opinions, but I'm sure many users would not take the time
>>>>> to compare their startup performance to other OS's and/or to find this
>>>>> thread to better understand the depths of the problem. Instead, they'll
>>>>> just have a bad taste and deal with it every time they install a new
>>>>> version of Emacs.
>>>>>
>>>>
>>>> From first principles, what package is waiting to be built in the
>>>> eln-cache that Emacs is waiting for that prevents input?  AFAIK, the byte
>>>> code is first loaded and available to run and only then replaced on-the-fly
>>>> when native code is produced without error to replace it in the symbol's
>>>> function cell.
>>>>
>>>
>>> No, this is not what the problem is. That's the thing that I'm saying
>>> has been pretty well established in this thread, though it seems not well
>>> enough.
>>>
>>> The problem is that macOS, upon first *loading* the bundle in the main
>>> Emacs process will undergo a security check (the one disabled by the
>>> entitlements call you posted). This is a synchronous, blocking operation.
>>> MacOS does not allow the process to load the bundle until it has verified
>>> it, so the load is blocking. During this time, Emacs cannot receive input
>>> because the main thread is completely blocked by the operating system. As
>>> soon as you chain together hundreds of these macOS-initiated blocking
>>> operations (every single bundle being loaded) you have a very poor,
>>> stuttery, slow experience.
>>>
>>
>> Okay so it is just the macOS binary scanner then and like any virus
>> scanner interferes at the OS level.  A binary has to either opt out (as
>> PROBLEMS document a method to do and the jimeh build does), or has to
>> otherwise be whitelisted.  Every binary on macOS undergoes the same thing
>> without being opted out so this isn't an Emacs issue.  I agree that Emacs
>> loads "DLLs" and those can amplify the OS-imposed delay.  But only when
>> they're loaded into memory.  Someone with a jillion packages surely doesn't
>> need them all at once and can benefit from a carefully-crafted init.el with
>> deferred package loading.
>>
>
> FWIW, I did investigate whether we could enable DYLD environment variable
> (a different codesign entitlement) to see if we could allow the Emacs
> loader to temporarily disable library validation when loading a file from
> the eln cache.  I also took a casual look at the source code here
> https://github.com/apple-oss-distributions/dyld I failed to find a way to
> create a finer-grained approach.  Perhaps an enterprising person will dig
> something up.  For now, it seems it's all or nothing.
>

One thing I did not look at that someone may want to look into is whether
it is possible to codesign the files in the eln cache using the same
credentials/opt outs as the Emacs bundle (or ad-hoc signing) to avoid
library validation.  If that works, perhaps a hook could be added to the
native compiler to allow some post processing of each file to codesign it
as a one-time cost.