Re: Deferring native compilation finalization until idle

On Sat, Jul 25, 2026 at 12:29 PM Stéphane Marks <[email protected]> wrote:

> On Sat, Jul 25, 2026 at 12:22 PM Aaron Jensen <[email protected]>
> wrote:
>
>> On Sat, Jul 25, 2026 at 9:18 AM, Stéphane Marks <[email protected]>
>> wrote:
>>
>>> On Sat, Jul 25, 2026 at 12:14 PM Aaron Jensen <[email protected]>
>>> wrote:
>>>
>>>> On Sat, Jul 25, 2026 at 9:04 AM, Stéphane Marks <[email protected]>
>>>> wrote:
>>>>
>>>>> FWIW, my eln-cache this morning, starting fresh from Thursday, has
>>>>> 146 files with seldom used packages uncompiled, of course.  Rebuilding that
>>>>> cache was entirely asynchronous, perhaps except for the trampolines.  I
>>>>> have not tried setting native-comp-enable-subr-trampolines to nil in
>>>>> early-init.el to see if this bypasses the delays that you're seeing.  Have
>>>>> you tried that?
>>>>>
>>>>>>
>>>>>>
>>>> Setting native-comp-enable-subr-trampolines to nil does not noticeably
>>>> reduce the stutter for me.
>>>>
>>>> Ending up with only 146 files likely means that even if you were to not
>>>> have one of the workarounds in place, you would not notice as much stutter
>>>> as one who had more.
>>>>
>>>> My guess is that if you were to try my config, or someone else's that
>>>> notices the issue and not disable macOS's security checks, you too would
>>>> notice the stutter.
>>>>
>>>> I work with others that use macOS. I can survey them to see how many of
>>>> them have issues if that would be convincing to anyone. However, I think
>>>> the facts are pretty well established and ultimately it's a decision as to
>>>> whether or not the maintainers want to account for the particularities of
>>>> macOS's bundle validation process by loading the bundle within the async
>>>> worker (and take on the additional maintenance burden).
>>>>
>>>> Personally, I have my workaround, so I'm "fine". But I'm not a fan of
>>>> macOS users that have have sufficient packages having an experience that
>>>> leaves them with the opinion that Emacs has poor startup performance, or
>>>> native compilation makes things worse, etc. Yes, those are
>>>> narrowly-informed opinions, but I'm sure many users would not take the time
>>>> to compare their startup performance to other OS's and/or to find this
>>>> thread to better understand the depths of the problem. Instead, they'll
>>>> just have a bad taste and deal with it every time they install a new
>>>> version of Emacs.
>>>>
>>>
>>> From first principles, what package is waiting to be built in the
>>> eln-cache that Emacs is waiting for that prevents input?  AFAIK, the byte
>>> code is first loaded and available to run and only then replaced on-the-fly
>>> when native code is produced without error to replace it in the symbol's
>>> function cell.
>>>
>>
>> No, this is not what the problem is. That's the thing that I'm saying has
>> been pretty well established in this thread, though it seems not well
>> enough.
>>
>> The problem is that macOS, upon first *loading* the bundle in the main
>> Emacs process will undergo a security check (the one disabled by the
>> entitlements call you posted). This is a synchronous, blocking operation.
>> MacOS does not allow the process to load the bundle until it has verified
>> it, so the load is blocking. During this time, Emacs cannot receive input
>> because the main thread is completely blocked by the operating system. As
>> soon as you chain together hundreds of these macOS-initiated blocking
>> operations (every single bundle being loaded) you have a very poor,
>> stuttery, slow experience.
>>
>
> Okay so it is just the macOS binary scanner then and like any virus
> scanner interferes at the OS level.  A binary has to either opt out (as
> PROBLEMS document a method to do and the jimeh build does), or has to
> otherwise be whitelisted.  Every binary on macOS undergoes the same thing
> without being opted out so this isn't an Emacs issue.  I agree that Emacs
> loads "DLLs" and those can amplify the OS-imposed delay.  But only when
> they're loaded into memory.  Someone with a jillion packages surely doesn't
> need them all at once and can benefit from a carefully-crafted init.el with
> deferred package loading.
>

FWIW, I did investigate whether we could enable DYLD environment variable
(a different codesign entitlement) to see if we could allow the Emacs
loader to temporarily disable library validation when loading a file from
the eln cache.  I also took a casual look at the source code here
https://github.com/apple-oss-distributions/dyld I failed to find a way to
create a finer-grained approach.  Perhaps an enterprising person will dig
something up.  For now, it seems it's all or nothing.