Re: Race condition on loading packages from 2 maxima processes in parallel?

I think the objection was "Why risk changing something that works?". And
actually, that's true. Is there something broken with the old version we're
using, except for the compiler warnings?
I'll commit the patch if that's OK. It's simple enough to reapply it
if/when we update defsystem. I'll also try to contact the developer and
propose to integrate it.

Am Mi., 17. Juni 2026 um 21:59 Uhr schrieb Raymond Toy <
[email protected]>:

> On 6/17/26 6:43 AM, David Scherfgen wrote:
>
> I think I proposed updating defsystem some time ago (the main reason being
> that I got annoyed by the tons of compiler warnings), but there were
> objections (I don't remember by whom).
> Anyways, I have looked at the newest version, and it has the same problem,
> as far as I can tell. It doesn't try to handle errors occurring while
> loading a binary.
>
> Yes, that’s true. Don’t know recall what the objections were either. But
> certainly I’d prefer modifying a more up-to-date version than the ancient
> version we have.
>
> I don’t know how to resolve this issue. Maybe just wait until the next
> release of gcl which will hopefully support asdf and drop support of
> defsystem? We’d have to update some things that use defsystem to build, but
> that should be straightforward.
>
>
> Am Mi., 17. Juni 2026 um 15:37 Uhr schrieb Raymond Toy <
> [email protected]>:
>
>> On 6/17/26 2:30 AM, David Scherfgen via Maxima-discuss wrote:
>>
>> LAPACK uses mk-defsystem (we ship that with Maxima, see
>> lisp-utils/defsystem.lisp) to compile itself. For each source file
>> belonging to the LAPACK system, mk-defsystem checks whether the binary
>> (.fasl) file already exists. If yes, it loads it. If not, it compiles the
>> source and creates the binary file.
>> We can patch it so that if the binary file exists, but cannot be loaded,
>> it tries to compile it again (only one single time, to prevent endless
>> loops).
>>
>> In lisp-utils/defsystem.lisp, replace
>>
>>      ((and binary-exists load-binary)
>>     (with-tell-user ("Loading binary"   component :binary)
>>       (or *oos-test*
>>   (progn
>>     (funcall (load-function component) binary-pname)
>>     (setf (component-load-time component)
>>   (file-write-date binary-pname)))))
>>     t)
>>
>> with
>>
>>      ((and binary-exists load-binary)
>>     (with-tell-user ("Loading binary"   component :binary)
>>       (or *oos-test*
>>   (handler-case
>>       (progn
>> (funcall (load-function component) binary-pname)
>> (setf (component-load-time component)
>>       (file-write-date binary-pname)))
>>     (error (condition)
>>       (format *error-output*
>>                  "MK:DEFSYSTEM: Error loading ~A: ~A~%; Deleting binary
>> and recompiling...~%"
>>     binary-pname condition)
>>       ;; Catch and report deletion errors without halting recovery
>>       (handler-case (delete-file binary-pname)
>> (error (del-err)
>>   (format *error-output*
>>                  "MK:DEFSYSTEM: Warning: Could not delete ~A: ~A~%;
>> Recompiling anyways...~%"
>> binary-pname del-err)))
>>       ;; Force recompilation
>>       (compile-file-operation component t)
>>       ;; Final load attempt
>>       (funcall (load-function component) binary-pname)
>>       (setf (component-load-time component)
>>     (file-write-date binary-pname))))))
>>     t)
>>
>> Works for me.
>>
>> Should I commit that?
>>
>>
>> I’m ok with that. But I’d rather update defsystem to a later version. See
>> https://gitlab.common-lisp.net/mantoniotti/mk-defsystem.
>>
>> I tried this a while ago and it worked with all lisp including gcl with a
>> minor patch. The latest version, however, doesn’t work. I filed a bug on
>> that.
>>
>> Or, from what I gather, the development version of gcl can actually run
>> asdf (maybe with a few minor updates to asdf?). That would be pretty sweet.
>>
>>
>> Best regards
>> David Scherfgen
>>
>> Am Mi., 17. Juni 2026 um 07:27 Uhr schrieb Gunter Königsmann <
>> [email protected]>:
>>
>>> ...and in the meantime I found out that we don't even require a race
>>> condition to end up with this problem:
>>>
>>>    - Tried a load(lapack); on SBCL that ran out of reserved heap space
>>>    => told maxima via command-line argument to reserve more heap. Result: A
>>>    system that would have been able to load and compile lapack, but lied about
>>>    not being able to load lapack from /usr/lib when not being able to load the
>>>    partial compilation results from ~/.maxima/binary
>>>    - Deleted ~/.maxima/binary and told a user to continue my work. User
>>>    tried to load lapack, after waiting 2 minutes assumed that maxima has
>>>    crashed, restarted maxima (with the correct command-line arguments) and
>>>    maxima lied about having failed loading lapack from /usr/lib while trying
>>>    to load another incomplete compilation result from the home dir, instead.
>>>
>>> => would another approach be possible?
>>>
>>>    - execute load() the normal way, and
>>>    - if that failed ignore the contents of ~/.maxima/binary and try
>>>    again?
>>>
>>> In the end ~./.maxima/binary is merely a cache - that is easy to cause
>>> to contain wrong data.
>>>
>>> Kind regards,
>>>
>>>        Gunter.
>>> On 6/17/26 07:13, David Scherfgen wrote:
>>>
>>> unwind-protect wouldn't help in case of a "hard crash". In theory, the
>>> process that currently holds the lock could use a thread to modify the
>>> lockfile in regular intervals, resetting its file-write-date, acting as a
>>> signal that the process is still alive. If it really crashes, it no longer
>>> modifies the file (file-write-date stays the same), and eventually another
>>> waiting process would notice, delete the lockfile and enter the lock
>>> itself.
>>> But that's quite sophisticated already, maybe it all isn't really much
>>> of a problem.
>>>
>>> Gunter K�nigsmann <[email protected]> schrieb am Mi., 17. Juni 2026,
>>> 07:02:
>>>
>>>> Would a unwind-protect that removes the lock typically help in case of
>>>> a crash? If power browns out while saving perhaps it is reasonable to
>>>> remove the lock after waiting for 3 minutes or so...
>>>>
>>>>
>>>> On 14 June 2026 16:00:19 CEST, David Scherfgen via Maxima-discuss <
>>>> [email protected]> wrote:
>>>>
>>>>> Gemini proposed this Common Lisp file locking mechanism (but read
>>>>> below):
>>>>>
>>>>> (defmacro with-file-lock ((lock-file &key (sleep-time 0.1)) &body body)
>>>>>   "Acquires an exclusive lock using purely ANSI Common Lisp atomic
>>>>> file creation."
>>>>>   (let ((stream-sym (gensym "STREAM")))
>>>>>     `(let ((,stream-sym nil))
>>>>>        (unwind-protect
>>>>>            (progn
>>>>>              ;; Spinlock: keep trying to create the file until
>>>>> successful
>>>>>              (loop
>>>>>                (handler-case
>>>>>                    (progn
>>>>>                      (setf ,stream-sym (open ,lock-file
>>>>>                                              :direction :output
>>>>>                                              :if-exists :error
>>>>>                                              :if-does-not-exist
>>>>> :create))
>>>>>                      (return)) ; Successfully acquired lock; exit the
>>>>> loop
>>>>>                  (file-error ()
>>>>>                    ;; Lock file exists; wait and retry
>>>>>                    (sleep ,sleep-time))))
>>>>>              ;; Execute the compilation or protected code
>>>>>              ,@body)
>>>>>          ;; Cleanup: close the stream and delete the lock file
>>>>>          (when ,stream-sym
>>>>>            (close ,stream-sym)
>>>>>            (ignore-errors (delete-file ,lock-file)))))))
>>>>>
>>>>> I tried it by having multiple processes do this:
>>>>>
>>>>> (with-file-lock (".lockfile") (format t "Entered~%") (sleep 30)
>>>>> (format t "Exited~%"))
>>>>>
>>>>> Seems to work ...
>>>>> There could be a problem, though, when a process crashes / gets killed
>>>>> while it has the lock. Then it won't release the lock, and no process could
>>>>> ever acquire it again.
>>>>> In theory, one could define a "timeout" and say that if the lockfile
>>>>> exists and is too old (check using file-write-date), then the process
>>>>> probably crashed.
>>>>> For a robust solution, one should use e.g. POSIX flock.
>>>>>
>>>>> Best regards
>>>>> David Scherfgen
>>>>>
>>>>> Am So., 14. Juni 2026 um 15:40 Uhr schrieb Raymond Toy <
>>>>> [email protected]>:
>>>>>
>>>>>> On 6/14/26 5:05 AM, Gunter Königsmann via Maxima-discuss wrote:
>>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> wxMaxima on every commit creates a new virtual machine, compiles
>>>>>> wxMaxima in that machine and runs a few hundred tests in that machine.
>>>>>>
>>>>>> If it runs more than one test in parallel sometimes maxima fails to
>>>>>> load draw.
>>>>>>
>>>>>> My theory is:
>>>>>>
>>>>>>    - One maxima process tries to load draw, compiles that package
>>>>>>    and starts saving it.
>>>>>>    - A second maxima process tries to load the compiled draw package
>>>>>>    while that compiled package still being written.
>>>>>>
>>>>>> Could that be the case?
>>>>>>
>>>>>> Yes. Look at share/draw/draw.lisp. It runs mk:oos to compile all the
>>>>>> files from the draw package.
>>>>>> &#8203;
>>>>>> _______________________________________________
>>>>>> Maxima-discuss mailing list
>>>>>> [email protected]
>>>>>> https://lists.sourceforge.net/lists/listinfo/maxima-discuss
>>>>>>
>>>>>
>>
>> _______________________________________________
>> Maxima-discuss mailing [email protected]://lists.sourceforge.net/lists/listinfo/maxima-discuss
>>
>> &#8203;
>> _______________________________________________
>> Maxima-discuss mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/maxima-discuss
>>
> &#8203;
>

_______________________________________________
Maxima-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/maxima-discuss