Re: Race condition on loading packages from 2 maxima processes in parallel?

LAPACK uses mk-defsystem (we ship that with Maxima, see
lisp-utils/defsystem.lisp) to compile itself. For each source file
belonging to the LAPACK system, mk-defsystem checks whether the binary
(.fasl) file already exists. If yes, it loads it. If not, it compiles the
source and creates the binary file.
We can patch it so that if the binary file exists, but cannot be loaded, it
tries to compile it again (only one single time, to prevent endless loops).

In lisp-utils/defsystem.lisp, replace

     ((and binary-exists load-binary)
    (with-tell-user ("Loading binary"   component :binary)
      (or *oos-test*
  (progn
    (funcall (load-function component) binary-pname)
    (setf (component-load-time component)
  (file-write-date binary-pname)))))
    t)

with

     ((and binary-exists load-binary)
    (with-tell-user ("Loading binary"   component :binary)
      (or *oos-test*
  (handler-case
      (progn
(funcall (load-function component) binary-pname)
(setf (component-load-time component)
      (file-write-date binary-pname)))
    (error (condition)
      (format *error-output*
                 "MK:DEFSYSTEM: Error loading ~A: ~A~%; Deleting binary and
recompiling...~%"
    binary-pname condition)
      ;; Catch and report deletion errors without halting recovery
      (handler-case (delete-file binary-pname)
(error (del-err)
  (format *error-output*
                 "MK:DEFSYSTEM: Warning: Could not delete ~A: ~A~%;
Recompiling anyways...~%"
binary-pname del-err)))
      ;; Force recompilation
      (compile-file-operation component t)
      ;; Final load attempt
      (funcall (load-function component) binary-pname)
      (setf (component-load-time component)
    (file-write-date binary-pname))))))
    t)

Works for me.

Should I commit that?

Best regards
David Scherfgen

Am Mi., 17. Juni 2026 um 07:27 Uhr schrieb Gunter Königsmann <
[email protected]>:

> ...and in the meantime I found out that we don't even require a race
> condition to end up with this problem:
>
>    - Tried a load(lapack); on SBCL that ran out of reserved heap space =>
>    told maxima via command-line argument to reserve more heap. Result: A
>    system that would have been able to load and compile lapack, but lied about
>    not being able to load lapack from /usr/lib when not being able to load the
>    partial compilation results from ~/.maxima/binary
>    - Deleted ~/.maxima/binary and told a user to continue my work. User
>    tried to load lapack, after waiting 2 minutes assumed that maxima has
>    crashed, restarted maxima (with the correct command-line arguments) and
>    maxima lied about having failed loading lapack from /usr/lib while trying
>    to load another incomplete compilation result from the home dir, instead.
>
> => would another approach be possible?
>
>    - execute load() the normal way, and
>    - if that failed ignore the contents of ~/.maxima/binary and try again?
>
> In the end ~./.maxima/binary is merely a cache - that is easy to cause to
> contain wrong data.
>
> Kind regards,
>
>        Gunter.
> On 6/17/26 07:13, David Scherfgen wrote:
>
> unwind-protect wouldn't help in case of a "hard crash". In theory, the
> process that currently holds the lock could use a thread to modify the
> lockfile in regular intervals, resetting its file-write-date, acting as a
> signal that the process is still alive. If it really crashes, it no longer
> modifies the file (file-write-date stays the same), and eventually another
> waiting process would notice, delete the lockfile and enter the lock
> itself.
> But that's quite sophisticated already, maybe it all isn't really much of
> a problem.
>
> Gunter K�nigsmann <[email protected]> schrieb am Mi., 17. Juni 2026,
> 07:02:
>
>> Would a unwind-protect that removes the lock typically help in case of a
>> crash? If power browns out while saving perhaps it is reasonable to remove
>> the lock after waiting for 3 minutes or so...
>>
>>
>> On 14 June 2026 16:00:19 CEST, David Scherfgen via Maxima-discuss <
>> [email protected]> wrote:
>>
>>> Gemini proposed this Common Lisp file locking mechanism (but read
>>> below):
>>>
>>> (defmacro with-file-lock ((lock-file &key (sleep-time 0.1)) &body body)
>>>   "Acquires an exclusive lock using purely ANSI Common Lisp atomic file
>>> creation."
>>>   (let ((stream-sym (gensym "STREAM")))
>>>     `(let ((,stream-sym nil))
>>>        (unwind-protect
>>>            (progn
>>>              ;; Spinlock: keep trying to create the file until successful
>>>              (loop
>>>                (handler-case
>>>                    (progn
>>>                      (setf ,stream-sym (open ,lock-file
>>>                                              :direction :output
>>>                                              :if-exists :error
>>>                                              :if-does-not-exist :create))
>>>                      (return)) ; Successfully acquired lock; exit the
>>> loop
>>>                  (file-error ()
>>>                    ;; Lock file exists; wait and retry
>>>                    (sleep ,sleep-time))))
>>>              ;; Execute the compilation or protected code
>>>              ,@body)
>>>          ;; Cleanup: close the stream and delete the lock file
>>>          (when ,stream-sym
>>>            (close ,stream-sym)
>>>            (ignore-errors (delete-file ,lock-file)))))))
>>>
>>> I tried it by having multiple processes do this:
>>>
>>> (with-file-lock (".lockfile") (format t "Entered~%") (sleep 30) (format
>>> t "Exited~%"))
>>>
>>> Seems to work ...
>>> There could be a problem, though, when a process crashes / gets killed
>>> while it has the lock. Then it won't release the lock, and no process could
>>> ever acquire it again.
>>> In theory, one could define a "timeout" and say that if the lockfile
>>> exists and is too old (check using file-write-date), then the process
>>> probably crashed.
>>> For a robust solution, one should use e.g. POSIX flock.
>>>
>>> Best regards
>>> David Scherfgen
>>>
>>> Am So., 14. Juni 2026 um 15:40 Uhr schrieb Raymond Toy <
>>> [email protected]>:
>>>
>>>> On 6/14/26 5:05 AM, Gunter Königsmann via Maxima-discuss wrote:
>>>>
>>>> Dear all,
>>>>
>>>> wxMaxima on every commit creates a new virtual machine, compiles
>>>> wxMaxima in that machine and runs a few hundred tests in that machine.
>>>>
>>>> If it runs more than one test in parallel sometimes maxima fails to
>>>> load draw.
>>>>
>>>> My theory is:
>>>>
>>>>    - One maxima process tries to load draw, compiles that package and
>>>>    starts saving it.
>>>>    - A second maxima process tries to load the compiled draw package
>>>>    while that compiled package still being written.
>>>>
>>>> Could that be the case?
>>>>
>>>> Yes. Look at share/draw/draw.lisp. It runs mk:oos to compile all the
>>>> files from the draw package.
>>>> &#8203;
>>>> _______________________________________________
>>>> Maxima-discuss mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/maxima-discuss
>>>>
>>>

_______________________________________________
Maxima-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/maxima-discuss