Re: Race condition on loading packages from 2 maxima processes in parallel?

I think I proposed updating defsystem some time ago (the main reason being
that I got annoyed by the tons of compiler warnings), but there were
objections (I don't remember by whom).
Anyways, I have looked at the newest version, and it has the same problem,
as far as I can tell. It doesn't try to handle errors occurring while
loading a binary.

Am Mi., 17. Juni 2026 um 15:37 Uhr schrieb Raymond Toy <
[email protected]>:

> On 6/17/26 2:30 AM, David Scherfgen via Maxima-discuss wrote:
>
> LAPACK uses mk-defsystem (we ship that with Maxima, see
> lisp-utils/defsystem.lisp) to compile itself. For each source file
> belonging to the LAPACK system, mk-defsystem checks whether the binary
> (.fasl) file already exists. If yes, it loads it. If not, it compiles the
> source and creates the binary file.
> We can patch it so that if the binary file exists, but cannot be loaded,
> it tries to compile it again (only one single time, to prevent endless
> loops).
>
> In lisp-utils/defsystem.lisp, replace
>
>      ((and binary-exists load-binary)
>     (with-tell-user ("Loading binary"   component :binary)
>       (or *oos-test*
>   (progn
>     (funcall (load-function component) binary-pname)
>     (setf (component-load-time component)
>   (file-write-date binary-pname)))))
>     t)
>
> with
>
>      ((and binary-exists load-binary)
>     (with-tell-user ("Loading binary"   component :binary)
>       (or *oos-test*
>   (handler-case
>       (progn
> (funcall (load-function component) binary-pname)
> (setf (component-load-time component)
>       (file-write-date binary-pname)))
>     (error (condition)
>       (format *error-output*
>                  "MK:DEFSYSTEM: Error loading ~A: ~A~%; Deleting binary
> and recompiling...~%"
>     binary-pname condition)
>       ;; Catch and report deletion errors without halting recovery
>       (handler-case (delete-file binary-pname)
> (error (del-err)
>   (format *error-output*
>                  "MK:DEFSYSTEM: Warning: Could not delete ~A: ~A~%;
> Recompiling anyways...~%"
> binary-pname del-err)))
>       ;; Force recompilation
>       (compile-file-operation component t)
>       ;; Final load attempt
>       (funcall (load-function component) binary-pname)
>       (setf (component-load-time component)
>     (file-write-date binary-pname))))))
>     t)
>
> Works for me.
>
> Should I commit that?
>
>
> I’m ok with that. But I’d rather update defsystem to a later version. See
> https://gitlab.common-lisp.net/mantoniotti/mk-defsystem.
>
> I tried this a while ago and it worked with all lisp including gcl with a
> minor patch. The latest version, however, doesn’t work. I filed a bug on
> that.
>
> Or, from what I gather, the development version of gcl can actually run
> asdf (maybe with a few minor updates to asdf?). That would be pretty sweet.
>
>
> Best regards
> David Scherfgen
>
> Am Mi., 17. Juni 2026 um 07:27 Uhr schrieb Gunter Königsmann <
> [email protected]>:
>
>> ...and in the meantime I found out that we don't even require a race
>> condition to end up with this problem:
>>
>>    - Tried a load(lapack); on SBCL that ran out of reserved heap space
>>    => told maxima via command-line argument to reserve more heap. Result: A
>>    system that would have been able to load and compile lapack, but lied about
>>    not being able to load lapack from /usr/lib when not being able to load the
>>    partial compilation results from ~/.maxima/binary
>>    - Deleted ~/.maxima/binary and told a user to continue my work. User
>>    tried to load lapack, after waiting 2 minutes assumed that maxima has
>>    crashed, restarted maxima (with the correct command-line arguments) and
>>    maxima lied about having failed loading lapack from /usr/lib while trying
>>    to load another incomplete compilation result from the home dir, instead.
>>
>> => would another approach be possible?
>>
>>    - execute load() the normal way, and
>>    - if that failed ignore the contents of ~/.maxima/binary and try
>>    again?
>>
>> In the end ~./.maxima/binary is merely a cache - that is easy to cause to
>> contain wrong data.
>>
>> Kind regards,
>>
>>        Gunter.
>> On 6/17/26 07:13, David Scherfgen wrote:
>>
>> unwind-protect wouldn't help in case of a "hard crash". In theory, the
>> process that currently holds the lock could use a thread to modify the
>> lockfile in regular intervals, resetting its file-write-date, acting as a
>> signal that the process is still alive. If it really crashes, it no longer
>> modifies the file (file-write-date stays the same), and eventually another
>> waiting process would notice, delete the lockfile and enter the lock
>> itself.
>> But that's quite sophisticated already, maybe it all isn't really much of
>> a problem.
>>
>> Gunter K�nigsmann <[email protected]> schrieb am Mi., 17. Juni 2026,
>> 07:02:
>>
>>> Would a unwind-protect that removes the lock typically help in case of a
>>> crash? If power browns out while saving perhaps it is reasonable to remove
>>> the lock after waiting for 3 minutes or so...
>>>
>>>
>>> On 14 June 2026 16:00:19 CEST, David Scherfgen via Maxima-discuss <
>>> [email protected]> wrote:
>>>
>>>> Gemini proposed this Common Lisp file locking mechanism (but read
>>>> below):
>>>>
>>>> (defmacro with-file-lock ((lock-file &key (sleep-time 0.1)) &body body)
>>>>   "Acquires an exclusive lock using purely ANSI Common Lisp atomic file
>>>> creation."
>>>>   (let ((stream-sym (gensym "STREAM")))
>>>>     `(let ((,stream-sym nil))
>>>>        (unwind-protect
>>>>            (progn
>>>>              ;; Spinlock: keep trying to create the file until
>>>> successful
>>>>              (loop
>>>>                (handler-case
>>>>                    (progn
>>>>                      (setf ,stream-sym (open ,lock-file
>>>>                                              :direction :output
>>>>                                              :if-exists :error
>>>>                                              :if-does-not-exist
>>>> :create))
>>>>                      (return)) ; Successfully acquired lock; exit the
>>>> loop
>>>>                  (file-error ()
>>>>                    ;; Lock file exists; wait and retry
>>>>                    (sleep ,sleep-time))))
>>>>              ;; Execute the compilation or protected code
>>>>              ,@body)
>>>>          ;; Cleanup: close the stream and delete the lock file
>>>>          (when ,stream-sym
>>>>            (close ,stream-sym)
>>>>            (ignore-errors (delete-file ,lock-file)))))))
>>>>
>>>> I tried it by having multiple processes do this:
>>>>
>>>> (with-file-lock (".lockfile") (format t "Entered~%") (sleep 30) (format
>>>> t "Exited~%"))
>>>>
>>>> Seems to work ...
>>>> There could be a problem, though, when a process crashes / gets killed
>>>> while it has the lock. Then it won't release the lock, and no process could
>>>> ever acquire it again.
>>>> In theory, one could define a "timeout" and say that if the lockfile
>>>> exists and is too old (check using file-write-date), then the process
>>>> probably crashed.
>>>> For a robust solution, one should use e.g. POSIX flock.
>>>>
>>>> Best regards
>>>> David Scherfgen
>>>>
>>>> Am So., 14. Juni 2026 um 15:40 Uhr schrieb Raymond Toy <
>>>> [email protected]>:
>>>>
>>>>> On 6/14/26 5:05 AM, Gunter Königsmann via Maxima-discuss wrote:
>>>>>
>>>>> Dear all,
>>>>>
>>>>> wxMaxima on every commit creates a new virtual machine, compiles
>>>>> wxMaxima in that machine and runs a few hundred tests in that machine.
>>>>>
>>>>> If it runs more than one test in parallel sometimes maxima fails to
>>>>> load draw.
>>>>>
>>>>> My theory is:
>>>>>
>>>>>    - One maxima process tries to load draw, compiles that package and
>>>>>    starts saving it.
>>>>>    - A second maxima process tries to load the compiled draw package
>>>>>    while that compiled package still being written.
>>>>>
>>>>> Could that be the case?
>>>>>
>>>>> Yes. Look at share/draw/draw.lisp. It runs mk:oos to compile all the
>>>>> files from the draw package.
>>>>> &#8203;
>>>>> _______________________________________________
>>>>> Maxima-discuss mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/maxima-discuss
>>>>>
>>>>
>
> _______________________________________________
> Maxima-discuss mailing [email protected]://lists.sourceforge.net/lists/listinfo/maxima-discuss
>
> &#8203;
> _______________________________________________
> Maxima-discuss mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/maxima-discuss
>

_______________________________________________
Maxima-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/maxima-discuss