Re: Race condition on loading packages from 2 maxima processes in parallel?
Raymond Toy <[email protected]> Wed, 17 Jun 2026 06:32:06 -0700
| Newsgroups | gmane.comp.mathematics.maxima.general |
|---|---|
| Message-ID | <[email protected]> |
On 6/17/26 2:30 AM, David Scherfgen via Maxima-discuss wrote:
> LAPACK uses mk-defsystem (we ship that with Maxima, see
> lisp-utils/defsystem.lisp) to compile itself. For each source file
> belonging to the LAPACK system, mk-defsystem checks whether the binary
> (.fasl) file already exists. If yes, it loads it. If not, it compiles
> the source and creates the binary file.
> We can patch it so that if the binary file exists, but cannot be
> loaded, it tries to compile it again (only one single time, to prevent
> endless loops).
>
> In lisp-utils/defsystem.lisp, replace
>
> ((and binary-exists load-binary)
> (with-tell-user ("Loading binary" component :binary)
> (or *oos-test*
> (progn
> (funcall (load-function component) binary-pname)
> (setf (component-load-time component)
> (file-write-date binary-pname)))))
> t)
>
> with
>
> ((and binary-exists load-binary)
> (with-tell-user ("Loading binary" component :binary)
> (or *oos-test*
> (handler-case
> (progn
> (funcall (load-function component) binary-pname)
> (setf (component-load-time component)
> (file-write-date binary-pname)))
> (error (condition)
> (format *error-output*
> "MK:DEFSYSTEM: Error loading ~A: ~A~%; Deleting
> binary and recompiling...~%"
> binary-pname condition)
> ;; Catch and report deletion errors without halting recovery
> (handler-case (delete-file binary-pname)
> (error (del-err)
> (format *error-output*
> "MK:DEFSYSTEM: Warning: Could not delete ~A: ~A~%;
> Recompiling anyways...~%"
> binary-pname del-err)))
> ;; Force recompilation
> (compile-file-operation component t)
> ;; Final load attempt
> (funcall (load-function component) binary-pname)
> (setf (component-load-time component)
> (file-write-date binary-pname))))))
> t)
>
> Works for me.
>
> Should I commit that?
I’m ok with that. But I’d rather update defsystem to a later version.
See https://gitlab.common-lisp.net/mantoniotti/mk-defsystem.
I tried this a while ago and it worked with all lisp including gcl with
a minor patch. The latest version, however, doesn’t work. I filed a bug
on that.
Or, from what I gather, the development version of gcl can actually run
asdf (maybe with a few minor updates to asdf?). That would be pretty sweet.
>
> Best regards
> David Scherfgen
>
> Am Mi., 17. Juni 2026 um 07:27 Uhr schrieb Gunter Königsmann
> <[email protected]>:
>
> ...and in the meantime I found out that we don't even require a
> race condition to end up with this problem:
>
> * Tried a load(lapack); on SBCL that ran out of reserved heap
> space => told maxima via command-line argument to reserve more
> heap. Result: A system that would have been able to load and
> compile lapack, but lied about not being able to load lapack
> from /usr/lib when not being able to load the partial
> compilation results from ~/.maxima/binary
> * Deleted ~/.maxima/binary and told a user to continue my work.
> User tried to load lapack, after waiting 2 minutes assumed
> that maxima has crashed, restarted maxima (with the correct
> command-line arguments) and maxima lied about having failed
> loading lapack from /usr/lib while trying to load another
> incomplete compilation result from the home dir, instead.
>
> => would another approach be possible?
>
> * execute load() the normal way, and
> * if that failed ignore the contents of ~/.maxima/binary and try
> again?
>
> In the end ~./.maxima/binary is merely a cache - that is easy to
> cause to contain wrong data.
>
> Kind regards,
>
> Gunter.
>
> On 6/17/26 07:13, David Scherfgen wrote:
>> unwind-protect wouldn't help in case of a "hard crash". In
>> theory, the process that currently holds the lock could use a
>> thread to modify the lockfile in regular intervals, resetting its
>> file-write-date, acting as a signal that the process is still
>> alive. If it really crashes, it no longer modifies the file
>> (file-write-date stays the same), and eventually another waiting
>> process would notice, delete the lockfile and enter the lock itself.
>> But that's quite sophisticated already, maybe it all isn't really
>> much of a problem.
>>
>> Gunter K�nigsmann <[email protected]> schrieb am Mi., 17. Juni
>> 2026, 07:02:
>>
>> Would a unwind-protect that removes the lock typically help
>> in case of a crash? If power browns out while saving perhaps
>> it is reasonable to remove the lock after waiting for 3
>> minutes or so...
>>
>>
>> On 14 June 2026 16:00:19 CEST, David Scherfgen via
>> Maxima-discuss <[email protected]> wrote:
>>
>> Gemini proposed this Common Lisp file locking mechanism
>> (but read below):
>>
>> (defmacro with-file-lock ((lock-file &key (sleep-time
>> 0.1)) &body body)
>> "Acquires an exclusive lock using purely ANSI Common
>> Lisp atomic file creation."
>> (let ((stream-sym (gensym "STREAM")))
>> `(let ((,stream-sym nil))
>> (unwind-protect
>> (progn
>> ;; Spinlock: keep trying to create the file
>> until successful
>> (loop
>> (handler-case
>> (progn
>> (setf ,stream-sym (open ,lock-file
>> :direction :output
>> :if-exists :error
>> :if-does-not-exist :create))
>> (return)) ; Successfully acquired
>> lock; exit the loop
>> (file-error ()
>> ;; Lock file exists; wait and retry
>> (sleep ,sleep-time))))
>> ;; Execute the compilation or protected code
>> ,@body)
>> ;; Cleanup: close the stream and delete the lock
>> file
>> (when ,stream-sym
>> (close ,stream-sym)
>> (ignore-errors (delete-file ,lock-file)))))))
>>
>> I tried it by having multiple processes do this:
>>
>> (with-file-lock (".lockfile") (format t "Entered~%")
>> (sleep 30) (format t "Exited~%"))
>>
>> Seems to work ...
>> There could be a problem, though, when a process crashes
>> / gets killed while it has the lock. Then it won't
>> release the lock, and no process could ever acquire it again.
>> In theory, one could define a "timeout" and say that if
>> the lockfile exists and is too old (check using
>> file-write-date), then the process probably crashed.
>> For a robust solution, one should use e.g. POSIX flock.
>>
>> Best regards
>> David Scherfgen
>>
>> Am So., 14. Juni 2026 um 15:40 Uhr schrieb Raymond Toy
>> <[email protected]>:
>>
>> On 6/14/26 5:05 AM, Gunter Königsmann via
>> Maxima-discuss wrote:
>>
>>> Dear all,
>>>
>>> wxMaxima on every commit creates a new virtual
>>> machine, compiles wxMaxima in that machine and runs
>>> a few hundred tests in that machine.
>>>
>>> If it runs more than one test in parallel sometimes
>>> maxima fails to load draw.
>>>
>>> My theory is:
>>>
>>> * One maxima process tries to load draw, compiles
>>> that package and starts saving it.
>>> * A second maxima process tries to load the
>>> compiled draw package while that compiled
>>> package still being written.
>>>
>>> Could that be the case?
>>>
>> Yes. Look at share/draw/draw.lisp. It runs |mk:oos|
>> to compile all the files from the draw package.
>> ​
>> _______________________________________________
>> Maxima-discuss mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/maxima-discuss
>>
>
>
> _______________________________________________
> Maxima-discuss mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/maxima-discuss
​
_______________________________________________
Maxima-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/maxima-discuss