Re: Race condition on loading packages from 2 maxima processes in parallel?
Raymond Toy <[email protected]> Wed, 17 Jun 2026 12:59:27 -0700
| Newsgroups | gmane.comp.mathematics.maxima.general |
|---|---|
| Message-ID | <[email protected]> |
On 6/17/26 6:43 AM, David Scherfgen wrote: > I think I proposed updating defsystem some time ago (the main reason > being that I got annoyed by the tons of compiler warnings), but there > were objections (I don't remember by whom). > Anyways, I have looked at the newest version, and it has the same > problem, as far as I can tell. It doesn't try to handle errors > occurring while loading a binary. Yes, that’s true. Don’t know recall what the objections were either. But certainly I’d prefer modifying a more up-to-date version than the ancient version we have. I don’t know how to resolve this issue. Maybe just wait until the next release of gcl which will hopefully support asdf and drop support of defsystem? We’d have to update some things that use defsystem to build, but that should be straightforward. > > Am Mi., 17. Juni 2026 um 15:37 Uhr schrieb Raymond Toy > <[email protected]>: > > On 6/17/26 2:30 AM, David Scherfgen via Maxima-discuss wrote: > >> LAPACK uses mk-defsystem (we ship that with Maxima, see >> lisp-utils/defsystem.lisp) to compile itself. For each source >> file belonging to the LAPACK system, mk-defsystem checks whether >> the binary (.fasl) file already exists. If yes, it loads it. If >> not, it compiles the source and creates the binary file. >> We can patch it so that if the binary file exists, but cannot be >> loaded, it tries to compile it again (only one single time, to >> prevent endless loops). >> >> In lisp-utils/defsystem.lisp, replace >> >> ((and binary-exists load-binary) >> (with-tell-user ("Loading binary" component :binary) >> (or *oos-test* >> (progn >> (funcall (load-function component) binary-pname) >> (setf (component-load-time component) >> (file-write-date binary-pname))))) >> t) >> >> with >> >> ((and binary-exists load-binary) >> (with-tell-user ("Loading binary" component :binary) >> (or *oos-test* >> (handler-case >> (progn >> (funcall (load-function component) binary-pname) >> (setf (component-load-time component) >> (file-write-date binary-pname))) >> (error (condition) >> (format *error-output* >> "MK:DEFSYSTEM: Error loading ~A: ~A~%; Deleting >> binary and recompiling...~%" >> binary-pname condition) >> ;; Catch and report deletion errors without halting recovery >> (handler-case (delete-file binary-pname) >> (error (del-err) >> (format *error-output* >> "MK:DEFSYSTEM: Warning: Could not delete ~A: >> ~A~%; Recompiling anyways...~%" >> binary-pname del-err))) >> ;; Force recompilation >> (compile-file-operation component t) >> ;; Final load attempt >> (funcall (load-function component) binary-pname) >> (setf (component-load-time component) >> (file-write-date binary-pname)))))) >> t) >> >> Works for me. >> >> Should I commit that? > > I’m ok with that. But I’d rather update defsystem to a later > version. See https://gitlab.common-lisp.net/mantoniotti/mk-defsystem. > > I tried this a while ago and it worked with all lisp including gcl > with a minor patch. The latest version, however, doesn’t work. I > filed a bug on that. > > Or, from what I gather, the development version of gcl can > actually run asdf (maybe with a few minor updates to asdf?). That > would be pretty sweet. > >> >> Best regards >> David Scherfgen >> >> Am Mi., 17. Juni 2026 um 07:27 Uhr schrieb Gunter Königsmann >> <[email protected]>: >> >> ...and in the meantime I found out that we don't even require >> a race condition to end up with this problem: >> >> * Tried a load(lapack); on SBCL that ran out of reserved >> heap space => told maxima via command-line argument to >> reserve more heap. Result: A system that would have been >> able to load and compile lapack, but lied about not being >> able to load lapack from /usr/lib when not being able to >> load the partial compilation results from ~/.maxima/binary >> * Deleted ~/.maxima/binary and told a user to continue my >> work. User tried to load lapack, after waiting 2 minutes >> assumed that maxima has crashed, restarted maxima (with >> the correct command-line arguments) and maxima lied about >> having failed loading lapack from /usr/lib while trying >> to load another incomplete compilation result from the >> home dir, instead. >> >> => would another approach be possible? >> >> * execute load() the normal way, and >> * if that failed ignore the contents of ~/.maxima/binary >> and try again? >> >> In the end ~./.maxima/binary is merely a cache - that is easy >> to cause to contain wrong data. >> >> Kind regards, >> >> Gunter. >> >> On 6/17/26 07:13, David Scherfgen wrote: >>> unwind-protect wouldn't help in case of a "hard crash". In >>> theory, the process that currently holds the lock could use >>> a thread to modify the lockfile in regular intervals, >>> resetting its file-write-date, acting as a signal that the >>> process is still alive. If it really crashes, it no longer >>> modifies the file (file-write-date stays the same), and >>> eventually another waiting process would notice, delete the >>> lockfile and enter the lock itself. >>> But that's quite sophisticated already, maybe it all isn't >>> really much of a problem. >>> >>> Gunter K�nigsmann <[email protected]> schrieb am Mi., 17. >>> Juni 2026, 07:02: >>> >>> Would a unwind-protect that removes the lock typically >>> help in case of a crash? If power browns out while >>> saving perhaps it is reasonable to remove the lock after >>> waiting for 3 minutes or so... >>> >>> >>> On 14 June 2026 16:00:19 CEST, David Scherfgen via >>> Maxima-discuss <[email protected]> wrote: >>> >>> Gemini proposed this Common Lisp file locking >>> mechanism (but read below): >>> >>> (defmacro with-file-lock ((lock-file &key >>> (sleep-time 0.1)) &body body) >>> "Acquires an exclusive lock using purely ANSI >>> Common Lisp atomic file creation." >>> (let ((stream-sym (gensym "STREAM"))) >>> `(let ((,stream-sym nil)) >>> (unwind-protect >>> (progn >>> ;; Spinlock: keep trying to create the >>> file until successful >>> (loop >>> (handler-case >>> (progn >>> (setf ,stream-sym (open ,lock-file >>> :direction :output >>> :if-exists :error >>> :if-does-not-exist :create)) >>> (return)) ; Successfully >>> acquired lock; exit the loop >>> (file-error () >>> ;; Lock file exists; wait and retry >>> (sleep ,sleep-time)))) >>> ;; Execute the compilation or protected >>> code >>> ,@body) >>> ;; Cleanup: close the stream and delete the >>> lock file >>> (when ,stream-sym >>> (close ,stream-sym) >>> (ignore-errors (delete-file ,lock-file))))))) >>> >>> I tried it by having multiple processes do this: >>> >>> (with-file-lock (".lockfile") (format t "Entered~%") >>> (sleep 30) (format t "Exited~%")) >>> >>> Seems to work ... >>> There could be a problem, though, when a process >>> crashes / gets killed while it has the lock. Then it >>> won't release the lock, and no process could ever >>> acquire it again. >>> In theory, one could define a "timeout" and say that >>> if the lockfile exists and is too old (check using >>> file-write-date), then the process probably crashed. >>> For a robust solution, one should use e.g. POSIX flock. >>> >>> Best regards >>> David Scherfgen >>> >>> Am So., 14. Juni 2026 um 15:40 Uhr schrieb Raymond >>> Toy <[email protected]>: >>> >>> On 6/14/26 5:05 AM, Gunter Königsmann via >>> Maxima-discuss wrote: >>> >>>> Dear all, >>>> >>>> wxMaxima on every commit creates a new virtual >>>> machine, compiles wxMaxima in that machine and >>>> runs a few hundred tests in that machine. >>>> >>>> If it runs more than one test in parallel >>>> sometimes maxima fails to load draw. >>>> >>>> My theory is: >>>> >>>> * One maxima process tries to load draw, >>>> compiles that package and starts saving it. >>>> * A second maxima process tries to load the >>>> compiled draw package while that compiled >>>> package still being written. >>>> >>>> Could that be the case? >>>> >>> Yes. Look at share/draw/draw.lisp. It runs >>> |mk:oos| to compile all the files from the draw >>> package. >>> ​ >>> _______________________________________________ >>> Maxima-discuss mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/maxima-discuss >>> >> >> >> _______________________________________________ >> Maxima-discuss mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/maxima-discuss > ​ > _______________________________________________ > Maxima-discuss mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/maxima-discuss > ​ _______________________________________________ Maxima-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/maxima-discuss