Re: Race condition on loading packages from 2 maxima processes in parallel?
Raymond Toy <[email protected]> Wed, 17 Jun 2026 14:23:05 -0700
| Newsgroups | gmane.comp.mathematics.maxima.general |
|---|---|
| Message-ID | <[email protected]> |
On 6/17/26 1:29 PM, David Scherfgen wrote: > I think the objection was "Why risk changing something that works?". > And actually, that's true. Is there something broken with the old Yeah, but then, for many people, they’d still be using Maxima 5.9 because it still works. > version we're using, except for the compiler warnings? > I'll commit the patch if that's OK. It's simple enough to reapply it if/ No objections. > when we update defsystem. I'll also try to contact the developer and > propose to integrate it. > > Am Mi., 17. Juni 2026 um 21:59 Uhr schrieb Raymond Toy > <[email protected]>: > > On 6/17/26 6:43 AM, David Scherfgen wrote: > >> I think I proposed updating defsystem some time ago (the main >> reason being that I got annoyed by the tons of compiler >> warnings), but there were objections (I don't remember by whom). >> Anyways, I have looked at the newest version, and it has the same >> problem, as far as I can tell. It doesn't try to handle errors >> occurring while loading a binary. > Yes, that’s true. Don’t know recall what the objections were > either. But certainly I’d prefer modifying a more up-to-date > version than the ancient version we have. > > I don’t know how to resolve this issue. Maybe just wait until the > next release of gcl which will hopefully support asdf and drop > support of defsystem? We’d have to update some things that use > defsystem to build, but that should be straightforward. > >> >> Am Mi., 17. Juni 2026 um 15:37 Uhr schrieb Raymond Toy >> <[email protected]>: >> >> On 6/17/26 2:30 AM, David Scherfgen via Maxima-discuss wrote: >> >>> LAPACK uses mk-defsystem (we ship that with Maxima, see >>> lisp-utils/defsystem.lisp) to compile itself. For each >>> source file belonging to the LAPACK system, mk-defsystem >>> checks whether the binary (.fasl) file already exists. If >>> yes, it loads it. If not, it compiles the source and creates >>> the binary file. >>> We can patch it so that if the binary file exists, but >>> cannot be loaded, it tries to compile it again (only one >>> single time, to prevent endless loops). >>> >>> In lisp-utils/defsystem.lisp, replace >>> >>> ((and binary-exists load-binary) >>> (with-tell-user ("Loading binary" component :binary) >>> (or *oos-test* >>> (progn >>> (funcall (load-function component) binary-pname) >>> (setf (component-load-time component) >>> (file-write-date binary-pname))))) >>> t) >>> >>> with >>> >>> ((and binary-exists load-binary) >>> (with-tell-user ("Loading binary" component :binary) >>> (or *oos-test* >>> (handler-case >>> (progn >>> (funcall (load-function component) binary-pname) >>> (setf (component-load-time component) >>> (file-write-date binary-pname))) >>> (error (condition) >>> (format *error-output* >>> "MK:DEFSYSTEM: Error loading ~A: ~A~%; >>> Deleting binary and recompiling...~%" >>> binary-pname condition) >>> ;; Catch and report deletion errors without halting >>> recovery >>> (handler-case (delete-file binary-pname) >>> (error (del-err) >>> (format *error-output* >>> "MK:DEFSYSTEM: Warning: Could not delete >>> ~A: ~A~%; Recompiling anyways...~%" >>> binary-pname del-err))) >>> ;; Force recompilation >>> (compile-file-operation component t) >>> ;; Final load attempt >>> (funcall (load-function component) binary-pname) >>> (setf (component-load-time component) >>> (file-write-date binary-pname)))))) >>> t) >>> >>> Works for me. >>> >>> Should I commit that? >> >> I’m ok with that. But I’d rather update defsystem to a later >> version. See >> https://gitlab.common-lisp.net/mantoniotti/mk-defsystem. >> >> I tried this a while ago and it worked with all lisp >> including gcl with a minor patch. The latest version, >> however, doesn’t work. I filed a bug on that. >> >> Or, from what I gather, the development version of gcl can >> actually run asdf (maybe with a few minor updates to asdf?). >> That would be pretty sweet. >> >>> >>> Best regards >>> David Scherfgen >>> >>> Am Mi., 17. Juni 2026 um 07:27 Uhr schrieb Gunter Königsmann >>> <[email protected]>: >>> >>> ...and in the meantime I found out that we don't even >>> require a race condition to end up with this problem: >>> >>> * Tried a load(lapack); on SBCL that ran out of >>> reserved heap space => told maxima via command-line >>> argument to reserve more heap. Result: A system that >>> would have been able to load and compile lapack, but >>> lied about not being able to load lapack from >>> /usr/lib when not being able to load the partial >>> compilation results from ~/.maxima/binary >>> * Deleted ~/.maxima/binary and told a user to continue >>> my work. User tried to load lapack, after waiting 2 >>> minutes assumed that maxima has crashed, restarted >>> maxima (with the correct command-line arguments) and >>> maxima lied about having failed loading lapack from >>> /usr/lib while trying to load another incomplete >>> compilation result from the home dir, instead. >>> >>> => would another approach be possible? >>> >>> * execute load() the normal way, and >>> * if that failed ignore the contents of >>> ~/.maxima/binary and try again? >>> >>> In the end ~./.maxima/binary is merely a cache - that is >>> easy to cause to contain wrong data. >>> >>> Kind regards, >>> >>> Gunter. >>> >>> On 6/17/26 07:13, David Scherfgen wrote: >>>> unwind-protect wouldn't help in case of a "hard crash". >>>> In theory, the process that currently holds the lock >>>> could use a thread to modify the lockfile in regular >>>> intervals, resetting its file-write-date, acting as a >>>> signal that the process is still alive. If it really >>>> crashes, it no longer modifies the file >>>> (file-write-date stays the same), and eventually >>>> another waiting process would notice, delete the >>>> lockfile and enter the lock itself. >>>> But that's quite sophisticated already, maybe it all >>>> isn't really much of a problem. >>>> >>>> Gunter K�nigsmann <[email protected]> schrieb am Mi., >>>> 17. Juni 2026, 07:02: >>>> >>>> Would a unwind-protect that removes the lock >>>> typically help in case of a crash? If power browns >>>> out while saving perhaps it is reasonable to remove >>>> the lock after waiting for 3 minutes or so... >>>> >>>> >>>> On 14 June 2026 16:00:19 CEST, David Scherfgen via >>>> Maxima-discuss >>>> <[email protected]> wrote: >>>> >>>> Gemini proposed this Common Lisp file locking >>>> mechanism (but read below): >>>> >>>> (defmacro with-file-lock ((lock-file &key >>>> (sleep-time 0.1)) &body body) >>>> "Acquires an exclusive lock using purely ANSI >>>> Common Lisp atomic file creation." >>>> (let ((stream-sym (gensym "STREAM"))) >>>> `(let ((,stream-sym nil)) >>>> (unwind-protect >>>> (progn >>>> ;; Spinlock: keep trying to create >>>> the file until successful >>>> (loop >>>> (handler-case >>>> (progn >>>> (setf ,stream-sym (open ,lock-file >>>> :direction :output >>>> :if-exists :error >>>> :if-does-not-exist :create)) >>>> (return)) ; Successfully acquired lock; exit >>>> the loop >>>> (file-error () >>>> ;; Lock file exists; wait and retry >>>> (sleep ,sleep-time)))) >>>> ;; Execute the compilation or >>>> protected code >>>> ,@body) >>>> ;; Cleanup: close the stream and >>>> delete the lock file >>>> (when ,stream-sym >>>> (close ,stream-sym) >>>> (ignore-errors (delete-file ,lock-file))))))) >>>> >>>> I tried it by having multiple processes do this: >>>> >>>> (with-file-lock (".lockfile") (format t >>>> "Entered~%") (sleep 30) (format t "Exited~%")) >>>> >>>> Seems to work ... >>>> There could be a problem, though, when a >>>> process crashes / gets killed while it has the >>>> lock. Then it won't release the lock, and no >>>> process could ever acquire it again. >>>> In theory, one could define a "timeout" and say >>>> that if the lockfile exists and is too old >>>> (check using file-write-date), then the process >>>> probably crashed. >>>> For a robust solution, one should use e.g. >>>> POSIX flock. >>>> >>>> Best regards >>>> David Scherfgen >>>> >>>> Am So., 14. Juni 2026 um 15:40 Uhr schrieb >>>> Raymond Toy <[email protected]>: >>>> >>>> On 6/14/26 5:05 AM, Gunter Königsmann via >>>> Maxima-discuss wrote: >>>> >>>>> Dear all, >>>>> >>>>> wxMaxima on every commit creates a new >>>>> virtual machine, compiles wxMaxima in that >>>>> machine and runs a few hundred tests in >>>>> that machine. >>>>> >>>>> If it runs more than one test in parallel >>>>> sometimes maxima fails to load draw. >>>>> >>>>> My theory is: >>>>> >>>>> * One maxima process tries to load draw, >>>>> compiles that package and starts >>>>> saving it. >>>>> * A second maxima process tries to load >>>>> the compiled draw package while that >>>>> compiled package still being written. >>>>> >>>>> Could that be the case? >>>>> >>>> Yes. Look at share/draw/draw.lisp. It runs >>>> |mk:oos| to compile all the files from the >>>> draw package. >>>> ​ >>>> _______________________________________________ >>>> Maxima-discuss mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/maxima-discuss >>>> >>> >>> >>> _______________________________________________ >>> Maxima-discuss mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/maxima-discuss >> ​ >> _______________________________________________ >> Maxima-discuss mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/maxima-discuss >> > ​ > ​ _______________________________________________ Maxima-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/maxima-discuss