Maxima searching all "binary" directories: conflicts & performance degradation

Dear all,

I'm revisiting Maxima's file searching and loading behavior, and I’ve
identified a problematic issue with how we handle (or: *don't* handle) the
binary directory.

Currently, when searching for a file, Maxima looks through directories in a
fixed order (simplified here):

   1. The maxima directory in the user's home directory (+ recursive
   subdirectories)
   2. Maxima's installation directory (+ recursive subdirectories)

The issue originates in the user's home maxima directory. Because it is
searched first, Maxima also recursively searches its binary subdirectory,
which contains all the compiled .fasl files.

As we know, for each new version of Maxima and each Lisp interpreter, a
distinct subdirectory is created to avoid conflicts (e.g.,
binary/5_47post/sbcl/2_6_3/share/draw). However, over time, these old
directories accumulate. On my machine alone, I have 63 subdirectories in
there.

Because Maxima searches *all* of them recursively every time a file is
requested — and does so multiple times for different extensions (.mac, .wxm,
.fasl, .lisp) — it causes two major issues:

*1. Fatal Loading Errors (Conflicts)*

When Maxima finds a requested file in multiple directories, it sorts them
alphabetically and returns the first one. This is highly problematic
because it can force Maxima to load a .fasl file compiled by an older,
incompatible Lisp version.

Here is a real example from my system running SBCL 2.6.3, where it fatally
crashes after trying to load a file from SBCL 2.5.3:

Maxima 5.47post https://maxima.sourceforge.io
using Lisp SBCL 2.6.3
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
The function bug_report() provides bug reporting information.
(%i1) load(gnuplot);
Warning: More than one file matches.  Selecting the first file from:
  C:/Users/David/maxima/binary/5_47post/sbcl/2_5_3/share/draw/gnuplot.fasl
  C:/Users/David/maxima/binary/5_47post/sbcl/2_5_4/share/draw/gnuplot.fasl
  C:/Users/David/maxima/binary/5_47post/sbcl/2_6_3/share/draw/gnuplot.fasl

;
; compilation unit aborted
;   caught 1 fatal ERROR condition
loadfile: failed to load
C:/Users/David/maxima/binary/5_47post/sbcl/2_5_3/share/draw/gnuplot.fasl
 -- an error. To debug this try: debugmode(true);

*2. Slow Performance Degradation*

As users upgrade Maxima and new directories are created, the search space
continuously grows, slowly but consistently degrading search performance
over time.
Proposed fixesTo fix this, I propose two adjustments:

   -


*Restrict the search path: *Exclude any subdirectory of the binary
   directory *except* for the specific path that matches the user's current
   Maxima and Lisp version.

   *Note: *Unfortunately, this alone won't fully prevent performance
   degradation, since all the binary subdirectories would still be
   enumerated by the directory function (which can be slow), but at least
   we could avoid actually checking them for the searched file.

   *Alternative: *Put the binary (and other "internal") files into a
   different folder inside the user's home directory, e.g. maxima-internal,
   and only search inside the subdirectory that matches the user's current
   Maxima and Lisp version. This would completely fix the issue, but I don't
   know how people feel about adding another "top-level" directory.

   -


*Add an OS distinction layer (optional but recommended): *With
   virtualization, emulation and shared network drives being common, we should
   consider adding the OS to the path structure. For example:
   binary/5_47post/sbcl/2_6_3/windows and binary/5_47post/sbcl/2_6_3/linux.
   This would prevent cross-platform interference.

I’d love to hear your thoughts on this approach before I start putting
together a patch.

Best regards,
David Scherfgen

_______________________________________________
Maxima-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/maxima-discuss