Re: folder naming conventions and installed version identifier, was: Maxima searching all "binary" directories: conflicts & performance degradation
Raymond Toy <[email protected]>
| Newsgroups | gmane.comp.mathematics.maxima.general |
|---|---|
| Message-ID | <[email protected]> |
On 4/19/26 10:09 PM, David Scherfgen wrote: > Good point. Should all folders whose names begin with a dot be ignored > in search, or only typically ones such as .git, .svn, .hg, .bzr, .vs, > .vscode, .idea, .xcode? Maybe introduce a list like > $file_search_ignore that is pre-populated to said folder names using > wildcard paths? That’s a reasonable choice, but I haven’t really given a lot of thought about that. It was just something I observed with all the debugging prints enabled. At the very least, I think it would be good if the cache didn’t include these things. Ideally, I think we wouldn’t even want to waste time descending into these directories, but I don’t see how that can be done unless we handle “**” wildcards ourselves. Don’t think that would be hard, and we’d still make |cl:directory| do all the hard work, but perhaps it would be good enough if the cache didn’t have stuff that we don’t want. > > Best regards > David Scherfgen > > Raymond Toy <[email protected]> schrieb am So., 19. Apr. 2026, 22:19: > > On 4/19/26 2:58 AM, David Scherfgen wrote: > >> I have just pushed a minimally-invasive fix for the problem. >> Files that were found within the binary directory, but not within >> the subdirectories corresponding to the current combination of >> Maxima and Lisp versions, are ignored. >> The same goes for files within the "test_directory_cached" >> directory that's created during the cache initialization (for >> checking whether Lisp and OS behave as expected). >> I kindly ask you to verify that Maxima still finds files it's >> expected to find. I tested lots of cases on my machine, and it >> worked perfectly. > > Thanks for the updates. Works fine, but then I wasn’t having the > reported bug issues either. > > In my .maxima dir, I have a few git repos checked out. With the > debug variables enabled, I see that it looks at all the files in > the .git directory. We should probably avoid that; there’s nothing > there that we really want. > >> >> Best regards >> David Scherfgen >> >> Am So., 19. Apr. 2026 um 07:43 Uhr schrieb David Scherfgen >> <[email protected]>: >> >> Forgot to reply to mailing list ... >> / >> / >> / >> > Raymond Toy <[email protected]>:/ >> /> FWIW, I have some fasls there. Not many, but not zero. But >> as a dev, I know how to add *.fasl to the search path./ >> >> Good to know. Then we should definitely keep supporting .fasl >> files in the user directory, in my opinion. >> >> /> Raymond Toy <[email protected]>:/ >> /> An alternative is on startup to search ~/.maxima and >> explicitly add all the directories there to the search path >> [...]/ >> >> The problem with that is of course that new directories >> created while Maxima is running wouldn't be searched - not so >> nice. >> >> At this point, I think the best way forward is to implement >> an exclusion logic because even if we add a new >> .maxima-binary top-level directory, users who upgrade their >> Maxima would still have the old directory structure, and >> old/incompatible .fasl files would be loaded from there when >> they're not found in the new .maxima-binary directory. >> >> I'll try to come up with a minimally messy solution. >> >> Maybe we can print a warning when the number of different >> version-specific folders exceeds a certain threshold, e.g. >> 10, so that users are made aware of potential >> space/performance problems? >> >> Best regards >> David Scherfgen >> >> Raymond Toy <[email protected]> schrieb am So., 19. Apr. >> 2026, 01:24: >> >> On 4/18/26 10:40 AM, David Scherfgen via Maxima-discuss >> wrote: >> >>> Hi Robert, >>> >>> Thanks for the feedback. >>> >>> I completely agree regarding the 5.nnnpost version >>> identifiers — that’s an excellent point, and it >>> absolutely complicates both bug reporting and the file >>> search logic. However, I have practically no knowledge >>> of that part of the Maxima codebase (like configure.ac >>> <http://configure.ac> or the build scripts), so it would >>> be much better if someone else could tackle that >>> specific issue. >> I can help with that, but I’m confused on what the >> proposal is. My |~/.maxima/binary| has things like >> 5_47post and branch_5_46_base_475_g7550340ea. So the >> proposal is that 5_47post should include more of the git >> version? >>> >>> Regarding the directory structure for the binary files, >>> your feedback brings up two technical hurdles we need to >>> consider: >>> >>> *1. Do users actually need $USER_DIR/**/*.fasl?* >>> >>> Currently, $file_search_lisp includes the wildcard entry >>> $USER_DIR/**/*.fasl (where $USER_DIR is the user's >>> Maxima user directory and .fasl can be different for >>> other Lisps). My question for you and the list is: do >>> users really expect to be able to drop their own >>> compiled Lisp files (.fasl) /anywhere/ in their user >>> directory (not inside binary) and have Maxima find them? >> FWIW, I have some fasls there. Not many, but not zero. >> But as a dev, I know how to add *.fasl to the search path. >>> >>> If we can agree that the answer is "no, they shouldn't," >>> then this whole problem becomes incredibly simple to >>> solve. We can just remove that wildcard and only add the >>> correct binary directory to the search path. >>> >>> If the answer is "yes, they do expect that," then >>> keeping the binary folder inside the top-level user >>> maxima directory becomes tricky. That wildcard will >>> inevitably sweep through and find all the old, >>> incompatible .fasl files that we are explicitly trying >>> to avoid. We would have to write explicit exclusion >>> logic: /If a matched file is located somewhere within >>> the binary folder, explicitly check that its path >>> matches the current version/Lisp combination; otherwise, >>> reject it./ This adds messy logic, which was my primary >>> motivation for thinking about a separate folder like >>> maxima-internal. >> An alternative is on startup to search |~/.maxima| and >> explicitly add all the directories there to the search >> path, excluding |~/.maxima/binary/|. We’d add >> |~/.maxima/binary/version/| separately. So we’d have >> |~/.maxima/dir1/**/|, |~/.maxima/dir2/**/| and so on. >> >> You had also previously mentioned that we search >> directories multiple times once each for |*.lisp|, >> |*.fasl|, |*.mac|, |*.wxm|. To reduce this, we could kind >> of go back to the old scheme, but maybe make each entry >> be something like |[".maxima/**", ["mac, "wxm","lisp", >> "fasl"]]|. Then, we’ll do a directory with >> |".maxima/**/*.*"| and then for each entry see if the >> extension (type) matches one in the list. This is kind of >> how the original file search paths were done. I didn’t do >> this because directory searches seemed pretty fast for >> all the Lisps I tested on. >> >>> >>> *2. The OS Distinction & Flattening the Directory* >>> * >>> *Regarding baking in the OS name: I understand the >>> concern, but adding the OS is essentially a one-line >>> code change that shouldn't introduce new points of >>> failure. A simple distinction between Windows, UNIX and >>> unknown can be achieved by looking at common *feature* >>> members. >>> >>> To address the concern about messy folders and deep >>> nesting, what if we flattened the structure instead of >>> nesting it? Rather than having >>> binary/5_47post/sbcl/2_6_3/win32, we could merge the >>> identifiers into a single directory name: >>> binary-5_47post-sbcl-2.6.3-windows. This keeps the >>> directory tree shallow, completely prevents >>> cross-platform/version interference, and requires very >>> little effort to implement. >>> >>> As a related side note, I actually suspect that the bug >>> Wolfgang recently ran into >>> <https://sourceforge.net/p/maxima/bugs/4713/> may be >>> related to his cross-OS workflow. It is just a suspicion >>> as of now — we haven't found the root of the issue yet — >>> but it does highlight why having a built-in OS >>> separation might actually save us from tricky edge-case >>> bugs. >>> >>> Let me know what you think about the user expectation >>> regarding .fasl files — that seems to be the biggest >>> technical blocker to keeping the binary files inside the >>> standard user directory. >>> >>> Best regards >>> David >>> >>> Am Sa., 18. Apr. 2026 um 18:54 Uhr schrieb Robert Dodier >>> <[email protected]>: >>> >>> David, thanks for working on this stuff. I agree >>> it's a good idea to >>> solve it somehow. >>> >>> About searching only the binary folder for the >>> current version -- agreed. >>> >>> About creating another top-level folder -- I'm >>> against it, I think the >>> binary folder should go inside the top-level >>> per-user maxima folder >>> (same as it is now). >>> >>> About baking the OS name into the binary path -- I'm >>> against it, I >>> don't think it's common enough to justify the >>> effort, which is mostly >>> going to be trying to fix bugs after the fact when >>> users report it's >>> not working for them. >>> >>> It's a mess to have multiple folders for different >>> combinations of >>> Maxima + Lisp versions, but I don't think Maxima >>> should try to clean >>> that up; I think we should leave it for the users to >>> deal with. I >>> expect the vast majority of users are going to have >>> one Maxima + Lisp >>> combination for each installed version -- I think >>> that's tolerable. >>> >>> You didn't mention it, but I've come to the >>> conclusion that installed >>> versions of Maxima should only identify themselves via >>> version-specific identifiers, as it is now with >>> uninstalled versions >>> (e.g. branch_5_49_base_392_g85c1954 as it is right >>> now); identifying >>> itself as "5.nnnpost", e.g. 5.49post, makes it >>> impossible to >>> distinguish different installed versions. That is a >>> serious >>> consequence for bug reporting, and also for the file >>> search problem. I >>> guess we would have to modify configure.ac >>> <http://configure.ac> or something to fix that. >>> >>> All the best, >>> >>> Robert >>> >>> >>> >>> _______________________________________________ >>> Maxima-discuss mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/maxima-discuss >> ​ >> _______________________________________________ >> Maxima-discuss mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/maxima-discuss >> > ​ > ​ _______________________________________________ Maxima-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/maxima-discuss