Re: compiling crashes, compiling and loading slowness
Andrew Wolven <[email protected]> Tue, 14 Apr 2026 15:18:29 -0500
| Newsgroups | gmane.lisp.steel-bank.general |
|---|---|
| Message-ID | <CANhGVvQeoLqGHOgQxDfbbXqFjvaq0_+kS2xL48D5UxAkvAAdMg@mail.gmail.com> |
--===============7533780672409399026== Content-Type: multipart/alternative; boundary="00000000000021075a064f714e4a" --00000000000021075a064f714e4a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Christophe, download the files: https://www.wolvendesignautomation.com/assets/noffi.tgz https://www.wolvendesignautomation.com/assets/big-file-bug.tgz untar them somewhere and follow the instructions in big-file-bug/instructions.txt I used SBCL 2.5.9 for Windows X86-64. On Tue, Apr 14, 2026 at 2:15=E2=80=AFPM Andrew Wolven <[email protected]> w= rote: > Thanks. > > I can just send you the file with the 20000 defmethods, and the file with > the 2500 or so defclasses and the 700 or so defpackages which it > depends on. You will need to load noffi, but we have tested that well on > SBCL. Give me a little bit to get organized and make sure the bug is sti= ll > there. (the "256 is not of type (unsigned-byte 8)") > > And yes, I can do some profiling on the initialize-instance defmethods an= d > subsets thereof. > > As far as the big defun goes, I have some smaller "big defuns" for > cubic/quadratic and quadratic/quadratic. > > On Tue, Apr 14, 2026 at 1:28=E2=80=AFPM Christophe Rhodes <[email protected]= et> > wrote: > >> Andrew Wolven <[email protected]> writes: >> >> > Hi. I'm not ready to file a bug, just have some anecdotes about my >> > use of SBCL. Could spend some time tracking down the issues if no one >> > has suggestions. >> > >> > 1. I have problems with extremely large source files. The workaround >> > is to break them up into smaller pieces, but just to report: When >> > compiling a file of about 20 thousand short defmethods, with generic >> > function method counts usually being less than 5, sbcl often hits a >> > bug "256 is not of type (unsigned-byte 8)" something to do with a hash >> > set. I could replicate it if necessary. >> >> I think it would be useful to have a bug report with a reproducer for >> this. (A program to generate the file with 20000 defmethods, if >> appropriate, would be fine). >> >> > 2. Extremely long function bodies take a really long time to compile. >> > So obviously my problems deal with generating code, the previous being >> > C++ bindings, but I also created a function which computes the >> > polynomial which can be given to a root finder to compute the closed >> > form intersection of two cubic rational bezier curves in 2-space. I >> > made the function body with maxima. Takes an extremely long time to >> > compile with various versions of SBCL. Like 8 minutes. Loads fast. >> >> This is a reasonably known problem: some of SBCL's optimizations scale >> worse-than-linearly in the size of the function. Previous culprits have >> included the type derivation engine, and particularly the repeated >> iteration of constraint propagation, towards a fixed point. Here's an >> example bug report for something similar: >> https://bugs.launchpad.net/sbcl/+bug/2112276 >> >> There are sometimes compiler settings that can be tweaked to discourage >> SBCL from doing some parts of work too often. To find out which might >> be useful in this specific case, it would be good to know which >> particular part of the compiler is the bottleneck. If it's possible to >> shrink the problem a bit, so that the compilation is of the order of a >> minute or two, it should be possible to get a reasonable picture of >> what's going on using the statistical profiler. (It might also work at >> 8 minutes; that's "only" 48000 samples by default). >> >> Other things that can alleviate this include splitting the routine into >> multiple functions. I think we have some specialized entry points >> (e.g. for passing double-floats unboxed) so with tasteful argument lists >> the compiled code might not be substantially slower. >> >> > 3. Initialize-instance, or other CLOS generic functions with large >> > numbers of methods. I generated a file for a fairly large C++ dll to >> > create instances of CLOS wrappers of C++ objects. So you can just >> > call make-instance and get and get the method to automatically call >> > "new" and "constructors" and set up finalizers with destructor and >> > delete. So I have an 80,000 LOC file with a few thousand >> > initialize-instance methods, some just a few lines, others many lines. >> > It's fast compiling but then takes like 10 minutes to load. >> > >> > In another case I compile and load a 250,000 LOC without large numbers >> > of methods and it compiles and loads in 30 seconds. So I think large >> > numbers of methods create a problem. >> >> Defining new methods (when loading defmethod forms) inherently invokes >> parts of CLOS. In an ideal world this would be linear in the work done, >> but it wouldn't surprise me to find that something is scaling >> more-than-linearly. There are definitely things that have to be done >> each time (e.g. running `add-method` on `initialize-instance` -- >> wouldn't it be nice to be able to run one `add-methods` rather than a >> few thousand `add-method` calls?) but maybe some other things can be >> done more lazily. >> >> If you shrink this down to half the size, does it take less than half >> the time? Again, a statistical profile report for what's going on here >> would be interesting. >> >> Christophe >> > --00000000000021075a064f714e4a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Christophe,<div><br></div><div>download the files:</div><d= iv><a href=3D"https://www.wolvendesignautomation.com/assets/noffi.tgz">http= s://www.wolvendesignautomation.com/assets/noffi.tgz</a></div><div><a href= =3D"https://www.wolvendesignautomation.com/assets/big-file-bug.tgz">https:/= /www.wolvendesignautomation.com/assets/big-file-bug.tgz</a></div><div><br><= /div><div>untar them somewhere and follow the instructions in big-file-bug/= instructions.txt</div><div><br></div><div>I used SBCL 2.5.9 for Windows X86= -64.</div></div><br><div class=3D"gmail_quote gmail_quote_container"><div d= ir=3D"ltr" class=3D"gmail_attr">On Tue, Apr 14, 2026 at 2:15=E2=80=AFPM And= rew Wolven <<a href=3D"mailto:[email protected]">[email protected]</a>&g= t; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div d= ir=3D"ltr">Thanks.<div><br></div><div>I can just send you the file with the= 20000 defmethods, and the file with the 2500 or so defclasses and the 700 = or so defpackages=C2=A0which it depends=C2=A0on.=C2=A0 You will need to loa= d noffi, but we have tested that well on SBCL.=C2=A0 Give me a little bit t= o get organized and make sure the bug is still there.=C2=A0 (the "256 = is not of type (unsigned-byte 8)")</div><div><br></div><div>And yes, I= can do some profiling on the initialize-instance defmethods=C2=A0and subse= ts thereof.</div><div><br></div><div>As far as the big defun goes, I have s= ome smaller "big defuns" for cubic/quadratic and quadratic/quadra= tic.</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gm= ail_attr">On Tue, Apr 14, 2026 at 1:28=E2=80=AFPM Christophe Rhodes <<a = href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>>= wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Andrew W= olven <<a href=3D"mailto:[email protected]" target=3D"_blank">awolven@gm= ail.com</a>> writes:<br> <br> > Hi.=C2=A0 I'm not ready to file a bug, just have some anecdotes ab= out my<br> > use of SBCL.=C2=A0 Could spend some time tracking down the issues if n= o one<br> > has suggestions.<br> ><br> > 1. I have problems with extremely large source files.=C2=A0 The workar= ound<br> > is to break them up into smaller pieces, but just to report: When<br> > compiling a file of about 20 thousand short defmethods, with generic<b= r> > function method counts usually being less than 5, sbcl often hits a<br= > > bug "256 is not of type (unsigned-byte 8)" something to do w= ith a hash<br> > set.=C2=A0 I could replicate it if necessary.<br> <br> I think it would be useful to have a bug report with a reproducer for<br> this.=C2=A0 (A program to generate the file with 20000 defmethods, if<br> appropriate, would be fine).<br> <br> > 2. Extremely long function bodies take a really long time to compile.<= br> > So obviously my problems deal with generating code, the previous being= <br> > C++ bindings, but I also created a function which computes the<br> > polynomial which can be given to a root finder to compute the closed<b= r> > form intersection of two cubic rational bezier curves in 2-space.=C2= =A0 I<br> > made the function body with maxima.=C2=A0 Takes an extremely long time= to<br> > compile with various versions of SBCL.=C2=A0 Like 8 minutes.=C2=A0 Loa= ds fast.<br> <br> This is a reasonably known problem: some of SBCL's optimizations scale<= br> worse-than-linearly in the size of the function.=C2=A0 Previous culprits ha= ve<br> included the type derivation engine, and particularly the repeated<br> iteration of constraint propagation, towards a fixed point.=C2=A0 Here'= s an<br> example bug report for something similar:<br> <a href=3D"https://bugs.launchpad.net/sbcl/+bug/2112276" rel=3D"noreferrer"= target=3D"_blank">https://bugs.launchpad.net/sbcl/+bug/2112276</a><br> <br> There are sometimes compiler settings that can be tweaked to discourage<br> SBCL from doing some parts of work too often.=C2=A0 To find out which might= <br> be useful in this specific case, it would be good to know which<br> particular part of the compiler is the bottleneck.=C2=A0 If it's possib= le to<br> shrink the problem a bit, so that the compilation is of the order of a<br> minute or two, it should be possible to get a reasonable picture of<br> what's going on using the statistical profiler.=C2=A0 (It might also wo= rk at<br> 8 minutes; that's "only" 48000 samples by default).<br> <br> Other things that can alleviate this include splitting the routine into<br> multiple functions.=C2=A0 I think we have some specialized entry points<br> (e.g. for passing double-floats unboxed) so with tasteful argument lists<br= > the compiled code might not be substantially slower.<br> <br> > 3. Initialize-instance, or other CLOS generic functions with large<br> > numbers of methods.=C2=A0 I generated a file for a fairly large C++ dl= l to<br> > create instances of CLOS wrappers of C++ objects.=C2=A0 So you can jus= t<br> > call make-instance and get and get the method to automatically call<br= > > "new" and "constructors" and set up finalizers wit= h destructor and<br> > delete.=C2=A0 So I have an 80,000 LOC file with a few thousand<br> > initialize-instance methods, some just a few lines, others many lines.= <br> > It's fast compiling but then takes like 10 minutes to load.<br> ><br> > In another case I compile and load a 250,000 LOC without large numbers= <br> > of methods and it compiles and loads in 30 seconds.=C2=A0 So I think l= arge<br> > numbers of methods create a problem.<br> <br> Defining new methods (when loading defmethod forms) inherently invokes<br> parts of CLOS.=C2=A0 In an ideal world this would be linear in the work don= e,<br> but it wouldn't surprise me to find that something is scaling<br> more-than-linearly.=C2=A0 There are definitely things that have to be done<= br> each time (e.g. running `add-method` on `initialize-instance` --<br> wouldn't it be nice to be able to run one `add-methods` rather than a<b= r> few thousand `add-method` calls?) but maybe some other things can be<br> done more lazily.<br> <br> If you shrink this down to half the size, does it take less than half<br> the time?=C2=A0 Again, a statistical profile report for what's going on= here<br> would be interesting.=C2=A0 <br> <br> Christophe<br> </blockquote></div> </blockquote></div> --00000000000021075a064f714e4a-- --===============7533780672409399026== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============7533780672409399026== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sbcl-help mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sbcl-help --===============7533780672409399026==--