Re: compiling crashes, compiling and loading slowness
Andrew Wolven <[email protected]> Tue, 14 Apr 2026 14:15:31 -0500
| Newsgroups | gmane.lisp.steel-bank.general |
|---|---|
| Message-ID | <CANhGVvT=sBxmp81AZ6ov6=+__r6+1yRHecPd8HmZX_qPJt+i2Q@mail.gmail.com> |
--===============4478884171065925970== Content-Type: multipart/alternative; boundary="000000000000e6fd4c064f706c8e" --000000000000e6fd4c064f706c8e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks. I can just send you the file with the 20000 defmethods, and the file with the 2500 or so defclasses and the 700 or so defpackages which it depends on. You will need to load noffi, but we have tested that well on SBCL. Give me a little bit to get organized and make sure the bug is still there. (the "256 is not of type (unsigned-byte 8)") And yes, I can do some profiling on the initialize-instance defmethods and subsets thereof. As far as the big defun goes, I have some smaller "big defuns" for cubic/quadratic and quadratic/quadratic. On Tue, Apr 14, 2026 at 1:28=E2=80=AFPM Christophe Rhodes <[email protected]= > wrote: > Andrew Wolven <[email protected]> writes: > > > Hi. I'm not ready to file a bug, just have some anecdotes about my > > use of SBCL. Could spend some time tracking down the issues if no one > > has suggestions. > > > > 1. I have problems with extremely large source files. The workaround > > is to break them up into smaller pieces, but just to report: When > > compiling a file of about 20 thousand short defmethods, with generic > > function method counts usually being less than 5, sbcl often hits a > > bug "256 is not of type (unsigned-byte 8)" something to do with a hash > > set. I could replicate it if necessary. > > I think it would be useful to have a bug report with a reproducer for > this. (A program to generate the file with 20000 defmethods, if > appropriate, would be fine). > > > 2. Extremely long function bodies take a really long time to compile. > > So obviously my problems deal with generating code, the previous being > > C++ bindings, but I also created a function which computes the > > polynomial which can be given to a root finder to compute the closed > > form intersection of two cubic rational bezier curves in 2-space. I > > made the function body with maxima. Takes an extremely long time to > > compile with various versions of SBCL. Like 8 minutes. Loads fast. > > This is a reasonably known problem: some of SBCL's optimizations scale > worse-than-linearly in the size of the function. Previous culprits have > included the type derivation engine, and particularly the repeated > iteration of constraint propagation, towards a fixed point. Here's an > example bug report for something similar: > https://bugs.launchpad.net/sbcl/+bug/2112276 > > There are sometimes compiler settings that can be tweaked to discourage > SBCL from doing some parts of work too often. To find out which might > be useful in this specific case, it would be good to know which > particular part of the compiler is the bottleneck. If it's possible to > shrink the problem a bit, so that the compilation is of the order of a > minute or two, it should be possible to get a reasonable picture of > what's going on using the statistical profiler. (It might also work at > 8 minutes; that's "only" 48000 samples by default). > > Other things that can alleviate this include splitting the routine into > multiple functions. I think we have some specialized entry points > (e.g. for passing double-floats unboxed) so with tasteful argument lists > the compiled code might not be substantially slower. > > > 3. Initialize-instance, or other CLOS generic functions with large > > numbers of methods. I generated a file for a fairly large C++ dll to > > create instances of CLOS wrappers of C++ objects. So you can just > > call make-instance and get and get the method to automatically call > > "new" and "constructors" and set up finalizers with destructor and > > delete. So I have an 80,000 LOC file with a few thousand > > initialize-instance methods, some just a few lines, others many lines. > > It's fast compiling but then takes like 10 minutes to load. > > > > In another case I compile and load a 250,000 LOC without large numbers > > of methods and it compiles and loads in 30 seconds. So I think large > > numbers of methods create a problem. > > Defining new methods (when loading defmethod forms) inherently invokes > parts of CLOS. In an ideal world this would be linear in the work done, > but it wouldn't surprise me to find that something is scaling > more-than-linearly. There are definitely things that have to be done > each time (e.g. running `add-method` on `initialize-instance` -- > wouldn't it be nice to be able to run one `add-methods` rather than a > few thousand `add-method` calls?) but maybe some other things can be > done more lazily. > > If you shrink this down to half the size, does it take less than half > the time? Again, a statistical profile report for what's going on here > would be interesting. > > Christophe > --000000000000e6fd4c064f706c8e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Thanks.<div><br></div><div>I can just send you the file wi= th the 20000 defmethods, and the file with the 2500 or so defclasses and th= e 700 or so defpackages=C2=A0which it depends=C2=A0on.=C2=A0 You will need = to load noffi, but we have tested that well on SBCL.=C2=A0 Give me a little= bit to get organized and make sure the bug is still there.=C2=A0 (the &quo= t;256 is not of type (unsigned-byte 8)")</div><div><br></div><div>And = yes, I can do some profiling on the initialize-instance defmethods=C2=A0and= subsets thereof.</div><div><br></div><div>As far as the big defun goes, I = have some smaller "big defuns" for cubic/quadratic and quadratic/= quadratic.</div></div><br><div class=3D"gmail_quote gmail_quote_container">= <div dir=3D"ltr" class=3D"gmail_attr">On Tue, Apr 14, 2026 at 1:28=E2=80=AF= PM Christophe Rhodes <<a href=3D"mailto:[email protected]">[email protected]= et</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margi= n:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex= ">Andrew Wolven <<a href=3D"mailto:[email protected]" target=3D"_blank">= [email protected]</a>> writes:<br> <br> > Hi.=C2=A0 I'm not ready to file a bug, just have some anecdotes ab= out my<br> > use of SBCL.=C2=A0 Could spend some time tracking down the issues if n= o one<br> > has suggestions.<br> ><br> > 1. I have problems with extremely large source files.=C2=A0 The workar= ound<br> > is to break them up into smaller pieces, but just to report: When<br> > compiling a file of about 20 thousand short defmethods, with generic<b= r> > function method counts usually being less than 5, sbcl often hits a<br= > > bug "256 is not of type (unsigned-byte 8)" something to do w= ith a hash<br> > set.=C2=A0 I could replicate it if necessary.<br> <br> I think it would be useful to have a bug report with a reproducer for<br> this.=C2=A0 (A program to generate the file with 20000 defmethods, if<br> appropriate, would be fine).<br> <br> > 2. Extremely long function bodies take a really long time to compile.<= br> > So obviously my problems deal with generating code, the previous being= <br> > C++ bindings, but I also created a function which computes the<br> > polynomial which can be given to a root finder to compute the closed<b= r> > form intersection of two cubic rational bezier curves in 2-space.=C2= =A0 I<br> > made the function body with maxima.=C2=A0 Takes an extremely long time= to<br> > compile with various versions of SBCL.=C2=A0 Like 8 minutes.=C2=A0 Loa= ds fast.<br> <br> This is a reasonably known problem: some of SBCL's optimizations scale<= br> worse-than-linearly in the size of the function.=C2=A0 Previous culprits ha= ve<br> included the type derivation engine, and particularly the repeated<br> iteration of constraint propagation, towards a fixed point.=C2=A0 Here'= s an<br> example bug report for something similar:<br> <a href=3D"https://bugs.launchpad.net/sbcl/+bug/2112276" rel=3D"noreferrer"= target=3D"_blank">https://bugs.launchpad.net/sbcl/+bug/2112276</a><br> <br> There are sometimes compiler settings that can be tweaked to discourage<br> SBCL from doing some parts of work too often.=C2=A0 To find out which might= <br> be useful in this specific case, it would be good to know which<br> particular part of the compiler is the bottleneck.=C2=A0 If it's possib= le to<br> shrink the problem a bit, so that the compilation is of the order of a<br> minute or two, it should be possible to get a reasonable picture of<br> what's going on using the statistical profiler.=C2=A0 (It might also wo= rk at<br> 8 minutes; that's "only" 48000 samples by default).<br> <br> Other things that can alleviate this include splitting the routine into<br> multiple functions.=C2=A0 I think we have some specialized entry points<br> (e.g. for passing double-floats unboxed) so with tasteful argument lists<br= > the compiled code might not be substantially slower.<br> <br> > 3. Initialize-instance, or other CLOS generic functions with large<br> > numbers of methods.=C2=A0 I generated a file for a fairly large C++ dl= l to<br> > create instances of CLOS wrappers of C++ objects.=C2=A0 So you can jus= t<br> > call make-instance and get and get the method to automatically call<br= > > "new" and "constructors" and set up finalizers wit= h destructor and<br> > delete.=C2=A0 So I have an 80,000 LOC file with a few thousand<br> > initialize-instance methods, some just a few lines, others many lines.= <br> > It's fast compiling but then takes like 10 minutes to load.<br> ><br> > In another case I compile and load a 250,000 LOC without large numbers= <br> > of methods and it compiles and loads in 30 seconds.=C2=A0 So I think l= arge<br> > numbers of methods create a problem.<br> <br> Defining new methods (when loading defmethod forms) inherently invokes<br> parts of CLOS.=C2=A0 In an ideal world this would be linear in the work don= e,<br> but it wouldn't surprise me to find that something is scaling<br> more-than-linearly.=C2=A0 There are definitely things that have to be done<= br> each time (e.g. running `add-method` on `initialize-instance` --<br> wouldn't it be nice to be able to run one `add-methods` rather than a<b= r> few thousand `add-method` calls?) but maybe some other things can be<br> done more lazily.<br> <br> If you shrink this down to half the size, does it take less than half<br> the time?=C2=A0 Again, a statistical profile report for what's going on= here<br> would be interesting.=C2=A0 <br> <br> Christophe<br> </blockquote></div> --000000000000e6fd4c064f706c8e-- --===============4478884171065925970== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============4478884171065925970== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sbcl-help mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sbcl-help --===============4478884171065925970==--