Re: compiling crashes, compiling and loading slowness

Andrew Wolven <[email protected]> writes:

> Hi.  I'm not ready to file a bug, just have some anecdotes about my
> use of SBCL.  Could spend some time tracking down the issues if no one
> has suggestions.
>
> 1. I have problems with extremely large source files.  The workaround
> is to break them up into smaller pieces, but just to report: When
> compiling a file of about 20 thousand short defmethods, with generic
> function method counts usually being less than 5, sbcl often hits a
> bug "256 is not of type (unsigned-byte 8)" something to do with a hash
> set.  I could replicate it if necessary.

I think it would be useful to have a bug report with a reproducer for
this.  (A program to generate the file with 20000 defmethods, if
appropriate, would be fine).

> 2. Extremely long function bodies take a really long time to compile.
> So obviously my problems deal with generating code, the previous being
> C++ bindings, but I also created a function which computes the
> polynomial which can be given to a root finder to compute the closed
> form intersection of two cubic rational bezier curves in 2-space.  I
> made the function body with maxima.  Takes an extremely long time to
> compile with various versions of SBCL.  Like 8 minutes.  Loads fast.

This is a reasonably known problem: some of SBCL's optimizations scale
worse-than-linearly in the size of the function.  Previous culprits have
included the type derivation engine, and particularly the repeated
iteration of constraint propagation, towards a fixed point.  Here's an
example bug report for something similar:
https://bugs.launchpad.net/sbcl/+bug/2112276

There are sometimes compiler settings that can be tweaked to discourage
SBCL from doing some parts of work too often.  To find out which might
be useful in this specific case, it would be good to know which
particular part of the compiler is the bottleneck.  If it's possible to
shrink the problem a bit, so that the compilation is of the order of a
minute or two, it should be possible to get a reasonable picture of
what's going on using the statistical profiler.  (It might also work at
8 minutes; that's "only" 48000 samples by default).

Other things that can alleviate this include splitting the routine into
multiple functions.  I think we have some specialized entry points
(e.g. for passing double-floats unboxed) so with tasteful argument lists
the compiled code might not be substantially slower.

> 3. Initialize-instance, or other CLOS generic functions with large
> numbers of methods.  I generated a file for a fairly large C++ dll to
> create instances of CLOS wrappers of C++ objects.  So you can just
> call make-instance and get and get the method to automatically call
> "new" and "constructors" and set up finalizers with destructor and
> delete.  So I have an 80,000 LOC file with a few thousand
> initialize-instance methods, some just a few lines, others many lines.
> It's fast compiling but then takes like 10 minutes to load.
>
> In another case I compile and load a 250,000 LOC without large numbers
> of methods and it compiles and loads in 30 seconds.  So I think large
> numbers of methods create a problem.

Defining new methods (when loading defmethod forms) inherently invokes
parts of CLOS.  In an ideal world this would be linear in the work done,
but it wouldn't surprise me to find that something is scaling
more-than-linearly.  There are definitely things that have to be done
each time (e.g. running `add-method` on `initialize-instance` --
wouldn't it be nice to be able to run one `add-methods` rather than a
few thousand `add-method` calls?) but maybe some other things can be
done more lazily.

If you shrink this down to half the size, does it take less than half
the time?  Again, a statistical profile report for what's going on here
would be interesting.  

Christophe