Re: poiu and openmcl

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--0-2007180401-1194630146=:41173
Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE

Sorry for not responding sooner.


Given:

;;; File "a.lisp"
(in-package "CL-USER")

(defun a () 'a)
;;; end of file "a.lisp"

;;; File "b.lisp"
(in-package "CL-USER")

(defun b () 'b)
;;; end of file "b.lisp"

(defun compile-n-times (path n)
   (dotimes (i n) (compile-file path))
   (format t "~& done compiling ~s ~d times" path))

does:

(progn
   (process-run-function "a" #'compile-n-times "a" 10000)
   (process-run-function "b" #'compile-n-times "b" 10000))

fail for you ?  I can't get it to do so, and I can't think of
anything that's not thread-safe about the compiler.  Of course
compiling tiny little files that contain trivial functions doesn't
exercise everything that'd happen in a more realistic scenario,
but the simple act of reading lisp source files that contain
function definitions, compiling them, and writing fasl files seems
to be something that two threads can each do without getting
in each other's way

If this fails for you, please check your OS kernel version.  Many
people at your site run FC3 with fairly old Linux kernels, and we've
found that those old kernels sometimes lose track of thread-local
data.  I don't know exactly what versions have this problem, but
there seems to be a version of 2.6.22 for FC3 that works correctly
in this regard.

More realistic cases might do arbitrary things at compile-time
(via (EVAL-WHEN (:COMPILE-TOPLEVEL ...) ...))  "Arbitrary things"
aren't in general thread-safe, and I don't know of anything reasonable
that the implementation could do to make them be (grabbing a global
eval-when-compile lock and serializing all compile-time processing
doesn't sound too "reasonable").  With very few exceptions (DEFTYPE
is one that comes to mind), the compile-time processing of standard
defining macros doesn't affect any global state (and I can't think
of anything that DEFTYPE does globally that it does in an inherently
non-thread-safe way.)

Changing COMPILE-N-TIMES so that it loads the fasl file each time
doesn't seem to trigger any obvious problems (and it shouldn't.)
Loading "arbitrary code" in parallel could of course lead to
equally arbitrary race conditions and resource contention problems.

Loading CLOS code that defines methods and/or classes in parallel
probably counts as "arbitrary code with unpredictable side-effects";
I don't think that it's possible to make changes to the class
hierarchy or to generic functions in a thread-safe way: it's
not possible to make those changes in a way that the objects
being changed are always in a consistent and usable state, and
I don't believe that it's possible to require that anything
that accesses a class or calls a GF particpate in a locking
protocol in order to ensure "observable consistency".

I assume that you've thought about this and that your parallel
build system handles these issues; if so, and if the OS kernel
isn't an issue, I'm stumped.  The OpenMCL compiler is certainly
supposed to be thread-safe and it'd be a bug if it isn't, but
there are all kinds of obvious and not-so-obvious ways to lose.


On Fri, 26 Oct 2007, Far=E9 wrote:

> Ok, so I wrote a version of poiu (parallelizing variant of asdf) that
> would use threads instead of fork, but I get random errors that look
> like the symptom of race conditions -- OpenMCL's compiler is probably
> not thread-safe. The fork version also had crazy errors that are
> probably due to threading misconfiguration after fork.
>
> Also, asdf-dependency-grovel is slower with openmcl than with sbcl,
> even though normal compilation is 3.5 times faster.
>
> I don't know if you can have access to the ITA code base through
> Clozure, but it would be great if you could help me solve these bugs.
>
> [ Fran=E7ois-Ren=E9 =D0VB Rideau | Reflection&Cybernethics | http://fare.=
tunes.org ]
> Those who place above society the power of politics and bureaucracies as
> solutions to all human problems ignore that when they are not all too hum=
ans,
> politicians and bureaucrats are oh so inhuman.
>
> _______________________________________________
> Bug-openmcl mailing list
> [email protected]
> http://clozure.com/mailman/listinfo/bug-openmcl
>
>
--0-2007180401-1194630146=:41173
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Bug-openmcl mailing list
[email protected]
http://clozure.com/mailman/listinfo/bug-openmcl

--0-2007180401-1194630146=:41173--