Re: [f2py] threading with f2py

Maybe you should try F2PY + Fortran code + OpenMP.

On Wed, Jul 15, 2009 at 2:38 AM, charlie strauss <[email protected]> wrote:

> I came across the follwoing discussion of threads in fortran, which seems
> to suggest that because fortran itself is non-renetrant that threading in
> f2py is going to be impossible as a way to gain performance.
>
> http://math.arizona.edu/~swig/documentation/pthreads/#fortran<http://math.arizona.edu/%7Eswig/documentation/pthreads/#fortran>
>
> Threads and
>           FORTRAN
>
>           It is a sad fact that FORTRAN makes implicit use of
>           global data. For example, in the code fragment
>
>
>
>
>
>         subroutine sub
>         integer i
>
>         do i=1,100
>           ...
>         end do
>
> the temporary loop variable i is probably in the global data
> segment. The only way this subroutine can run in parallel (using
> pthreads) is to lock the entire loop with a mutex -- there is no
> other way to protect the loop counter i. In other words, you cannot
> run it in parallel using pthreads (this is not quite true: the DEC
> FORTRAN compiler has an option that will allocate i on the stack;
> however, this is highly nonportable and probably not worth doing).
> The fundamental problem is that FORTRAN code is, by definition,
> non-reentrant; it is illegal for a FORTRAN subroutine to call
> itself recursively, either directly or indirectly.
>
>
> -------
>
> to try another route I just tried using os.fork() on a macintosh os 10.5
>
> this time it worked up to a point.  namely, the fork produced multiple
> concurrent jobs all running at 100% each for a total of 800% processor
> utilization.
>
> however,  after some time my ipython window just started spewing strange
> error messages. then the mac os crash reporter fired up.  I don't know where
> these are coming from, ipyhton?  f2py?  fortran?  something else?
>
> in anycase the jobs did exit and ipython came back to the command line
> promt.
>
> here's the message repeated many many mnay times :
>
> ).
> Break on
> __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__()
> to debug.
> The process has forked and you cannot use this CoreFoundation functionality
> safely. You MUST exec().
> Break on
> __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__()
> to debug.
>
> ......etc.....
>
>
> and here is the code snippet:
>
> import os
> import sys
> def tryToFork(cbk, fork=True):
>
>     print "trying to fork",cbk
>     #UNIX/LINUX: FORK
>     if fork:
>         try:
>             #Fork and commit suicide
>             if os.fork():
>                print "child"
>                return
>
>             #What to do in parent process
>             else:
>                 print "parent"
>
>                 # call the f2py imported code here:
>                 r_index =
> z.chunk(ww2,ww,data,new_ww,new_ww_count,n_weights,n_vecs,vec_width)
>                 print " fork done"
>                 sys.exit()
>
> for i in range(8):
>     tryToFork(i)
>
>
>
>
>
>
> On Jul 14, 2009, at 6:07 PM, charlie strauss wrote:
>
> As you may know python threading, unless their is a new module for it,
> suffers from the curse of the global lock.
>
> to remind you, it means that python threading never is able take advantage
> of having more than one CPU because every time it goes to access any
> variable, it locks all variables, blocking all other threads for accessing
> any variable.  Thus it can only timeslice a single process not have
> concurrent processes.  i.e. threads truly stink in python in terms of
> processor utilization, but still have their uses for responding to events.
>
> Now I had read that the exception to this is that any calls to c-code or
> fortran, such as happen during I/O release the global lock and thus are not
> blocking while in the binary.
>
> I must have misunderstood that.  Or else I'm doing something wrong.
>
> what I did was I created some fortran code and wrapped if with f2py.   I
> then call this code from inside multiple python threads.
>
> now what I expected to happen was to see my processor utilization jump up
> from a mere 100% to 800% since I have 8 cpus and I have 8 threads.
>
> instead my processor utilization is still 100% not 800%.
>
> -----
>
> regression:
>
> at first I was calling my code like this:
>
> from threading import Thread
>
> class testit(Thread):
>
>  def __init__(self, r,x):
>        Thread.__init__(self)
> self.x = x
> self.r = r
>
>  def run (self):
>    print "starting"
>    self.r = mycode( self.x)
>
> # instantiate 8 threads
>
> x = arange(3)
> r = arange(3)
> for r in range(8):
>   current = testit(r,x)
>   tlist.append(current)
>
> # start the theads
> for t in tlist:
>   t.start()
>
>
>
> where x and r were arrays I passed in during the __init__ that are then
> used in the mycode() call.
>
> what this did was once I started one thread, none of the others could be
> started till the first one finished.
>
> the apparent reason for this was that the mycode() call was passing in and
> out arrays that were not local to the instance, that these were getting
> locked and blocking the parent thread from starting any more threads.
>
> Okay so I "fixed" this by making local copies of the arrays so there were
> no non-local variables in the run() method.
>
>
> from threading import Thread
>
> class testit(Thread):
>
>  def __init__(self, r,x):
>        Thread.__init__(self)
> self.x = x.copy()
> self.r = r.copy()
>
> def run (self):
>    print "thread is running"
>    self.r = mycode( self.x)
>
> note the added .copy() in the init(), so now run() only  has local instance
> variables.
>
> when I run this, indeed all the run() commands execute and announce they
> are running.  time passes and then a long time later they all finish without
> errors.
>
> but the CPU never gets up to more than 100%
>
> The only thing I can think of is that either
>
> 1)  I don't understand threading in python
>
> 2) that when I linked in mycode() using  f2py  that this put in some layer
> that has some python variable that is shared by all calls to mycode().  as a
> result, python is blocking in the wrapper before it gets into the fortran
> binary.
>
> any suggestions  (and don't say  "hey if you want real threading try perl"
>  )
>
> maybe there's some new threading library in python besides Thread that does
> thread right or some trick i can do with f2py that does not share any
> variables?
>
>
>
>
>
>
> _______________________________________________
> f2py-users mailing list
> f2py-users-Y4l6ocDipWCuvFJfX82//[email protected]
> http://cens.ioc.ee/mailman/listinfo/f2py-users
>
>
> Charlie Strauss
> Bioscience Division
> [email protected]
> 505 665 4838
> *Quidquid latine dictum sit, altum sonatur.*
>
>
> _______________________________________________
> f2py-users mailing list
> f2py-users-Y4l6ocDipWCuvFJfX82//[email protected]
> http://cens.ioc.ee/mailman/listinfo/f2py-users
>
>


-- 
M. Modrzejewski

_______________________________________________
f2py-users mailing list
f2py-users-Y4l6ocDipWCuvFJfX82//[email protected]
http://cens.ioc.ee/mailman/listinfo/f2py-users