Re: [f2py] threading with f2py

I came across the follwoing discussion of threads in fortran, which  
seems to suggest that because fortran itself is non-renetrant that  
threading in f2py is going to be impossible as a way to gain  
performance.

http://math.arizona.edu/~swig/documentation/pthreads/#fortran

Threads and
           FORTRAN

           It is a sad fact that FORTRAN makes implicit use of
           global data. For example, in the code fragment





         subroutine sub
         integer i

         do i=1,100
           ...
         end do

the temporary loop variable i is probably in the global data
segment. The only way this subroutine can run in parallel (using
pthreads) is to lock the entire loop with a mutex -- there is no
other way to protect the loop counter i. In other words, you cannot
run it in parallel using pthreads (this is not quite true: the DEC
FORTRAN compiler has an option that will allocate i on the stack;
however, this is highly nonportable and probably not worth doing).
The fundamental problem is that FORTRAN code is, by definition,
non-reentrant; it is illegal for a FORTRAN subroutine to call
itself recursively, either directly or indirectly.


-------

to try another route I just tried using os.fork() on a macintosh os 10.5

this time it worked up to a point.  namely, the fork produced multiple  
concurrent jobs all running at 100% each for a total of 800% processor  
utilization.

however,  after some time my ipython window just started spewing  
strange error messages. then the mac os crash reporter fired up.  I  
don't know where these are coming from, ipyhton?  f2py?  fortran?   
something else?

in anycase the jobs did exit and ipython came back to the command line  
promt.

here's the message repeated many many mnay times :

).
Break on  
__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ 
() to debug.
The process has forked and you cannot use this CoreFoundation  
functionality safely. You MUST exec().
Break on  
__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__ 
() to debug.

......etc.....


and here is the code snippet:

import os
import sys
def tryToFork(cbk, fork=True):

     print "trying to fork",cbk
     #UNIX/LINUX: FORK
     if fork:
         try:
             #Fork and commit suicide
             if os.fork():
                print "child"
                return

             #What to do in parent process
             else:
                 print "parent"

                 # call the f2py imported code here:
                 r_index =  
z.chunk(ww2,ww,data,new_ww,new_ww_count,n_weights,n_vecs,vec_width)
                 print " fork done"
                 sys.exit()

for i in range(8):
     tryToFork(i)






On Jul 14, 2009, at 6:07 PM, charlie strauss wrote:

> As you may know python threading, unless their is a new module for  
> it, suffers from the curse of the global lock.
>
> to remind you, it means that python threading never is able take  
> advantage of having more than one CPU because every time it goes to  
> access any variable, it locks all variables, blocking all other  
> threads for accessing any variable.  Thus it can only timeslice a  
> single process not have concurrent processes.  i.e. threads truly  
> stink in python in terms of processor utilization, but still have  
> their uses for responding to events.
>
> Now I had read that the exception to this is that any calls to c- 
> code or fortran, such as happen during I/O release the global lock  
> and thus are not blocking while in the binary.
>
> I must have misunderstood that.  Or else I'm doing something wrong.
>
> what I did was I created some fortran code and wrapped if with  
> f2py.   I then call this code from inside multiple python threads.
>
> now what I expected to happen was to see my processor utilization  
> jump up from a mere 100% to 800% since I have 8 cpus and I have 8  
> threads.
>
> instead my processor utilization is still 100% not 800%.
>
> -----
>
> regression:
>
> at first I was calling my code like this:
>
> from threading import Thread
>
> class testit(Thread):
>
>  def __init__(self, r,x):
>        Thread.__init__(self)
> 	self.x = x
> 	self.r = r
>
>  def run (self):
>    print "starting"	
>    self.r = mycode( self.x)
>
> # instantiate 8 threads
>
> x = arange(3)
> r = arange(3)
> for r in range(8):
>   current = testit(r,x)
>   tlist.append(current)
>
> # start the theads
> for t in tlist:
>   t.start()
>
>
>
> where x and r were arrays I passed in during the __init__ that are  
> then used in the mycode() call.
>
> what this did was once I started one thread, none of the others  
> could be started till the first one finished.
>
> the apparent reason for this was that the mycode() call was passing  
> in and out arrays that were not local to the instance, that these  
> were getting locked and blocking the parent thread from starting any  
> more threads.
>
> Okay so I "fixed" this by making local copies of the arrays so there  
> were no non-local variables in the run() method.
>
>
> from threading import Thread
>
> class testit(Thread):
>
>  def __init__(self, r,x):
>        Thread.__init__(self)
> 	self.x = x.copy()
> 	self.r = r.copy()
>
> def run (self):
>    print "thread is running"
>    self.r = mycode( self.x)
>
> note the added .copy() in the init(), so now run() only  has local  
> instance variables.
>
> when I run this, indeed all the run() commands execute and announce  
> they are running.  time passes and then a long time later they all  
> finish without errors.
>
> but the CPU never gets up to more than 100%
>
> The only thing I can think of is that either
>
> 1)  I don't understand threading in python
>
> 2) that when I linked in mycode() using  f2py  that this put in some  
> layer that has some python variable that is shared by all calls to  
> mycode().  as a result, python is blocking in the wrapper before it  
> gets into the fortran binary.
>
> any suggestions  (and don't say  "hey if you want real threading try  
> perl"  )
>
> maybe there's some new threading library in python besides Thread  
> that does thread right or some trick i can do with f2py that does  
> not share any variables?
>
>
>
>
>
>
> _______________________________________________
> f2py-users mailing list
> f2py-users-Y4l6ocDipWCuvFJfX82//[email protected]
> http://cens.ioc.ee/mailman/listinfo/f2py-users

Charlie Strauss
Bioscience Division
[email protected]
505 665 4838
Quidquid latine dictum sit, altum sonatur.

_______________________________________________
f2py-users mailing list
f2py-users-Y4l6ocDipWCuvFJfX82//[email protected]
http://cens.ioc.ee/mailman/listinfo/f2py-users