Re: [f2py] threading with f2py

How about using the multiprocessing module which allows you to have  
multiple concurrent processes rather than using threads? The API is  
very similar to that of the threading module, but the processes are  
not limited by the GIL so this allows you to reach 800% utilization on  
an 8-core machine.

Cheers,

Tom

On Jul 15, 2009, at 2:07 AM, charlie strauss wrote:

> As you may know python threading, unless their is a new module for  
> it, suffers from the curse of the global lock.
>
> to remind you, it means that python threading never is able take  
> advantage of having more than one CPU because every time it goes to  
> access any variable, it locks all variables, blocking all other  
> threads for accessing any variable.  Thus it can only timeslice a  
> single process not have concurrent processes.  i.e. threads truly  
> stink in python in terms of processor utilization, but still have  
> their uses for responding to events.
>
> Now I had read that the exception to this is that any calls to c- 
> code or fortran, such as happen during I/O release the global lock  
> and thus are not blocking while in the binary.
>
> I must have misunderstood that.  Or else I'm doing something wrong.
>
> what I did was I created some fortran code and wrapped if with  
> f2py.   I then call this code from inside multiple python threads.
>
> now what I expected to happen was to see my processor utilization  
> jump up from a mere 100% to 800% since I have 8 cpus and I have 8  
> threads.
>
> instead my processor utilization is still 100% not 800%.
>
> -----
>
> regression:
>
> at first I was calling my code like this:
>
> from threading import Thread
>
> class testit(Thread):
>
>  def __init__(self, r,x):
>        Thread.__init__(self)
> 	self.x = x
> 	self.r = r
>
>  def run (self):
>    print "starting"	
>    self.r = mycode( self.x)
>
> # instantiate 8 threads
>
> x = arange(3)
> r = arange(3)
> for r in range(8):
>   current = testit(r,x)
>   tlist.append(current)
>
> # start the theads
> for t in tlist:
>   t.start()
>
>
>
> where x and r were arrays I passed in during the __init__ that are  
> then used in the mycode() call.
>
> what this did was once I started one thread, none of the others  
> could be started till the first one finished.
>
> the apparent reason for this was that the mycode() call was passing  
> in and out arrays that were not local to the instance, that these  
> were getting locked and blocking the parent thread from starting any  
> more threads.
>
> Okay so I "fixed" this by making local copies of the arrays so there  
> were no non-local variables in the run() method.
>
>
> from threading import Thread
>
> class testit(Thread):
>
>  def __init__(self, r,x):
>        Thread.__init__(self)
> 	self.x = x.copy()
> 	self.r = r.copy()
>
> def run (self):
>    print "thread is running"
>    self.r = mycode( self.x)
>
> note the added .copy() in the init(), so now run() only  has local  
> instance variables.
>
> when I run this, indeed all the run() commands execute and announce  
> they are running.  time passes and then a long time later they all  
> finish without errors.
>
> but the CPU never gets up to more than 100%
>
> The only thing I can think of is that either
>
> 1)  I don't understand threading in python
>
> 2) that when I linked in mycode() using  f2py  that this put in some  
> layer that has some python variable that is shared by all calls to  
> mycode().  as a result, python is blocking in the wrapper before it  
> gets into the fortran binary.
>
> any suggestions  (and don't say  "hey if you want real threading try  
> perl"  )
>
> maybe there's some new threading library in python besides Thread  
> that does thread right or some trick i can do with f2py that does  
> not share any variables?
>
>
>
>
>
>
> _______________________________________________
> f2py-users mailing list
> f2py-users-Y4l6ocDipWCuvFJfX82//[email protected]
> http://cens.ioc.ee/mailman/listinfo/f2py-users