Re: [f2py] threading with f2py
charlie strauss <[email protected]>
| Newsgroups | gmane.comp.python.f2py.user |
|---|---|
| Message-ID | <[email protected]> |
On Jul 15, 2009, at 11:21 AM, Thomas Robitaille wrote: > How about using the multiprocessing module which allows you to have > multiple concurrent processes rather than using threads? The API is > very similar to that of the threading module, but the processes are > not limited by the GIL so this allows you to reach 800% utilization > on an 8-core machine. could you say specifically which module you are talking about, and maybe a code snippet? ( also in separate news, it looks like ipython --pylab is busted for forks on a macintosh. the child process can't exit() without crashing. this is, apparently affects many program besides ipthyon. I wonder if multi-processing will be busted this way too on a mac since I assume it uses forks) > > Cheers, > > Tom > > On Jul 15, 2009, at 2:07 AM, charlie strauss wrote: > >> As you may know python threading, unless their is a new module for >> it, suffers from the curse of the global lock. >> >> to remind you, it means that python threading never is able take >> advantage of having more than one CPU because every time it goes to >> access any variable, it locks all variables, blocking all other >> threads for accessing any variable. Thus it can only timeslice a >> single process not have concurrent processes. i.e. threads truly >> stink in python in terms of processor utilization, but still have >> their uses for responding to events. >> >> Now I had read that the exception to this is that any calls to c- >> code or fortran, such as happen during I/O release the global lock >> and thus are not blocking while in the binary. >> >> I must have misunderstood that. Or else I'm doing something wrong. >> >> what I did was I created some fortran code and wrapped if with >> f2py. I then call this code from inside multiple python threads. >> >> now what I expected to happen was to see my processor utilization >> jump up from a mere 100% to 800% since I have 8 cpus and I have 8 >> threads. >> >> instead my processor utilization is still 100% not 800%. >> >> ----- >> >> regression: >> >> at first I was calling my code like this: >> >> from threading import Thread >> >> class testit(Thread): >> >> def __init__(self, r,x): >> Thread.__init__(self) >> self.x = x >> self.r = r >> >> def run (self): >> print "starting" >> self.r = mycode( self.x) >> >> # instantiate 8 threads >> >> x = arange(3) >> r = arange(3) >> for r in range(8): >> current = testit(r,x) >> tlist.append(current) >> >> # start the theads >> for t in tlist: >> t.start() >> >> >> >> where x and r were arrays I passed in during the __init__ that are >> then used in the mycode() call. >> >> what this did was once I started one thread, none of the others >> could be started till the first one finished. >> >> the apparent reason for this was that the mycode() call was passing >> in and out arrays that were not local to the instance, that these >> were getting locked and blocking the parent thread from starting >> any more threads. >> >> Okay so I "fixed" this by making local copies of the arrays so >> there were no non-local variables in the run() method. >> >> >> from threading import Thread >> >> class testit(Thread): >> >> def __init__(self, r,x): >> Thread.__init__(self) >> self.x = x.copy() >> self.r = r.copy() >> >> def run (self): >> print "thread is running" >> self.r = mycode( self.x) >> >> note the added .copy() in the init(), so now run() only has local >> instance variables. >> >> when I run this, indeed all the run() commands execute and announce >> they are running. time passes and then a long time later they all >> finish without errors. >> >> but the CPU never gets up to more than 100% >> >> The only thing I can think of is that either >> >> 1) I don't understand threading in python >> >> 2) that when I linked in mycode() using f2py that this put in >> some layer that has some python variable that is shared by all >> calls to mycode(). as a result, python is blocking in the wrapper >> before it gets into the fortran binary. >> >> any suggestions (and don't say "hey if you want real threading >> try perl" ) >> >> maybe there's some new threading library in python besides Thread >> that does thread right or some trick i can do with f2py that does >> not share any variables? >> >> >> >> >> >> >> _______________________________________________ >> f2py-users mailing list >> f2py-users-Y4l6ocDipWCuvFJfX82//[email protected] >> http://cens.ioc.ee/mailman/listinfo/f2py-users > > > _______________________________________________ > f2py-users mailing list > f2py-users-Y4l6ocDipWCuvFJfX82//[email protected] > http://cens.ioc.ee/mailman/listinfo/f2py-users Charlie Strauss Bioscience Division [email protected] 505 665 4838 Quidquid latine dictum sit, altum sonatur. _______________________________________________ f2py-users mailing list f2py-users-Y4l6ocDipWCuvFJfX82//[email protected] http://cens.ioc.ee/mailman/listinfo/f2py-users