RE: DorothyLocker and performance
"Garth T Kidd" <garth-OnzZ1s1DREKDegMON/[email protected]> Tue, 28 Sep 2004 22:23:26 +1000
| Newsgroups | gmane.comp.pythin.pyds.devel |
|---|---|
| Organization | Deadly Bloody Serious |
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. ------=_NextPart_000_0029_01C4A5A9.CA07A050 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit That's a weird one. Oh, hang on. E.E..E.........E.E....E Yep. Python 2.2 works differently to Python 2.3. Here's a fix. -----Original Message----- From: pyds-dev-admin-iYtK5bfT9M//Ad8WF/[email protected] [mailto:pyds-dev-admin-iYtK5bfT9M//Ad8WF/[email protected]] On Behalf Of Bauer, Georg Sent: Tuesday, 28 September 2004 5:24 PM To: 'Garth T Kidd'; Bauer, Georg; 'Thomas Klaeger' Cc: PyDS Developer List (E-Mail) Subject: AW: [Pyds-dev] DorothyLocker and performance Hi! >Oh, did I forget to send that one in? Sorry. :| I had that, alternative >logging for those post-shutdown logs, and code to make sure EventsTool >was last. Your patch wasn't able to be applied - I wrote you this directly and off-list :-) My solution is a bit more flexible, though: you now have a flag to tell where to sort the tool on shutdown. I use that to move the PreferencesTool directly before EventsTool so that the preferences can be still fetched until the very last moment in shutdown. >You're calling PyDS.Tool.getToolsForShutdown twice, just like you used >to getTools. Is there anything wrong with saving the value and using it twice? Changed that, although it really doesn't matter much - the shutdown is only called once in a running instance ;-) >Could you drop in this latest version of DorothyLocker? You're now a >fair way behind, and I've lost track of when I fixed certain bugs. That >might Did that. But it would really be a bit better if you stick to CVS mechanisms - just do an update and merge in the changes I commited and then reproduce your patch with regard to current CVS. It's much more likely that I put in a working patch than full module sources, as the latter would require me to make the diff myself to see what changed. And I am known to be lazy ;-) For example this time: after adding the full module I could'nt start PyDS, because of some changes to the interface of DorothyLocker that required additional patches. But your patch didn't work, as your full module already was installed. Mixing different patch/dist methods allways makes things more complicated. Oh, and now the CVS is broken. One thing was a small typo in Tool.py (wrong indention) that was easily fixed. But now it barfs about "sequence must be integer" on start. What's that? It happens in your __getitem__ method in your FramingContext where it accesses the framestack (line 103). Looks like ltrim_common doesn't produce integers as index somehow (maybe pos not initialized?). This happens directly at startup of PyDS: Exception exceptions.TypeError: sequence index must be integer ('/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py', 1407, '_release', 'return self.lock.release()') ('/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py', 359, 'release', "self.__sanity_check('release', my_context)") ('/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py', 260, '__sanity_check', 'lock_uniques, my_uniques = ltrim_common(lock_context, my_context)') ('/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py', 54, 'ltrim_common', 'return oblist1[pos:], oblist2[pos:]') ('/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py', 103, '__getitem__', 'return self.framestack[index]') Traceback (most recent call last): File "/opt/pyds/bin/pyds-start", line 51, in ? PyDS.Server.start_server(daemon=_PyDS.daemon) File "/opt/pyds/lib/python2.2/site-packages/PyDS/Server.py", line 298, in start_server PyDS.Tool.initTools(PyDS.Tool.mainstatus) File "/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py", line 317, in initTools tool._initdb() File "/opt/pyds/lib/python2.2/site-packages/PyDS/WeblogTool.py", line 77, in _initdb finally: self._release() File "/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py", line 1407, in _release return self.lock.release() File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py", line 359, in release self.__sanity_check('release', my_context) File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py", line 260, in __sanity_check lock_uniques, my_uniques = ltrim_common(lock_context, my_context) File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py", line 54, in ltrim_common return oblist1[pos:], oblist2[pos:] File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py", line 103, in __getitem__ return self.framestack[index] TypeError: sequence index must be integer >also be behind some of the performance hits you're seeing (I used the >latest, assuming you'd incorporated it after that submission I made >that got >blocked for size). I usually write about what I commited (and at least it's in the CVS RSS feed). So if you don't see me mentioning it, it supposedly isn't in CVS. I either wasn't able to apply it (then I usually write something) or I just plain missed it. bye, Georg _______________________________________________ Pyds-dev mailing list Pyds-dev-iYtK5bfT9M//Ad8WF/[email protected] http://www.westfalen.de/cgi-bin/mailman/listinfo/pyds-dev ------=_NextPart_000_0029_01C4A5A9.CA07A050 Content-Type: application/octet-stream; name="20040928-2222-dorothyagain.diff" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="20040928-2222-dorothyagain.diff" Index: PyDS/DorothyLocker.py =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /pyds/PyDS/PyDS/DorothyLocker.py,v retrieving revision 1.5 diff -c -c -r1.5 DorothyLocker.py *** PyDS/DorothyLocker.py 26 Sep 2004 09:11:22 -0000 1.5 --- PyDS/DorothyLocker.py 28 Sep 2004 12:22:18 -0000 *************** *** 1,18 **** """ "There's no place like home." -- Dorothy =20 ! The DorothyLocker module provides a subclass of threading.RLock that=20 ! insists upon release() being called from the same execution frame that = ! acquire()d it in the first place. RLock itself is vulnerable to = missing=20 ! a release() or tossing in an acquire() too many times within a single=20 ! thread, and it won't show until some *other* thread tries to acquire a = ! lock you think is free, and blocks.=20 !=20 ! As a PyDS-specific feature, DorothyRLock skips functions named in = IGNORE.=20 ! This is because each tool routes calls to it's lock's acquire method=20 ! via PyDS.Tool._acquire, and similarly treats release. By ignoring = frames=20 ! from _acquire and _release, DorothyRLock can concentrate on the frames = ! actually causing the locks and releases.=20 """ =20 import threading --- 1,34 ---- """ "There's no place like home." -- Dorothy =20 ! The DorothyLocker module provides a reimplementation of = threading.RLock=20 ! that assists debugging lock contention problems by=20 !=20 ! a) checking whether any previous locks have been correctly released=20 ! before performing subsequent acquire() and release() calls, or=20 ! whenever sanity_check() is called; and !=20 ! b) giving stack trace information of the locking call context when=20 ! another thread has to wait too long for the lock to be released.=20 !=20 ! `DorothyRLock` will raise `LockAssertionError` exceptions for any lock = ! usage other than::=20 !=20 ! try:=20 ! lock.acquire() ! # do stuff ! finally:=20 ! lock.release() !=20 ! DorothyRLock does permit `RLock`'s nesting behaviour, without which = it'd=20 ! just be a normal `Lock`. Each method must, however, release the lock = if=20 ! it acquired it.=20 !=20 ! To support environments like PyDS in which an instance's lock is = managed=20 ! via bound acquire and release methods, DorothyRLock instances can be=20 ! configured with an ignore list. By ignoring frames from these methods, = ! DorothyRLock can concentrate on the frames actually causing the locks = and=20 ! releases. For more details, see the implementation of `IgnoreTests`. """ =20 import threading *************** *** 21,514 **** import time import inspect import sys ! import traceback ! import os.path =20 ! IGNORE =3D ['_acquire', '_release', '__acquire', '__release', = '__DorothyRLock_acquire', '__DorothyRLock_release'] =20 ! def ltrim_common(oblist1, oblist2):=20 ! "Return the arguments with all common first elements removed." ! for pos in range(min(len(oblist1), len(oblist2))):=20 ! if oblist1[pos] is not oblist2[pos]:=20 ! break ! else:=20 ! pos =3D pos + 1 ! return oblist1[pos:], oblist2[pos:] !=20 ! def ping(*args):=20 ! myContext =3D context(ignorecodes=3D[ping.func_code]) ! frame =3D myContext[-1] ! filename, line, name =3D frame.f_code.co_filename, frame.f_lineno, = frame.f_code.co_name ! filename =3D os.path.basename(filename) ! me =3D currentThread() ! args =3D ", ".join([repr(a) for a in args]) ! print "ping! thread %d/%s %s line %d %s" % ( ! id(me), me.getName(), filename, line, args ! ) ! =09 ! def extract_minitrace(frame): ! "Extract a mini-trace from a locking context." ! mycontext =3D context() ! items =3D [] ! while frame:=20 ! if frame in mycontext:=20 ! break ! items.append(( ! frame.f_code.co_filename,=20 ! frame.f_lineno,=20 ! frame.f_code.co_name)) ! frame =3D frame.f_back ! return items !=20 ! def format_minitrace(frame):=20 ! "Format a mini-trace from a locking context." ! msg =3D [] ! for filename, line, name in extract_minitrace(frame):=20 ! msg.append('File "%s", line %d, in %s' % ( ! filename, line, name)) ! return msg !=20 ! def print_minitrace(frame):=20 ! "Print a mini-trace from a locking context." ! print '\n'.join(format_minitrace(frame)) !=20 ! def context(ignorecodes=3D[], ignorenames=3D[]):=20 ! """Distil a calling context, ignoring certain code objects and=20 ! function names.""" ! try:=20 ! stack =3D inspect.stack(context=3D0) ! codestack =3D [] ! for frame, filename, lineno, co_name, lines, index in stack:=20 ! if frame.f_code is context.func_code \ ! or frame.f_code in ignorecodes \ ! or co_name in ignorenames:=20 ! continue ! codestack.append(frame) ! codestack.reverse() ! return tuple(codestack) ! finally:=20 ! del frame =20 class LockAssertionError(AssertionError):=20 ! "Raised if DorothyRLock detects a failure to release an acquired = lock." ! pass =20 class ReAcquireFromSameFrame(LockAssertionError):=20 ! """Release failure was detected when a frame tried to acquire = again.""" =20 - class ReAcquireFromCallingFrame(LockAssertionError):=20 - """Release failure was detected when a calling frame of the locking=20 - frame tried to acquire the lock.""" - =09 - class ReAcquireFromForeignFrame(LockAssertionError):=20 - """Release failure was detected when another frame in the same=20 - thread tried to acquire the lock.""" -=20 - class ReleaseFromForeignFrame(LockAssertionError):=20 - """Release failure was detected when another frame in the same=20 - thread tried to release the lock. Strictly speaking, this requires=20 - both a failure to release and a failure to acquire.""" - =09 class ReleaseFromForeignThread(LockAssertionError):=20 ! """An attempt to release the lock from a thread that didn't own it.=20 ! Strictly speaking, this is probably a failure to acquire.""" ! # RAISED BUT NOT YET TESTED FOR =20 class ReleaseOfUnAcquiredLock(LockAssertionError):=20 ! """An attempt to release an un-acquired lock. Probably caused by=20 ! failure to acquire.""" ! # RAISED BUT NOT YET TESTED FOR =20 class DorothyRLock(_Verbose):=20 ! """DorothyRLock insists upon release() being called from the same=20 ! execution frame that acquire()d it in the first place. Frames named=20 ! in DorothyLocker.IGNORE will be ignored. ! =09 ! If a subframe re-acquires, that's okay, so long as it releases before = ! acquire or release is called via further up the frame stack.""" !=20 ! def squawk(cls):=20 ! "Let users know if we're in use." ! att =3D 'squawked' ! if not hasattr(cls, att):=20 ! msg =3D "Enabled verbose lock debugging via DorothyLocker." ! dash =3D '-' * len(msg) ! print "\n%s\n%s\n%s\n" % (dash, msg, dash) ! setattr(cls, att, True) ! squawk =3D classmethod(squawk) ! =09 ! def __init__(self, verbose=3DNone, name=3D'(unknown)'):=20 ! "Initialise the DorothyRLock." ! _Verbose.__init__(self, verbose) ! self.name =3D name ! self.__ignores =3D [ ! self.acquire.func_code,=20 ! self.release.func_code, ! self.callContext.func_code] ! self.__owner =3D None ! self.__lockStack =3D [] ! self.__block =3D allocate_lock() ! self.__class__.squawk() ! =09 ! def callContext(self):=20 ! "Distil our calling context, with instance-specific ignores." ! return context(self.__ignores, IGNORE) ! =09 ! def acquire(self, blocking=3D1):=20 ! """Acquire the lock, first checking that any prior locks by this=20 ! thread have been released if they're not still on our call chain. ! """ ! =09 ! # First, get our context.=20 ! me =3D currentThread() ! mycontext, myt =3D self.callContext(), time.time() !=20 ! # Are we the owner?=20 ! # This check is safe because owner is me only if we've called=20 ! # acquire() from the same thread.=20 ! if self.__owner is me:=20 ! # Try to figure out whether the previous lock should have been=20 ! # released.=20 ! prevcontext, prevt =3D self.__lockStack[-1] ! prevframe =3D prevcontext[-1] ! # eliminate the common parts of the call stack ! pc, mc =3D ltrim_common(prevcontext, mycontext) ! if pc: ! if mc:=20 ! # different parts of the call stack ! # =3D> failure to release ! raise ReAcquireFromForeignFrame, ("failure to release", = prevframe, prevt) ! else:=20 ! # mc matched, but shorter ! # =3D> re-acquire from calling frame ! raise ReAcquireFromCallingFrame, ("failure to release", = prevframe, prevt) ! else:=20 ! if mc:=20 ! # re-acquired from further down the call stack ! pass ! else:=20 ! # not pc AND not mc ! # =3D> identical contexts ! raise ReAcquireFromSameFrame, ("frame re-called acquire", = prevframe, prevt) !=20 ! # If we got this far, we're OK.=20 ! # Add our details to the lock stack and return success.=20 ! self.__lockStack.append((mycontext, myt)) ! if __debug__:=20 ! self._note("%s.acquire(%s): recursive success", self, blocking) ! return 1 ! =09 ! # The lock isn't owned by this thread. The only way to know for ! # sure it isn't owned by anyone else is to acquire it. Let's try=20 ! # doing it without blocking, first.=20 ! result =3D self.__block.acquire(0) ! =09 ! if not result: # Our attempt to lock failed.=20 ! if not blocking:=20 ! # If we didn't want a blocking call, we can fail outright here.=20 ! if __debug__:=20 ! self._note("%s.acquire(%s): failure", self, blocking) ! return 0 !=20 ! # We know for sure someone else had self.__block at the time=20 ! # we tried to get it, but by now the ownership might have=20 ! # changed or it might have been released. So, we'll start a=20 ! # LockWhiner and then perform a blocking wait.=20 ! whiner =3D LockWhiner(self, mycontext, myt) ! whiner.start() ! result =3D self.__block.acquire(1) ! =09 ! # We can ONLY get here if we succeeded... ! assert result # ... but you can never be too careful.=20 ! =09 ! # Stop the whiner.=20 ! whiner.stop() ! =09 ! # Success! Let's grab the goodies and run.=20 ! self.__owner =3D me ! self.__lockStack =3D [(mycontext, myt)] ! if __debug__:=20 ! self._note("%s.acquire(%s): initial success", self, blocking) ! return 1 !=20 ! def release(self):=20 ! """Release the lock, first checking that we're releasing from the=20 ! same frame that acquired us.""" ! =09 ! # First, get our context.=20 ! me =3D currentThread() ! mycontext, myt =3D self.callContext(), time.time() ! owner =3D self.__owner # just in case it changes !=20 ! if owner is me:=20 ! prevcontext, prevt =3D self.__lockStack[-1] ! if not prevcontext =3D=3D mycontext:=20 ! # TODO: unwind to some adequate level ! raise ReleaseFromForeignFrame, ("failure to release", = prevcontext[-1], prevt) ! elif owner is None:=20 ! raise ReleaseOfUnAcquiredLock, ("release of un-acquired lock", = mycontext[-1], myt) ! else:=20 ! raise ReleaseFromForeignThread, ("releasing thread doesn't own the = lock", mycontext[-1], myt) ! =09 ! # If we got here, all is well.=20 ! topContext, topTime =3D self.__lockStack.pop() ! delay =3D myt - topTime ! if self.__lockStack:=20 ! if __debug__:=20 ! self._note("%s.release(): non-final release after %.2fs", self, = delay) ! else:=20 ! if __debug__:=20 ! self._note("%s.release(): final release after %.2fs", self, delay) ! self.__owner =3D None ! self.__block.release() !=20 ! def __repr__(self):=20 ! "Return a representation of this object." ! return "<%s owned by %s with count %d>" % ( ! self.name,=20 ! self.__owner and self.__owner.getName(),=20 ! len(self.__lockStack)) !=20 ! def lockDetails(self):=20 ! """Return top lock details: owner, lock count, top context, time. ! Because of race conditions, details might not be consistent.""" ! try:=20 ! top =3D self.__lockStack[-1] ! mycontext, myt =3D top ! except IndexError:=20 ! mycontext =3D myt =3D None ! return self.__owner, len(self.__lockStack), mycontext, myt =20 class LockWhiner(Thread):=20 ! def __init__(self, dorothy, whineContext, whineTime, every=3D10, = loudEvery=3D60):=20 ! Thread.__init__(self) ! self.dorothy =3D dorothy ! self.whineTime =3D whineTime ! self.whineContext =3D whineContext ! self.whineEvery =3D every ! self.whineLoudlyEvery =3D loudEvery ! self.whineThread =3D currentThread() ! self.active =3D 1 ! self.setDaemon(1) # let PyDS shut down even if we're active ! =09 ! def stop(self):=20 ! self.stopTime =3D time.time() ! self.active =3D 0 ! print "%s: LockWhiner %d stopped for %s; acquired by thread %d (%s) = after %.2f seconds" % ( ! time.ctime(self.stopTime),=20 ! id(self),=20 ! self.dorothy.name,=20 ! id(self.whineThread),=20 ! self.whineThread.getName(), ! self.stopTime - self.whineTime) !=20 ! def run(self):=20 ! try:=20 ! #import pprint ! #pprint.pprint(self.__dict__) ! self._run() ! except:=20 ! print "_LockWhiner.run: Bugger." ! (e, d, tb) =3D sys.exc_info() ! print 'Exception %s: %s' % (e, d) ! for row in traceback.extract_tb(tb): ! print repr(row) ! print 'Dict:'=20 ! for key in self.__dict__.keys():=20 ! print ' %s =3D %s' % (key, repr(self.__dict__[key])) !=20 ! def _run(self):=20 ! print "%s: LockWhiner %d started for %s; thread %d (%s) waiting" % = (\ ! time.ctime(self.whineTime),=20 ! id(self),=20 ! self.dorothy.name,=20 ! id(self.whineThread), ! self.whineThread.getName()) ! waits =3D 0 ! louds =3D int(self.whineLoudlyEvery/self.whineEvery) ! while self.active:=20 ! time.sleep(self.whineEvery) ! if not self.active:=20 ! break ! waits =3D waits + 1 ! now =3D time.time() ! print "%s: LockWhiner %d has been waiting for %.2fs" % ( ! time.ctime(), ! id(self),=20 ! now - self.whineTime) ! if not (waits-1) % louds:=20 ! msg =3D [] ! add =3D msg.append ! add("-"*70) ! add("Just in case this is a deadlock, here's some additional = information") ! add("for debugging purposes:\n") ! owner, lockCount, topContext, topTime =3D = self.dorothy.lockDetails() ! if topTime is None:=20 ! timeRep =3D "(unknown)" ! else:=20 ! timeRep =3D time.ctime(topTime) ! add("Lock most recently obtained at %s by thread id %d, name %s" % = (timeRep, id(owner), owner.getName())) ! if not owner.isAlive():=20 ! add("**THREAD IS DEAD**") ! add("\nLock holding stack:") ! if topContext is None:=20 ! add("(unknown)") ! else: ! msg.extend(format_minitrace(topContext[-1])) ! if waits=3D=3D1:=20 ! add("\nWaiting thread: id %d, name %s" % (id(self.whineThread), = self.whineThread.getName())) ! add("\nWaiting stack:") ! msg.extend(format_minitrace(self.whineContext[-1])) ! add("-"*70) ! print '\n'.join(msg) =20 if __name__ =3D=3D '__main__':=20 ! import unittest ! import Queue =20 ! class BasicTests(unittest.TestCase):=20 ! "Basic tests." =20 ! def test_creation(self):=20 ! "Creation" ! dorothy =3D DorothyRLock() ! =09 ! def test_verbose_creation(self):=20 ! "Verbose creation" ! dorothy =3D DorothyRLock(verbose=3DTrue) ! =09 ! def test_named_creation(self):=20 ! "Named creation" ! dorothy =3D DorothyRLock(name=3D'lockName') ! =09 ! def test_verbose_named_creation(self):=20 ! "Verbose, named creation" ! dorothy =3D DorothyRLock(verbose=3DTrue, name=3D'lockName') ! =09 ! def test_acquire_release(self):=20 ! "Basic acquisition and release" ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! dorothy.release() ! =09 ! def test_mistaken_release_of_unacquired_lock(self):=20 ! "React properly to someone releasing an un-acquired lock" ! def mistaken_release_of_unacquired_lock():=20 ! dorothy =3D DorothyRLock() ! dorothy.release() ! self.assertRaises(ReleaseOfUnAcquiredLock, = mistaken_release_of_unacquired_lock) !=20 ! class ReleaseFailureTests(unittest.TestCase): ! "Tests of the release failure detection mechanism." !=20 ! def acquire_with_no_release(self, dorothy):=20 ! "Acquire, but don't release, `dorothy`. Used by many tests." ! dorothy.acquire() ! # didn't call dorothy.release() -- oops! ! =09 ! def test_boxed_pairs(self):=20 ! "Permit subframes to re-acquire" !=20 ! def reacquire(dorothy):=20 ! dorothy.acquire() ! dorothy.release() !=20 ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! reacquire(dorothy) ! dorothy.release() !=20 ! def test_release_from_calling_frame(self):=20 ! "Detect release failure when a calling frame releases" !=20 ! def release_from_calling_frame():=20 ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! self.acquire_with_no_release(dorothy) ! dorothy.release() ! =09 ! self.assertRaises(ReleaseFromForeignFrame, = release_from_calling_frame) !=20 ! def test_reacquire_from_calling_frame(self):=20 ! "Detect release failure when a calling frame re-acquires" !=20 ! def acquire_with_no_release(dorothy):=20 ! dorothy.acquire() ! # didn't call dorothy.release() -- oops! ! =09 ! def reacquire_from_calling_frame():=20 ! dorothy =3D DorothyRLock() ! self.acquire_with_no_release(dorothy) ! dorothy.acquire() ! =09 ! self.assertRaises(ReAcquireFromCallingFrame, = reacquire_from_calling_frame) ! =09 ! def test_reacquire_from_foreign_frame(self):=20 ! "Detect release failure when the lock is re-acquired from = elsewhere" !=20 ! def reacquire_from_foreign_frame(): ! dorothy =3D DorothyRLock() ! self.acquire_with_no_release(dorothy) ! # the second time around, this is a different *frame*=20 ! # despite it being the same callable object ! self.acquire_with_no_release(dorothy) !=20 ! self.assertRaises(ReAcquireFromForeignFrame, = reacquire_from_foreign_frame) ! =09 ! def test_reacquire_from_same_frame(self):=20 ! "Detect release failure when the lock is re-acquired from the same = frame" !=20 ! def reacquire_from_same_frame(): ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! dorothy.acquire() !=20 ! self.assertRaises(ReAcquireFromSameFrame, = reacquire_from_same_frame) ! =09 ! class LockSittingThread(threading.Thread):=20 ! "Thread that sits on a lock. Automatically starts itself" !=20 ! def __init__(self, lock):=20 ! "Initialise and start the `LockSittingThread`." ! self.lock =3D lock ! self.acquired =3D 0 ! self.keepGoing =3D 1 ! threading.Thread.__init__(self) !=20 ! # start, and wait for acquisition ! self.start() ! while not self.acquired:=20 ! time.sleep(0.1) ! =09 ! def run(self):=20 ! "In the thread: acquire the lock, wait, and release when told." ! self.lock.acquire() ! self.acquired =3D 1 ! while self.keepGoing:=20 ! time.sleep(0.1) ! self.lock.release() !=20 ! def stop(self):=20 ! "Ask the thread to stop and release, and wait for it to obey." ! self.keepGoing =3D 0 ! self.join() ! =09 ! class MultiThreadedTests(unittest.TestCase):=20 ! "Tests of multi-threaded locking behaviour." !=20 ! def test_nonblocking_acquire_of_acquired_lock(self):=20 ! "Fail non-blocking acquires of a lock acquired by another thread" ! dorothy =3D DorothyRLock() ! sitter =3D LockSittingThread(dorothy) ! self.assert_(not dorothy.acquire(0)) ! sitter.stop() !=20 ! def test_foreign_release(self):=20 ! "Catch one thread releasing a lock acquired by another thread" ! dorothy =3D DorothyRLock() ! sitter =3D LockSittingThread(dorothy) ! self.assertRaises(ReleaseFromForeignThread, dorothy.release) ! sitter.stop() =20 ! # Run the tests. ! unittest.main() --- 37,879 ---- import time import inspect import sys ! import re ! import types =20 ! __all__ =3D ['DorothyRLock', 'LockAssertionError', 'CallingContext'] =20 ! __revision__ =3D '$Id$' =20 + def ltrim_common(oblist1, oblist2):=20 + "Return the arguments with all common first elements removed." + for pos in range(min(len(oblist1), len(oblist2))):=20 + if not oblist1[pos] =3D=3D oblist2[pos]:=20 + break + else:=20 + pos =3D pos + 1 + return oblist1[pos:], oblist2[pos:] +=20 + class FrameInfo(object):=20 + "Frozen frame information." +=20 + def __init__(self, frame):=20 + "Initialise ourself from a frame." + object.__init__(self) + self.frame_id =3D id(frame) + self.f_code =3D frame.f_code + self.f_lineno =3D frame.f_lineno +=20 + def __eq__(self, other):=20 + "Does this refer to the same frame ID as the other?" + return self.frame_id =3D=3D other.frame_id + =20 + class CallingContext(object):=20 + """Object representing the calling context as a tuple of FrameInfo = + objects, one per calling frame. The context is reversed so that=20 + earlier frames are first, not last.""" +=20 + def __init__(self, ignores=3DNone):=20 + """Distil the calling context, ignoring nominated code objects = and=20 + function names. + =20 + ignores -- a list of function names or code objects to ignore + """ + object.__init__(self) # good form + self.when =3D time.time() + self.thread =3D currentThread() + frame =3D None + if ignores is None:=20 + ignores =3D [] + try:=20 + stack =3D inspect.stack(context=3D0) + framestack =3D [] + for frame, filename, lineno, co_name, lines, index in = stack:=20 + if frame.f_code is self.__init__.im_func.func_code \ + or frame.f_code in ignores or co_name in ignores:=20 + continue + assert type(frame) =3D=3D types.FrameType + framestack.append(FrameInfo(frame)) + framestack.reverse() + self.framestack =3D tuple(framestack) + finally:=20 + del frame # apparently helpful with frame objects + =20 + def __getslice__(self, i, j):=20 + "Get a slice of frames from the context." + return self.framestack.__getslice__(i, j) +=20 + def __getitem__(self, index):=20 + "Get a frame from the context by number (-1 is most recent)." + return self.framestack[index] +=20 + def __len__(self):=20 + "How many frames are in the context?" + return len(self.framestack) +=20 + def minitrace(self):=20 + "Extract a mini-trace from the context." + tups =3D [] + for frame in self:=20 + tups.append((frame.f_code.co_filename,=20 + frame.f_lineno,=20 + frame.f_code.co_name)) + tups.reverse() + return tups +=20 + def formatted_minitrace(self):=20 + "Return a formatted mini-trace of the context." + lines =3D [] + for filename, line, name in self.minitrace():=20 + lines.append('File "%s", line %d, in %s' % ( + filename, line, name)) + return lines +=20 + def print_minitrace(self, out=3DNone):=20 + "Print a mini-trace of the context." + if out is None:=20 + out =3D sys.stdout + print >> out, '\n'.join(self.formatted_minitrace()) + =20 + def consolidate_string(string):=20 + """Consolidate a string, eliminating redundant whitespace. + Intended to turn docstrings into single-line strings.""" + return re.sub('\s+', ' ', string).strip() + =20 class LockAssertionError(AssertionError):=20 ! "Raised if DorothyRLock detects a failure to release an acquired = lock." ! def __init__(self, lock_context, detection_context, message=3D""): = ! self.lock_context =3D lock_context ! self.detection_context =3D detection_context !=20 ! # If the user didn't specify a message, use our doc string. ! if not message:=20 ! message =3D consolidate_string(self.__doc__) ! self.message =3D message !=20 ! # Super call; good form, and helps anyone who looks at .args ! AssertionError.__init__(self, lock_context, detection_context, = ! message) !=20 ! def formatted_diagnosis(self):=20 ! "Return lines of a formatted diagnosis of the = LockAssertionError." ! name =3D self.__class__.__name__ ! dash =3D '-' * len(name) ! msg =3D [dash, name, dash, self.message] ! if self.detection_context:=20 ! msg.append("Detection context:") ! msg.extend([' '+line for line in=20 ! self.detection_context.formatted_minitrace()]) ! if self.lock_context:=20 ! msg.append("Lock context:") ! msg.extend([' '+line for line in=20 ! self.lock_context.formatted_minitrace()]) ! return msg !=20 ! def print_diagnosis(self, out=3DNone):=20 ! "Print a diagnosis of the LockAssertionError." ! if out is None:=20 ! out =3D sys.stdout ! print >> out, '\n'.join(self.formatted_diagnosis()) !=20 ! class LeftAcquiringFrame(LockAssertionError):=20 ! """The lock owner's call stack no longer contains the frame=20 ! from which the owner acquired the lock. This is almost certainly=20 ! a failure to release the lock.""" !=20 ! class ReleaseFromSubFrame(LockAssertionError):=20 ! """A sub-frame of the locking frame tried to release. Either the=20 ! locking frame failed to release, or the sub-frame failed to=20 ! acquire.""" =20 class ReAcquireFromSameFrame(LockAssertionError):=20 ! """A frame tried to acquire again, indicating a probable release=20 ! in a method which has more than one acquire/release pair or which=20 ! acquires and releases inside a loop.""" =20 class ReleaseFromForeignThread(LockAssertionError):=20 ! """An attempt to release the lock from a thread that didn't own = it.=20 ! Strictly speaking, this is probably a failure to acquire.""" =20 class ReleaseOfUnAcquiredLock(LockAssertionError):=20 ! """An attempt to release an un-acquired lock. Probably caused by=20 ! failure to acquire.""" =20 class DorothyRLock(_Verbose):=20 ! """DorothyRLock insists upon release() being called from the same=20 ! execution frame that acquire()d it in the first place. Frames = named=20 ! in DorothyLocker.IGNORE will be ignored. ! =20 ! If a subframe re-acquires, that's okay, so long as it releases = before=20 ! acquire or release is called via further up the frame stack.""" !=20 ! def squawk(cls):=20 ! "Let users know if we're in use." ! att =3D 'squawked' ! if not hasattr(cls, att):=20 ! msg =3D "Enabled verbose lock debugging via = DorothyLocker." ! dash =3D '-' * len(msg) ! print "\n%s\n%s\n%s\n" % (dash, msg, dash) ! setattr(cls, att, True) ! squawk =3D classmethod(squawk) ! =20 ! def __init__(self, verbose=3DNone, name=3D'(unknown)', = ignores=3DNone):=20 ! "Initialise the DorothyRLock." ! _Verbose.__init__(self, verbose) ! self.name =3D name ! self.__ignores =3D [ ! self.acquire.func_code,=20 ! self.release.func_code, ! self.call_context.func_code, ! self.sanity_check.func_code] ! if ignores is not None:=20 ! self.__ignores.extend(ignores) ! self.__owner =3D None ! self.__lockstack =3D [] ! self.__block =3D allocate_lock() ! self.__class__.squawk() ! =20 ! def call_context(self):=20 ! "Distil our calling context, with instance-specific ignores." ! return CallingContext(ignores=3Dself.__ignores) ! =20 ! def sanity_check(self):=20 ! """Perform a sanity check: if we're locked by this thread, and = ! the calling frame doesn't share the locking frame, we know the = ! locking frame didn't release. ! =20 ! Be careful calling this if you're about to release.""" !=20 ! self.__sanity_check('anytime', self.call_context()) !=20 ! def __sanity_check(self, mode, my_context):=20 ! """Perform sanity checks when asked by acquire, release, or = the=20 ! public sanity_check method. !=20 ! mode -- 'anytime', 'acquire', or 'release'. ! my_thread -- my thread object. ! my_context -- a CallingContext obtained via call_context. ! """ ! =20 ! assert mode in ['anytime', 'acquire', 'release'] !=20 ! owner =3D self.__owner !=20 ! if owner is my_context.thread:=20 ! lock_context =3D self.__lockstack[-1] ! # eliminate the common parts of the call stack ! lock_uniques, my_uniques =3D ltrim_common(lock_context, = my_context) ! if lock_uniques: ! # if my_uniques, it's different part of the call stack ! # if not my_uniques, we're in one of our calling = frames ! # this is bad nomatter what the sanity check mode ! raise LeftAcquiringFrame, (lock_context, my_context) ! else:=20 ! if my_uniques:=20 ! # we're further down the call stack from the = acquire,=20 ! # which is:=20 ! # * wrong for release,=20 ! # * fine for sanity check, and ! # * fine for acquire ! if mode =3D=3D 'release':=20 ! raise ReleaseFromSubFrame, (lock_context, = my_context) ! else:=20 ! # the contexts are identical, which is:=20 ! # * perfect for release, ! # * unusual for sanity_check, and ! # * highly unusual for acquire ! if mode =3D=3D 'acquire': ! raise ReAcquireFromSameFrame, (lock_context, = my_context) ! =20 ! elif owner is None:=20 ! # fine for sanity and acquire, bad for release ! if mode =3D=3D 'release':=20 ! raise ReleaseOfUnAcquiredLock, (None, my_context) !=20 ! else: # foreign owner ! # fine for sanity, acquire will happily block, bad for = release ! lock_context =3D self.__lockstack[-1] ! if mode =3D=3D 'release':=20 ! raise ReleaseFromForeignThread, (lock_context, = my_context) !=20 ! def acquire(self, blocking=3D1):=20 ! """Acquire the lock, first checking that any prior locks by = this=20 ! thread have been released if they're not still on our call = chain. ! """ ! =20 ! # First, get our context.=20 ! my_context =3D self.call_context() !=20 ! # Are we the owner?=20 ! # This check is safe because owner is me only if we've called=20 ! # acquire() from the same thread.=20 ! if self.__owner is my_context.thread:=20 ! # Try to figure out whether the previous lock should have = been=20 ! # released.=20 ! self.__sanity_check('acquire', my_context) !=20 ! # If we got this far, we're OK.=20 ! # Add our details to the lock stack and return success.=20 ! self.__lockstack.append(my_context) ! if __debug__:=20 ! self._note("%s.acquire(%s): recursive success", self, = blocking) ! return 1 ! =20 ! # The lock isn't owned by this thread. The only way to know = for ! # sure it isn't owned by anyone else is to acquire it. Let's = try=20 ! # doing it without blocking, first.=20 ! result =3D self.__block.acquire(0) ! =20 ! if not result: # Our attempt to lock failed.=20 ! if not blocking:=20 ! # If we didn't want a blocking call, we can fail = outright here.=20 ! if __debug__:=20 ! self._note("%s.acquire(%s): failure", self, = blocking) ! return 0 !=20 ! # We know for sure someone else had self.__block at the = time=20 ! # we tried to get it, but by now the ownership might have=20 ! # changed or it might have been released. So, we'll start = a=20 ! # LockWhiner and then perform a blocking wait.=20 ! whiner =3D LockWhiner(self, my_context) ! whiner.start() ! result =3D self.__block.acquire() ! =20 ! # We can ONLY get here if we succeeded... ! assert result # ... but you can never be too careful.=20 ! =20 ! # Stop the whiner.=20 ! whiner.stop() ! =20 ! # Success! Let's grab the goodies and run.=20 ! self.__owner =3D my_context.thread ! self.__lockstack =3D [my_context] ! if __debug__:=20 ! self._note("%s.acquire(%s): initial success", self, = blocking) ! return 1 !=20 ! def release(self):=20 ! """Release the lock, first checking that we're releasing from = the=20 ! same frame that acquired us.""" ! =20 ! # First, get our context.=20 ! my_context =3D self.call_context() !=20 ! # Raise an appropriate exception if we detect a problem or=20 ! # shouldn't release (say, because we don't own the lock).=20 ! self.__sanity_check('release', my_context) ! =20 ! # If we got here, all is well.=20 ! lock_context =3D self.__lockstack.pop() ! delay =3D my_context.when - lock_context.when ! if self.__lockstack:=20 ! if __debug__:=20 ! self._note("%s.release(): non-final release after = %.2fs",=20 ! self, delay) ! else:=20 ! if __debug__:=20 ! self._note("%s.release(): final release after %.2fs",=20 ! self, delay) ! self.__owner =3D None ! self.__block.release() ! # That better be the last statement that affects state, = because ! # a thread waiting in the middle of acquire() will soon be = ! # adjusting the state itself.=20 !=20 ! def __repr__(self):=20 ! "Return a representation of this object." ! return "<%s owned by %s with count %d>" % ( ! self.name,=20 ! self.__owner and self.__owner.getName(),=20 ! len(self.__lockstack)) !=20 ! def lock_details(self):=20 ! """Return top lock details: owner, lock count, top context, = time. ! Because of race conditions, details might not be = consistent.""" ! try:=20 ! lock_context =3D self.__lockstack[-1] ! except IndexError:=20 ! lock_context =3D None ! return self.__owner, len(self.__lockstack), lock_context =20 class LockWhiner(Thread):=20 ! """Thread that whines until a DorothyRLock.acquire() succeeds,=20 ! occasionally whining loudly with [hopefully] helpful debugging=20 ! information about the current owner of the lock.""" ! =20 ! def __init__(self, dorothy, whine_context, out=3DNone, every=3D6,=20 ! loudly_every=3DNone):=20 ! """Initialise the LockWhiner. ! =20 ! dorothy -- the lock for which we're waiting ! whine_context -- the call context that's waiting ! out -- file-like-object to whine to ! every -- how many seconds to wait between whines ! loudly_every -- how many seconds to wait between loud whines ! """ ! self.dorothy =3D dorothy ! self.whine_context =3D whine_context ! self.every =3D every ! if out is None:=20 ! out =3D sys.stdout ! self.out =3D out ! if loudly_every:=20 ! self.loudly_every =3D loudly_every ! else:=20 ! self.loudly_every =3D every * 10 ! self.active =3D 1 ! self.stop_time =3D 0 # filled in by stop() ! self.exc_info =3D None # filled in if necessary by run() !=20 ! # Super call -- enforced by Thread.start and other methods ! Thread.__init__(self) ! self.setDaemon(1) # let Python shut down even if we're still = active !=20 ! def stop(self):=20 ! """Ask the LockWhiner to stop.""" ! self.stop_time =3D time.time() ! self.active =3D 0 ! delay =3D self.stop_time - self.whine_context.when ! print >> self.out, "%s: LockWhiner %d stopped for %s; "\ ! "acquired by thread %d (%s) after %.2f seconds" % ( ! time.ctime(self.stop_time),=20 ! id(self),=20 ! self.dorothy.name,=20 ! id(self.whine_context.thread),=20 ! self.whine_context.thread.getName(), ! delay) !=20 ! def run(self):=20 ! """Whine until stopped.""" ! try:=20 ! self._run() ! except Exception:=20 ! # Stash the exception for later if we're supposed to still = ! # be waiting. If not, we probably crashed creating a = report=20 ! # because the lock changed status.=20 ! if self.active:=20 ! self.exc_info =3D sys.exc_info() !=20 ! def check(self):=20 ! """Check for exceptions caught by `run`.""" ! if self.exc_info is not None:=20 ! extype, exvalue, extraceback =3D self.exc_info ! raise extype, exvalue, extraceback !=20 ! def _run(self):=20 ! """Whine until stopped.""" ! out =3D self.out ! whine_time =3D self.whine_context.when ! print >> out, "%s: LockWhiner %d started for %s; "\ ! "thread %d (%s) waiting" % (\ ! time.ctime(whine_time),=20 ! id(self),=20 ! self.dorothy.name,=20 ! id(self.whine_context.thread), ! self.whine_context.thread.getName()) ! waits =3D 0 ! louds =3D int(self.loudly_every/self.every) ! while self.active:=20 ! time.sleep(self.every) ! if not self.active:=20 ! break ! waits =3D waits + 1 ! now =3D time.time() ! print >> out, "%s: LockWhiner %d has been waiting for = %.2fs" % ( ! time.ctime(now), ! id(self),=20 ! now - whine_time) ! if not (waits-1) % louds:=20 ! # Prepare a detailed complaint. This will fail if the=20 ! # lock is suddenly released and its state changes,=20 ! # but that won't stop any thread but the whiner.=20 ! msg =3D [] ! add =3D msg.append ! add("-"*70) ! add("Just in case this is a deadlock, here's some "\ ! "additional information") ! add("for debugging purposes:\n") ! owner, lock_count, lock_context =3D \ ! self.dorothy.lock_details() ! if lock_context is None:=20 ! time_rep =3D "(unknown)" ! else:=20 ! time_rep =3D time.ctime(lock_context.when) ! add("Lock most recently obtained at %s "\ ! "by thread id %d, name %s" % ( ! time_rep, id(owner), owner.getName())) ! add("Lock count by this thread: %d" % lock_count) ! if not owner.isAlive():=20 ! add("**THREAD IS DEAD**") ! add("\nLock holding stack:") ! if lock_context is None:=20 ! add("(unknown)") ! else: ! msg.extend(lock_context.formatted_minitrace()) ! if waits =3D=3D 1:=20 ! add("\nWaiting thread: id %d, name %s" % ( ! id(self.whine_context.thread),=20 ! self.whine_context.thread.getName())) ! add("\nWaiting stack:") ! = msg.extend(self.whine_context.formatted_minitrace()) ! add("-"*70) ! print >> out, '\n'.join(msg) =20 if __name__ =3D=3D '__main__':=20 ! import unittest ! import StringIO !=20 ! class CallingContextTests(unittest.TestCase):=20 ! "Tests of multi-threaded locking behaviour." =20 ! # First, define some methods used by the tests.=20 ! # Argument funkiness used to pass arguments to CallingContext. = =20 ! def one(self, ignores=3DNone):=20 ! "Return whatever `two` returns." ! return self.two(ignores) ! =20 ! def two(self, ignores=3DNone):=20 ! "Return whatever 'three' returns." ! return self.three(ignores) ! =20 ! def three(self, ignores=3DNone): ! "Return `three`'s call context." ! return CallingContext(ignores) !=20 ! # Now, the tests:=20 !=20 ! def test_creation(self):=20 ! "Test call context creation" ! CallingContext() !=20 ! def test_basic(self):=20 ! "Basic context check" ! context =3D self.one() ! assert context[-1].f_code is self.three.func_code ! assert context[-2].f_code is self.two.func_code ! assert context[-3].f_code is self.one.func_code !=20 ! def test_ignorecodes(self):=20 ! "Make sure ignoring codes works" ! context =3D self.one(ignores=3D[self.three.func_code]) ! assert context[-1].f_code is self.two.func_code !=20 ! def test_ignorenames(self):=20 ! "Make sure ignoring names works" ! context =3D self.one(ignores=3D['three']) ! assert context[-1].f_code is self.two.func_code !=20 ! def test_getitem(self):=20 ! "Get last frame as context[-1]" ! context =3D self.one() ! context[-1] ! =20 ! def test_len(self):=20 ! "Return number of frames in context as len(context)" ! context =3D self.one() ! assert len(context) > 3 ! =20 ! def test_minitrace(self):=20 ! "Context mini-trace" ! context =3D self.one() ! minitrace =3D context.minitrace() ! assert isinstance(minitrace, list) ! if hasattr(__builtins__, 'basestring'):=20 ! stringtype =3D basestring ! else:=20 ! stringtype =3D str ! for filename, lineno, name in minitrace:=20 ! assert isinstance(filename, stringtype) ! assert isinstance(lineno, int) ! assert isinstance(name, stringtype) ! assert minitrace[0][2] =3D=3D 'three' ! try:=20 ! assert minitrace[0][0] =3D=3D __file__ ! except NameError:=20 ! pass # Python 2.2 ! =20 ! def test_formatted_minitrace(self):=20 ! "Formatted context mini-trace" ! context =3D self.one() ! formatted_minitrace =3D context.formatted_minitrace() ! assert isinstance(formatted_minitrace, list) !=20 ! def test_print_minitrace(self):=20 ! "Printed context mini-trace" ! out =3D StringIO.StringIO() ! context =3D self.one() ! context.print_minitrace(out) ! assert len(out.getvalue()) ! out.close() !=20 ! class BasicTests(unittest.TestCase):=20 ! "Basic tests." !=20 ! def test_creation(self):=20 ! "Creation" ! DorothyRLock() ! =20 ! def test_verbose_creation(self):=20 ! "Verbose creation" ! DorothyRLock(verbose=3DTrue) ! =20 ! def test_named_creation(self):=20 ! "Named creation" ! DorothyRLock(name=3D'lockName') ! =20 ! def test_verbose_named_creation(self):=20 ! "Verbose, named creation" ! DorothyRLock(verbose=3DTrue, name=3D'lockName') ! =20 ! def test_acquire_release(self):=20 ! "Basic acquisition and release" ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! dorothy.release() !=20 ! def test_mistaken_release_of_unacquired_lock(self):=20 ! "React properly to someone releasing an un-acquired lock" !=20 ! def mistaken_release_of_unacquired_lock():=20 ! "Release a lock without acquiring it." ! dorothy =3D DorothyRLock() ! dorothy.release() !=20 ! self.assertRaises(ReleaseOfUnAcquiredLock,=20 ! mistaken_release_of_unacquired_lock) !=20 ! def test_formatted_diagnosis(self):=20 ! "LockAssertionError formatted diagnostics" ! dorothy =3D DorothyRLock() ! try:=20 ! dorothy.acquire() ! dorothy.acquire() ! except LockAssertionError, ex:=20 ! formatted_diagnosis =3D ex.formatted_diagnosis() ! assert len(formatted_diagnosis) !=20 ! def test_printed_diagnosis(self):=20 ! "LockAssertionError printed diagnostics" ! dorothy =3D DorothyRLock() ! try:=20 ! dorothy.acquire() ! dorothy.acquire() ! except LockAssertionError, ex:=20 ! out =3D StringIO.StringIO() ! ex.print_diagnosis(out) ! assert len(out.getvalue()) ! out.close() ! =20 ! class ReleaseFailureTests(unittest.TestCase): ! "Tests of the release failure detection mechanism." !=20 ! def acquire_and_release(self, dorothy):=20 ! "Acquire and release `dorothy`. Used by tests." ! dorothy.acquire() ! dorothy.release() !=20 ! def acquire_with_no_release(self, dorothy):=20 ! "Acquire, but don't release, `dorothy`. Used by tests." ! dorothy.acquire() ! # didn't call dorothy.release() -- oops! ! =20 ! def release_with_no_acquire(self, dorothy):=20 ! "Release, but don't acquire, `dorothy`. Used by tests." ! # didn't call dorothy.acquire() -- oops! ! dorothy.release() ! =20 ! def test_boxed_pairs(self):=20 ! "Permit subframes to re-acquire" ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! self.acquire_and_release(dorothy) ! dorothy.release() !=20 ! def test_release_from_calling_frame(self):=20 ! "Detect release failure when a calling frame releases" !=20 ! def release_from_calling_frame():=20 ! "Force `LeftAcquiringFrame`." ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! self.acquire_with_no_release(dorothy) ! dorothy.release() ! =20 ! self.assertRaises(LeftAcquiringFrame, = release_from_calling_frame) !=20 ! def test_release_from_sub_frame(self):=20 ! "Detect acquire failure when a sub-frame releases" !=20 ! def release_from_sub_frame():=20 ! "Force `ReleaseFromSubFrame`." ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! self.release_with_no_acquire(dorothy) ! dorothy.release() ! =20 ! self.assertRaises(ReleaseFromSubFrame, = release_from_sub_frame) !=20 ! def test_reacquire_from_calling_frame(self):=20 ! "Detect release failure when a calling frame re-acquires" !=20 ! def reacquire_from_calling_frame():=20 ! "Force `LeftAcquiringFrame`." ! dorothy =3D DorothyRLock() ! self.acquire_with_no_release(dorothy) ! dorothy.acquire() ! =20 ! self.assertRaises(LeftAcquiringFrame, = reacquire_from_calling_frame) ! =20 ! def test_reacquire_from_foreign_frame(self):=20 ! "Detect release failure when the lock is re-acquired from = elsewhere" !=20 ! def reacquire_from_foreign_frame(): ! "Force `LeftAcquiringFrame`." ! dorothy =3D DorothyRLock() ! self.acquire_with_no_release(dorothy) ! # the second time around, this is a different *frame*=20 ! # despite it being the same callable object ! self.acquire_with_no_release(dorothy) !=20 ! self.assertRaises(LeftAcquiringFrame, = reacquire_from_foreign_frame) ! =20 ! def test_reacquire_from_same_frame(self):=20 ! "Detect release failure when lock re-acquired from the = same frame" !=20 ! def reacquire_from_same_frame(): ! "Force `ReAcquireFromSameFrame`." ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! dorothy.acquire() !=20 ! self.assertRaises(ReAcquireFromSameFrame, = reacquire_from_same_frame) ! =20 ! class LockSittingThread(threading.Thread):=20 ! "Thread that sits on a lock. Automatically starts itself" !=20 ! def __init__(self, lock):=20 ! "Initialise and start the `LockSittingThread`." ! self.lock =3D lock ! self.acquired =3D 0 ! self.keepGoing =3D 1 ! threading.Thread.__init__(self) ! self.setDaemon(1) # lets test suite exit if thread still = going !=20 ! # start, and wait for acquisition ! self.start() ! while not self.acquired:=20 ! time.sleep(0.1) ! =20 ! def run(self):=20 ! "In the thread: acquire the lock, wait, and release when = told." ! self.lock.acquire() ! self.acquired =3D 1 ! while self.keepGoing:=20 ! time.sleep(0.1) ! self.lock.release() !=20 ! def stop(self):=20 ! "Ask the thread to stop and release, and wait for it to = obey." ! self.keepGoing =3D 0 ! self.join() ! =20 ! class MultiThreadedTests(unittest.TestCase):=20 ! "Tests of multi-threaded locking behaviour." !=20 ! def test_nonblocking_acquire_of_acquired_lock(self):=20 ! "Fail non-blocking acquires of a lock acquired by another = thread" ! dorothy =3D DorothyRLock() ! sitter =3D LockSittingThread(dorothy) ! self.assert_(not dorothy.acquire(0)) ! sitter.stop() !=20 ! def test_foreign_release(self):=20 ! "Catch one thread releasing a lock acquired by another = thread" ! dorothy =3D DorothyRLock() ! sitter =3D LockSittingThread(dorothy) ! self.assertRaises(ReleaseFromForeignThread, = dorothy.release) ! sitter.stop() !=20 ! class IgnoreTests(unittest.TestCase):=20 ! """Tests of CallContext's ignore behaviour as used by = DorothyRLock. ! Also demonstrates the use case for ignores: an instance with a = ! private lock to which it needs to grant controlled access from = ! other objects.""" !=20 ! def setUp(self):=20 ! "Set up the private lock." ! self.__dorothy =3D DorothyRLock(ignores=3D[ ! self.acquire.func_code, self.release.func_code]) !=20 ! def acquire(self): ! "Acquire the private lock." ! self.__dorothy.acquire() !=20 ! def release(self):=20 ! "Release the private lock." ! self.__dorothy.release() !=20 ! def test_ignores(self):=20 ! """Verify instance's acquire and release methods are = ignored=20 ! for the purpose of DorothyRLock's context comparisons.""" ! self.acquire() ! self.release() !=20 ! class LockWhinerTests(unittest.TestCase):=20 ! "Tests of the `LockWhiner`." !=20 ! def test_creation(self):=20 ! "LockWhiner creation" ! dorothy =3D DorothyRLock() ! LockWhiner(dorothy, CallingContext()) !=20 ! def test_fast_creation(self):=20 ! "Fast LockWhiner creation" ! dorothy =3D DorothyRLock() ! LockWhiner(dorothy, CallingContext(), every=3D0.1) !=20 ! def test_startstop(self):=20 ! "LockWhiner start and stop" ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! # whine on behalf of this context into a StringIO object. ! out =3D StringIO.StringIO() ! context =3D CallingContext() ! whiner =3D LockWhiner(dorothy, context, every=3D0.1, = out=3Dout) ! whiner.start() # start it ! whiner.stop() # ask it to stop ! whiner.join(whiner.every + 1) # wait for it to stop = running ! assert not whiner.isAlive() # fail test if we timed out = waiting ! whiner.check() # raise any exceptions caught by = LockWhiner.run !=20 ! def test_whinereports(self):=20 ! "LockWhiner whining" ! dorothy =3D DorothyRLock() ! dorothy.acquire() ! out =3D StringIO.StringIO() ! context =3D CallingContext() ! whiner =3D LockWhiner(dorothy, context, every=3D0.1, = out=3Dout) ! whiner.start() # start it !=20 ! # Now, make sure there's some output.=20 ! time.sleep(0.2) # wait a while ! output1 =3D out.getvalue() ! assert len(output1), "no whining detected" !=20 ! time.sleep(0.2) # wait some more ! output2 =3D out.getvalue() ! assert len(output2)>len(output1), "no additional whining = detected" !=20 ! # See if we can get a full report ! time.sleep(1) ! output3 =3D out.getvalue() ! assert output3.find("additional information") >=3D 0, \ ! "no report detected" !=20 ! whiner.stop() # ask it to stop ! whiner.join(whiner.every + 1) # wait for it to stop = running ! assert not whiner.isAlive() # fail test if we timed out = waiting ! whiner.check() # raise any exceptions caught by = LockWhiner.run =20 ! # Run the tests. ! unittest.main() ------=_NextPart_000_0029_01C4A5A9.CA07A050--