Re: AW: DorothyLocker and performance

This is a multi-part message in MIME format.
--------------020506090004080100030308
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable

Hello!

My favority patch:

thread.lock.acquire() returns nothing if called without argument,=20
according to the docs for python 2.2/2.3, even if on win32 it seems to=20
return something.

The attached patch calls thread.lock.acquire(1), which returns true if=20
the acquiration succeded, so the following assert won't fail.

Regards, Thomas

Bauer, Georg schrieb:
> Hi!
>=20
> I now get the following. Looks like the _acquire fails in DorothyLocker=
 -
> but I don't know why. Without DorothyLocker it works fine. Another 2.2
> problem?
>=20
> bye, Georg
>=20
> _acquire:=20
> Exception exceptions.AssertionError:=20
> ('/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py', 1393, '_acquire'=
,
> 'return self.lock.acquire(blocking)')
> ('/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py', 342,
> 'acquire', 'assert result # ... but you can never be too careful.')
> -----------------------
> ReleaseOfUnAcquiredLock
> -----------------------
> An attempt to release an un-acquired lock. Probably caused by failure t=
o
> acquire.
> Detection context:
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/EventsTool.py", line=
 93,
> in _log
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/EventsTool.py", line=
 99,
> in _logVerbose
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py", line 1631,=
 in
> logVerbose
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py", line 1322,=
 in
> thunk
>   File "/opt/pyds/lib/python2.2/threading.py", line 396, in run
>   File "/opt/pyds/lib/python2.2/threading.py", line 408, in __bootstrap
> Exception in thread aggregator:
> Traceback (most recent call last):
>   File "/opt/pyds/lib/python2.2/threading.py", line 408, in __bootstrap
>     self.run()
>   File "/opt/pyds/lib/python2.2/threading.py", line 396, in run
>     apply(self.__target, self.__args, self.__kwargs)
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py", line 1322,=
 in
> thunk
>     tool.logVerbose(_('Background process started'))
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py", line 1631,=
 in
> logVerbose
>     getTool('events')._logVerbose(self.name, msg)
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/EventsTool.py", line=
 99,
> in _logVerbose
>     self._log(tool, message)
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/EventsTool.py", line=
 93,
> in _log
>     finally: self._release()
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py", line 1407,=
 in
> _release
>     return self.lock.release()
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py", l=
ine
> 363, in release
>     self.__sanity_check('release', my_context)
>   File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py", l=
ine
> 290, in __sanity_check
>     raise ReleaseOfUnAcquiredLock, (None, my_context)
> ReleaseOfUnAcquiredLock: (None, <PyDS.DorothyLocker.CallingContext obje=
ct at
> 0x8927614>, 'An attempt to release an un-acquired lock. Probably caused=
 by
> failure to acquire.')
>=20
> bye, Georg
>=20
> -----Urspr=FCngliche Nachricht-----
> Von: Garth T Kidd [mailto:garth-OnzZ1s1DREKDegMON/[email protected]]
> Gesendet: Dienstag, 28. September 2004 22:29
> An: 'Bauer, Georg'; 'Thomas Klaeger'
> Cc: 'PyDS Developer List (E-Mail)'
> Betreff: RE: [Pyds-dev] DorothyLocker and performance
>=20
>=20
> cvs diff -c PyDS\DorothyLocker.py > 20040928-2222-dorothyagain.diff
>=20
> I've updated, backed out the conflict damage (CVS deciding to replace w=
hole
> sections of identical code when only __revision__ had changed), and
> re-generated. Attached.=20
>=20
> I'm beginning to like unit tests. It was quite reassuring to run the mo=
dule
> under 2.2 and see a line of dots followed by OK. I'd otherwise have spe=
nt a
> lot of time stressing that I had some *other* 2.2 compatibility issues
> somewhere. I can't run PyDS up wholesale under 2.2 because all my third
> party libraries are loaded for 2.3...=20
>=20
> -----Original Message-----
> From: pyds-dev-admin-iYtK5bfT9M//Ad8WF/[email protected]
> [mailto:pyds-dev-admin-iYtK5bfT9M//Ad8WF/[email protected]] On Behalf Of Bauer, Georg
> Sent: Wednesday, 29 September 2004 1:02 AM
> To: 'Garth T Kidd'; 'Thomas Klaeger'
> Cc: 'PyDS Developer List (E-Mail)'
> Subject: AW: [Pyds-dev] DorothyLocker and performance
>=20
> Hi!
>=20
> Sorry, but that's not a patch against the current CVS. That patch can't=
 be
> merged because it contains stuff that was already applied or something =
like
> that:
>=20
> bash-2.05a$ patch --dry-run -p0 <20040928-2222-dorothyagain.diff patchi=
ng
> file PyDS/DorothyLocker.py Reversed (or previously applied) patch detec=
ted!
> Assume -R? [n] n Apply anyway? [n] y Hunk #1 FAILED at 1.
> Hunk #2 FAILED at 37.
> 2 out of 2 hunks FAILED -- saving rejects to file PyDS/DorothyLocker.py=
.rej
> bash-2.05a$ patch --dry-run -p0 <20040928-2222-dorothyagain.diff patchi=
ng
> file PyDS/DorothyLocker.py Reversed (or previously applied) patch detec=
ted!
> Assume -R? [n] y Hunk #2 FAILED at 21.
> 1 out of 2 hunks FAILED -- saving rejects to file PyDS/DorothyLocker.py=
.rej
>=20
> bye, Georg
>=20
> -----Urspr=FCngliche Nachricht-----
> Von: Garth T Kidd [mailto:garth-OnzZ1s1DREKDegMON/[email protected]]
> Gesendet: Dienstag, 28. September 2004 14:23
> An: 'Bauer, Georg'; 'Thomas Klaeger'
> Cc: 'PyDS Developer List (E-Mail)'
> Betreff: RE: [Pyds-dev] DorothyLocker and performance
>=20
>=20
> That's a weird one.=20
>=20
> Oh, hang on.
>=20
> E.E..E.........E.E....E
>=20
> Yep. Python 2.2 works differently to Python 2.3. Here's a fix.=20
>=20
> -----Original Message-----
> From: pyds-dev-admin-iYtK5bfT9M//Ad8WF/[email protected]
> [mailto:pyds-dev-admin-iYtK5bfT9M//Ad8WF/[email protected]] On Behalf Of Bauer, Georg
> Sent: Tuesday, 28 September 2004 5:24 PM
> To: 'Garth T Kidd'; Bauer, Georg; 'Thomas Klaeger'
> Cc: PyDS Developer List (E-Mail)
> Subject: AW: [Pyds-dev] DorothyLocker and performance
>=20
> Hi!
>=20
>=20
>>Oh, did I forget to send that one in? Sorry. :| I had that, alternative=
=20
>>logging for those post-shutdown logs, and code to make sure EventsTool=20
>>was last.
>=20
>=20
> Your patch wasn't able to be applied - I wrote you this directly and
> off-list :-)
>=20
> My solution is a bit more flexible, though: you now have a flag to tell
> where to sort the tool on shutdown. I use that to move the PreferencesT=
ool
> directly before EventsTool so that the preferences can be still fetched
> until the very last moment in shutdown.
>=20
>=20
>>You're calling PyDS.Tool.getToolsForShutdown twice, just like you used=20
>>to getTools. Is there anything wrong with saving the value and using it
>=20
> twice?
>=20
>=20
> Changed that, although it really doesn't matter much - the shutdown is =
only
> called once in a running instance ;-)
>=20
>=20
>>Could you drop in this latest version of DorothyLocker? You're now a=20
>>fair way behind, and I've lost track of when I fixed certain bugs. That=
=20
>>might
>=20
>=20
> Did that. But it would really be a bit better if you stick to CVS mecha=
nisms
> - just do an update and merge in the changes I commited and then reprod=
uce
> your patch with regard to current CVS. It's much more likely that I put=
 in a
> working patch than full module sources, as the latter would require me =
to
> make the diff myself to see what changed. And I am known to be lazy ;-)
>=20
> For example this time: after adding the full module I could'nt start Py=
DS,
> because of some changes to the interface of DorothyLocker that required
> additional patches. But your patch didn't work, as your full module alr=
eady
> was installed. Mixing different patch/dist methods allways makes things=
 more
> complicated.
>=20
> Oh, and now the CVS is broken. One thing was a small typo in Tool.py (w=
rong
> indention) that was easily fixed. But now it barfs about "sequence must=
 be
> integer" on start. What's that? It happens in your __getitem__ method i=
n
> your FramingContext where it accesses the framestack (line 103). Looks =
like
> ltrim_common doesn't produce integers as index somehow (maybe pos not
> initialized?). This happens directly at startup of PyDS:
>=20
> Exception exceptions.TypeError: sequence index must be integer
> ('/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py', 1407, '_release'=
,
> 'return self.lock.release()')
> ('/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py', 359,
> 'release', "self.__sanity_check('release', my_context)")
> ('/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py', 260,
> '__sanity_check', 'lock_uniques, my_uniques =3D ltrim_common(lock_conte=
xt,
> my_context)')
> ('/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py', 54,
> 'ltrim_common', 'return oblist1[pos:], oblist2[pos:]')
> ('/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py', 103,
> '__getitem__', 'return self.framestack[index]') Traceback (most recent =
call
> last):
>    File "/opt/pyds/bin/pyds-start", line 51, in ?
>      PyDS.Server.start_server(daemon=3D_PyDS.daemon)
>    File "/opt/pyds/lib/python2.2/site-packages/PyDS/Server.py", line 29=
8, in
> start_server
>      PyDS.Tool.initTools(PyDS.Tool.mainstatus)
>    File "/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py", line 317,=
 in
> initTools
>      tool._initdb()
>    File "/opt/pyds/lib/python2.2/site-packages/PyDS/WeblogTool.py", lin=
e 77,
> in _initdb
>      finally: self._release()
>    File "/opt/pyds/lib/python2.2/site-packages/PyDS/Tool.py", line 1407=
, in
> _release
>      return self.lock.release()
>    File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py",
> line 359, in release
>      self.__sanity_check('release', my_context)
>    File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py",
> line 260, in __sanity_check
>      lock_uniques, my_uniques =3D ltrim_common(lock_context, my_context=
)
>    File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py",
> line 54, in ltrim_common
>      return oblist1[pos:], oblist2[pos:]
>    File "/opt/pyds/lib/python2.2/site-packages/PyDS/DorothyLocker.py",
> line 103, in __getitem__
>      return self.framestack[index]
> TypeError: sequence index must be integer
>=20
>=20
>=20
>>also be behind some of the performance hits you're seeing (I used the=20
>>latest, assuming you'd incorporated it after that submission I made=20
>>that
>=20
> got
>=20
>>blocked for size).
>=20
>=20
> I usually write about what I commited (and at least it's in the CVS RSS
> feed). So if you don't see me mentioning it, it supposedly isn't in CVS=
. I
> either wasn't able to apply it (then I usually write something) or I ju=
st
> plain missed it.
>=20
> bye, Georg
>=20
>=20
> _______________________________________________
> Pyds-dev mailing list
> Pyds-dev-iYtK5bfT9M//Ad8WF/[email protected]
> http://www.westfalen.de/cgi-bin/mailman/listinfo/pyds-dev
>=20
>=20
> _______________________________________________
> Pyds-dev mailing list
> Pyds-dev-iYtK5bfT9M//Ad8WF/[email protected]
> http://www.westfalen.de/cgi-bin/mailman/listinfo/pyds-dev
>=20
>=20

--------------020506090004080100030308
Content-Type: text/plain;
 name="20040929-1329-Dorothy.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="20040929-1329-Dorothy.diff"

Index: PyDS/DorothyLocker.py
===================================================================
RCS file: /pyds/PyDS/PyDS/DorothyLocker.py,v
retrieving revision 1.7
diff -u -r1.7 DorothyLocker.py
--- PyDS/DorothyLocker.py	29 Sep 2004 09:17:56 -0000	1.7
+++ PyDS/DorothyLocker.py	29 Sep 2004 11:27:31 -0000
@@ -336,7 +336,7 @@
             # LockWhiner and then perform a blocking wait. 
             whiner = LockWhiner(self, my_context)
             whiner.start()
-            result = self.__block.acquire()
+            result = self.__block.acquire(1)
         
             # We can ONLY get here if we succeeded...
             assert result # ... but you can never be too careful. 

--------------020506090004080100030308--