New submission from Vultaire <None>:
Hello,
I'm working on an app which heavily interfaces with MySQL. During performance testing and profiling, I found that the program spent a lot of time at or below the adbapi function calls.
After running this program for about 2 hours and 13 minutes, I terminated it and collected the profiling data. I also processed the output and put it through GraphVis. A portion of that graph is attached to this post.
It seems that according to this test run, it spent nearly 25% of its time, over 33 minutes, sitting and waiting on standard library threading.Lock.acquire calls.
Digging a little deeper, it seems that this happens due to the ThreadPool object on callInThreadWithCallback. callInThreadWithCallback calls Queue.put and pushes an object into the queue, which worker threads are already present and waiting to receive. One of the workers grabs the queue's lock and grabs the object, while simultaneously, callInThreadWithCallback calls _startSomeWorkers, calling Queue.qsize which then blocks the reactor while waiting for the lock to release.
...Or at least that's what I can gather. I've only been professionally coding in Python for half a year, so skepticism may be in order.
Attached is a tarball with profiling data gathered via cProfile (trimmed and with proprietary files censored), plus the previously mentioned GraphVis chart.
Thanks for any feedback.
----------
Type : defect
Component: core
Keywords :
Priority : normal
Nosy :
----------
http://twistedmatrix.com/trac/ticket/3836
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.