Re: Lock-ups with multi-threading - draft patch

Richard Davies <[email protected]>
Newsgroups gmane.comp.python.db.psycopg.devel
Message-ID <[email protected]> 
> OK, I've managed to generate the simple test case, which is a big step
> forward!

And here's a proposed patch, which works for me.


I've duplicated the deadlock inside gdb with debug symbols and I attach a
backtrace for my test case in the deadlocked state (attached, running with
Python 2.6.2, Psycopg2 2.0.12 and PostgreSQL 8.1.4).

The cause of the deadlock is now clear, with both threads at various stages
near the end of pq_execute():

Thread 2 is inside conn_notice_process(), trying to claim the
curs->conn->lock at line 67. I believe that this thread holds the Python GIL
(global interpreter lock) so will prevent other threads from running.

Thread 3 is inside pq_fetch() at line 777. It is holding the
curs->conn->lock since line 731. It is running Py_BLOCK_THREADS; and hence
is waiting for the other thread release the GIL.


The bug is that conn_notice_process() in Thread 2 needs to release the GIL
so that Thread 3 can complete and release curs->conn->lock before Thread 2
claims it.


I notice that most Psycopg calls to pthread_mutex_lock/unlock() are wrapped
inside Py_BEGIN/END_ALLOW_THREADS but that conn_notice_process() does not do
this (probably an oversight?).

I attach a patch which to use Py_BEGIN/END_ALLOW_THREADS inside
conn_notice_process() in the same way at the reset. I also found two other
functions which also look like they're missing this and included these in
the patch.

This needs double-checking, in particular for whether Py_BLOCK_THREADS and
Py_UNBLOCK_THREADS should be used inside any of these sections to protect
any non-thread-safe code.

However, it works for me!

Cheers,

Richard.

_______________________________________________
Psycopg mailing list
Psycopg-IAPFreCvJWPBWskQ1e/[email protected]
http://lists.initd.org/mailman/listinfo/psycopg
backtrace-locked-up (text/plain, 6.3 KB) 
(gdb) thread apply all bt

Thread 3 (Thread 0xf717abb0 (LWP 17716)):
#0  0xf7f4e430 in __kernel_vsyscall ()
#1  0xf7dd4b54 in sem_wait@GLIBC_2.0 () from /lib/libpthread.so.0
#2  0xf7f127b8 in ?? () from /home/test/install/lib/libpython2.6.so.1.0
#3  0xf7ed80fc in PyThread_acquire_lock (lock=0x8bc76e8, waitflag=1) at Python/thread_pthread.h:334
#4  0xf7ea1980 in PyEval_RestoreThread (tstate=0x8c440b8) at Python/ceval.c:334
#5  0xf7b2e4fa in pq_fetch (curs=0xf7b9e6bc) at psycopg/pqpath.c:777
#6  0xf7b2ee9d in pq_execute (curs=0xf7b9e6bc, query=0xf7c227b4 "select * from test", async=0) at psycopg/pqpath.c:702
#7  0xf7b338e7 in _psyco_curs_execute (self=0xf7b9e6bc, operation=0xf7c227a0, vars=0x0, async=-512) at psycopg/cursor_type.c:429
#8  0xf7b33cc8 in psyco_curs_execute (self=0xf7b9e6bc, args=0xfffffe00, kwargs=0xfffffe00) at psycopg/cursor_type.c:484
#9  0xf7e49aa1 in PyCFunction_Call (func=0xf7b52d0c, arg=0xf7b52a8c, kw=0x0) at Objects/methodobject.c:85
#10 0xf7ea7a2d in PyEval_EvalFrameEx (f=0x8c6d1c4, throwflag=0) at Python/ceval.c:3706
#11 0xf7ea89d3 in PyEval_EvalFrameEx (f=0x8c6d01c, throwflag=0) at Python/ceval.c:3792
#12 0xf7ea89d3 in PyEval_EvalFrameEx (f=0x8c6cebc, throwflag=0) at Python/ceval.c:3792
#13 0xf7ea94e4 in PyEval_EvalCodeEx (co=0xf7badcc8, globals=0x0, locals=0xfffffe00, args=0x8c6cebc, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2968
#14 0xf7e35f84 in function_call (func=0xf7b46bc4, arg=0xf7b52a6c, kw=0x0) at Objects/funcobject.c:524
#15 0xf7e09e6b in PyObject_Call (func=0xf7b46bc4, arg=0xfffffe00, kw=0xfffffe00) at Objects/abstract.c:2492
#16 0xf7e1dd4e in instancemethod_call (func=0xfffffe00, arg=0xf7b52a6c, kw=0xfffffe00) at Objects/classobject.c:2579
#17 0xf7e09e6b in PyObject_Call (func=0xf7c0fbe4, arg=0xfffffe00, kw=0xfffffe00) at Objects/abstract.c:2492
#18 0xf7ea1dbf in PyEval_CallObjectWithKeywords (func=0xfffffe00, arg=0xf7c4002c, kw=0x0) at Python/ceval.c:3575
#19 0xf7eddb74 in t_bootstrap (boot_raw=0x8c44028) at ./Modules/threadmodule.c:425
#20 0xf7dd03a7 in start_thread () from /lib/libpthread.so.0
#21 0xf7d3d4be in clone () from /lib/libc.so.6

Thread 2 (Thread 0xf797bbb0 (LWP 17715)):
#0  0xf7f4e430 in __kernel_vsyscall ()
#1  0xf7dd52ee in __lll_mutex_lock_wait () from /lib/libpthread.so.0
#2  0xf7dd2099 in _L_mutex_lock_27 () from /lib/libpthread.so.0
#3  0xf797bb90 in ?? ()
#4  0xf7f127b8 in ?? () from /home/test/install/lib/libpython2.6.so.1.0
#5  0xf7ed80fc in PyThread_acquire_lock (lock=0x4533, waitflag=1) at Python/thread_pthread.h:334
#6  0xf7b32835 in conn_notice_process (self=0xf7c5e89c) at psycopg/connection_int.c:67
#7  0xf7b2ee05 in pq_execute (curs=0xf7b9e72c, query=0xf7c227b4 "select * from test", async=0) at psycopg/pqpath.c:696
#8  0xf7b338e7 in _psyco_curs_execute (self=0xf7b9e72c, operation=0xf7c227a0, vars=0x0, async=-512) at psycopg/cursor_type.c:429
#9  0xf7b33cc8 in psyco_curs_execute (self=0xf7b9e72c, args=0xfffffe00, kwargs=0xfffffe00) at psycopg/cursor_type.c:484
#10 0xf7e49aa1 in PyCFunction_Call (func=0xf7b52ccc, arg=0xf7b529ec, kw=0xf7ddaff4) at Objects/methodobject.c:85
#11 0xf7ea7a2d in PyEval_EvalFrameEx (f=0x8c6c90c, throwflag=0) at Python/ceval.c:3706
#12 0xf7ea89d3 in PyEval_EvalFrameEx (f=0x8c6c47c, throwflag=0) at Python/ceval.c:3792
#13 0xf7ea89d3 in PyEval_EvalFrameEx (f=0x8c6c31c, throwflag=0) at Python/ceval.c:3792
#14 0xf7ea94e4 in PyEval_EvalCodeEx (co=0xf7badcc8, globals=0xf7ddaff4, locals=0xfffffe00, args=0x8c6c31c, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2968
#15 0xf7e35f84 in function_call (func=0xf7b46bc4, arg=0xf7c2c4ac, kw=0x0) at Objects/funcobject.c:524
#16 0xf7e09e6b in PyObject_Call (func=0xf7b46bc4, arg=0xfffffe00, kw=0xfffffe00) at Objects/abstract.c:2492
#17 0xf7e1dd4e in instancemethod_call (func=0xfffffe00, arg=0xf7c2c4ac, kw=0xfffffe00) at Objects/classobject.c:2579
#18 0xf7e09e6b in PyObject_Call (func=0xf7bb7f04, arg=0xfffffe00, kw=0xfffffe00) at Objects/abstract.c:2492
#19 0xf7ea1dbf in PyEval_CallObjectWithKeywords (func=0xfffffe00, arg=0xf7c4002c, kw=0x0) at Python/ceval.c:3575
#20 0xf7eddb74 in t_bootstrap (boot_raw=0x8c3f998) at ./Modules/threadmodule.c:425
#21 0xf7dd03a7 in start_thread () from /lib/libpthread.so.0
#22 0xf7d3d4be in clone () from /lib/libc.so.6

Thread 1 (Thread 0xf7c806b0 (LWP 17711)):
#0  0xf7f4e430 in __kernel_vsyscall ()
#1  0xf7dd4b54 in sem_wait@GLIBC_2.0 () from /lib/libpthread.so.0
#2  0xf7f127b8 in ?? () from /home/test/install/lib/libpython2.6.so.1.0
#3  0xf7ed80fc in PyThread_acquire_lock (lock=0x8c44088, waitflag=1) at Python/thread_pthread.h:334
#4  0xf7edd358 in lock_PyThread_acquire_lock (self=0xfffffe00, args=0xfffffe00) at ./Modules/threadmodule.c:46
#5  0xf7e49ae6 in PyCFunction_Call (func=0xf7b52cec, arg=0xf7c4002c, kw=0x0) at Objects/methodobject.c:116
#6  0xf7ea7a2d in PyEval_EvalFrameEx (f=0x8c6c624, throwflag=0) at Python/ceval.c:3706
#7  0xf7ea94e4 in PyEval_EvalCodeEx (co=0xf7ba9ba8, globals=0x0, locals=0xfffffe00, args=0x8c6c624, argcount=1, kws=0x8c124c0, kwcount=0, defs=0xf7bc0cb8, defcount=1, closure=0x0) at Python/ceval.c:2968
#8  0xf7ea7c58 in PyEval_EvalFrameEx (f=0x8c12374, throwflag=0) at Python/ceval.c:3801
#9  0xf7ea94e4 in PyEval_EvalCodeEx (co=0xf7baf020, globals=0x0, locals=0xfffffe00, args=0x8c12374, argcount=1, kws=0x8bc1e78, kwcount=0, defs=0xf7bc0df8, defcount=1, closure=0x0) at Python/ceval.c:2968
#10 0xf7ea7c58 in PyEval_EvalFrameEx (f=0x8bc1d34, throwflag=0) at Python/ceval.c:3801
#11 0xf7ea94e4 in PyEval_EvalCodeEx (co=0xf7bafba8, globals=0x0, locals=0xfffffe00, args=0x8bc1d34, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2968
#12 0xf7e35f84 in function_call (func=0xf7b470d4, arg=0xf7c6ec8c, kw=0x0) at Objects/funcobject.c:524
#13 0xf7e09e6b in PyObject_Call (func=0xf7b470d4, arg=0xfffffe00, kw=0xfffffe00) at Objects/abstract.c:2492
#14 0xf7e1dd4e in instancemethod_call (func=0xfffffe00, arg=0xf7c6ec8c, kw=0xfffffe00) at Objects/classobject.c:2579
#15 0xf7e09e6b in PyObject_Call (func=0xf7c00edc, arg=0xfffffe00, kw=0xfffffe00) at Objects/abstract.c:2492
#16 0xf7e0a1b4 in PyObject_CallMethod (o=0x0, name=0xf7c4002c "$\a", format=0xf7efb4c4 "") at Objects/abstract.c:2524
#17 0xf7edb16b in Py_Main (argc=1, argv=0xffcc6094) at Modules/main.c:243
#18 0x0804866a in main (argc=-512, argv=0xfffffe00) at ./Modules/python.c:23
patch (text/plain, 1.6 KB) 
diff -uNr psycopg2-2.0.12/psycopg/connection_int.c psycopg2-2.0.12.patched/psycopg/connection_int.c
--- psycopg2-2.0.12/psycopg/connection_int.c	2009-08-09 15:09:46.000000000 +0100
+++ psycopg2-2.0.12.patched/psycopg/connection_int.c	2009-09-26 17:54:53.000000000 +0100
@@ -64,6 +64,7 @@
 void
 conn_notice_process(connectionObject *self)
 {
+    Py_BEGIN_ALLOW_THREADS;
     pthread_mutex_lock(&self->lock);
 
     struct connectionObject_notice *notice =  self->notice_pending;
@@ -84,6 +85,7 @@
     }
     
     pthread_mutex_unlock(&self->lock);
+    Py_END_ALLOW_THREADS;
     
     conn_notice_clean(self);  
 }
@@ -91,6 +93,7 @@
 void
 conn_notice_clean(connectionObject *self)
 {
+    Py_BEGIN_ALLOW_THREADS;
     pthread_mutex_lock(&self->lock);
     
     struct connectionObject_notice *tmp, *notice = self->notice_pending;
@@ -105,6 +108,7 @@
     self->notice_pending = NULL;
     
     pthread_mutex_unlock(&self->lock);    
+    Py_END_ALLOW_THREADS;
 }
 
 /* conn_setup - setup and read basic information about the connection */
diff -uNr psycopg2-2.0.12/psycopg/connection_type.c psycopg2-2.0.12.patched/psycopg/connection_type.c
--- psycopg2-2.0.12/psycopg/connection_type.c	2009-08-09 15:08:59.000000000 +0100
+++ psycopg2-2.0.12.patched/psycopg/connection_type.c	2009-09-26 18:03:24.000000000 +0100
@@ -366,9 +366,11 @@
     if (pq_reset(self) < 0)
         return NULL;
 
+    Py_BEGIN_ALLOW_THREADS;
     pthread_mutex_lock(&self->lock);
     res = conn_setup(self, self->pgconn);
     pthread_mutex_unlock(&self->lock);
+    Py_END_ALLOW_THREADS;
     if (res < 0)
         return NULL;
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.