Re: Multi-threaded Remote Calls vs. Python imterpreter Lock

Please ignore my previous email! Apparently I was wrong. (Not the first
time :-)) My third thread eventually got blocked regardless of what
cursor factory was used. Also, the cursors were serialized as James
suggested. 

So the issue again is why the python interpreter lock is not released
when a second thread attempts a psycopg call on a connection that is
already busy?

Thanks, 
Jozsef


-----Original Message-----
From: Jozsef Szalay 
Sent: Saturday, March 15, 2008 7:04 PM
To: 'James Henstridge'
Cc: psycopg-IAPFreCvJWPBWskQ1e/[email protected]
Subject: RE: [Psycopg] Multi-threaded Remote Calls vs. Python
imterpreter Lock

Further investigation revealed that the command my code was using to
create the cursors was incorrectly stated in my email. The code was
actually using the default factory to get the cursors:

cursor = con.cursor()
cursor2 = con.cursor()

The rest of my findings were correct. While the two cursors were busy
waiting for the Postgres backend, the third thread (RunningCrazyThread),
which did not have anything to do with psycopg or Postgres, was blocked.
My assumption is that the python interpreter lock was not released. What
I would have liked to see was that the third thread would be running. 

Now the interesting part... When I changed the cursor commands to 

cursor = con.cursor(cursor_factory = psycopg2.extras.DictCursor)
cursor2 = con.cursor(cursor_factory = psycopg2.extras.DictCursor)

I found, to my delight, that the third thread was not blocked but was
happily running. It appears that the cursor with the DictCursor factory
behaves differently (and correctly IMHO) than the cursor that was
created with the default factory.

> Note that all cursors for a connection are sharing the database
> connection, and execute queries in series

James, I don't believe this statement is true. According to the Linux
"top" command, the two Postgres processes that served the DictCursor
cursors were both running concurrently. 
 
It appears that if, within a single connection, you create your cursors
with the DictCursor factory you can run queries simultaneously with
these cursors without blocking unrelated python threads. 

Let me know if you'd like a simple test case that would support my
findings!

Thanks for the help!
Jozsef