[pysqlite] Releasing GIL around sqlite3_prepare

[email protected] Fri, 22 Aug 2008 15:27:26 -0700

Newsgroups	gmane.comp.python.db.pysqlite.user
Message-ID	<[email protected]>

Hi,

I noticed that not all calls to sqlite3_prepare() in pysqlite 2.3.2
(which is the version that ships with both Python 2.5.1 and 2.5.2)
are wrapped with Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS.

This can cause spurious timeouts ("OperationalError: database is
locked", because if sqlite3_prepare() finds the database locked by
another Python thread, it will invoke the sqlite busy handler, sleep,
and try again -- not releasing the GIL. The thread that owns the lock
will not be able to acquire the GIL, so it will not have a chance to
finish what it's doing and release the database lock so that the first
thread could make progress.

Is there a reason for this? Would the attached patch be sufficient
to fix it? Can anyone think of other sqlite3 API calls that can block,
that I should check to make sure pysqlite does not call them without
releasing the GIL?

(this patch is against the Python source distribution -- let me know
if I should make a patch against pysqlite-2.4.1 instead)

Thanks!

	~Ben

_______________________________________________
list-pysqlite mailing list
list-pysqlite-FR6EJeJVuqdwc357pe9rcyQmJico6nz3epZhswDD4dQ@public.gmane.org
http://itsystementwicklung.de/cgi-bin/mailman/listinfo/list-pysqlite

(unnamed) (text/x-patch, 1.6 KB)

diff -ru old/Modules/_sqlite/cursor.c new/Modules/_sqlite/cursor.c
--- old/Modules/_sqlite/cursor.c	2006-07-28 11:36:01.000000000 -0700
+++ new/Modules/_sqlite/cursor.c	2008-08-22 13:31:41.000000000 -0700
@@ -772,11 +772,13 @@
         }
         statement_completed = 1;
 
+        Py_BEGIN_ALLOW_THREADS
         rc = sqlite3_prepare(self->connection->db,
                              script_cstr,
                              -1,
                              &statement,
                              &script_cstr);
+        Py_END_ALLOW_THREADS
         if (rc != SQLITE_OK) {
             _seterror(self->connection->db);
             goto error;
diff -ru old/Modules/_sqlite/statement.c new/Modules/_sqlite/statement.c
--- old/Modules/_sqlite/statement.c	2006-04-23 08:24:26.000000000 -0700
+++ new/Modules/_sqlite/statement.c	2008-08-22 13:32:39.000000000 -0700
@@ -69,11 +69,13 @@
 
     sql_cstr = PyString_AsString(sql_str);
 
+    Py_BEGIN_ALLOW_THREADS
     rc = sqlite3_prepare(connection->db,
                          sql_cstr,
                          -1,
                          &self->st,
                          &tail);
+    Py_END_ALLOW_THREADS
 
     self->db = connection->db;
 
@@ -224,11 +226,13 @@
 
     sql_cstr = PyString_AsString(self->sql);
 
+    Py_BEGIN_ALLOW_THREADS
     rc = sqlite3_prepare(self->db,
                          sql_cstr,
                          -1,
                          &new_st,
                          &tail);
+    Py_END_ALLOW_THREADS
 
     if (rc == SQLITE_OK) {
         /* The efficient sqlite3_transfer_bindings is only available in SQLite