Re: all python threads hung

On Fri, 07 Sep 2007 00:38:46 -0400, dcharno <[email protected]> wrote:
> I have a number of threads which collect and queue data.  A separate
> thread inserts the queued data in to an sqlite database.  Another thread
> services network requests for statics from the database.
> 
> The problem I have is long periods of time where all threads in the app
> appear to be hung. I've found that the duration of the 'stall' is always
> a multiple of the pysqlite timeout I've set -- typically 2x or 3x.  My
> timeout was set to 20seconds, but none of the database transactions are
> longer than 200ms.
> 
> At most, I'd expect my two threads to hang.  I'm not understanding how
> all the threads in the system could be effected.  Any thoughts would be
> greatly appreciated.

Are you using the latest version of pysqlite? 2.3.5 has improved concurrency a little.

Of course pysqlite does release the GIL (*) when calling the SQLite API, so that can hardly be the cause for your problem. One writer, multiple readers sounds like it should not be a problem, though.

In the past I've experienced that schema changes (any, even CREATE TEMPORARY VIEW) can cause problems for other database connections. They'll then get schema errors and will need to be recompiled (pysqlite does that transparently).

-- Gerhard

(*) The GIL is released for nontrivial calls, like compiling queries and fetching results. Aquiring and releasing locks is a small overhead as well, so I removed some GIL releases for trivial SQLite API calls for the last release.