Re: improving async support in psycopg

On Fri, Mar 26, 2010 at 7:29 PM, Daniele Varrazzo
<[email protected]> wrote:

> I've written a first version of the wait_callback idea: code is
> available from the 'async' branch of
> http://piro.develer.com/psycopg2.git (gitweb:
> https://www.develer.com/gitweb/pub?p=users/piro/psycopg2.git;a=summary).

> [...] Probably the cb should
> have a signature f(conn, cur=None) so that poll() can be invoked on
> the right object.

I updated the above branch with definition of the wait callback as
f(conn, cur=None).

Here is a testing script I wrote to check that the async feature works
as expected (using gevent). Here, after connect(), the callback is
used manually to put the connection in the correct state: as discussed
a better interface for this operation can be provided.

# make urllib2 coroutine-friendly
import gevent
import gevent.monkey
gevent.monkey.patch_all()
import urllib2  # green

# have an async connection to play well with gevent
import psycopg2
from test_async_gevent import gevent_wait_callback
conn = psycopg2.connect("dbname=postgres", async=1)
gevent_wait_callback(conn)
conn.wait_callback = gevent_wait_callback

import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
logger = logging.getLogger()

def download(num, secs):
    url = "http://localhost:8000/sleep/%d/" % secs
    for i in range(num):
        logger.info("download %d start", i)
        data = urllib2.urlopen(url).read()
        logger.info("download %d end", i)

def fetch(num, secs):
    cur = conn.cursor()
    for i in range(num):
        logger.info("fetch %d start", i)
        cur.execute("select pg_sleep(%s)", (secs,))
        logger.info("fetch %d end", i)

# two concurrent tasks: one downloading 2 urls, each taking 3 secs,
# the other running 3 queries, each blocking for 2 secs
logger.info("making jobs")
jobs = [
    gevent.spawn(download, 2, 3),
    gevent.spawn(fetch, 3, 2),
    ]

logger.info("join begin")
gevent.joinall(jobs)
logger.info("join end")


The output of the script is what expected:

$ python nonblock.py
2010-03-27 02:01:02,004 making jobs
2010-03-27 02:01:02,022 join begin
2010-03-27 02:01:02,031 download 0 start
2010-03-27 02:01:02,046 fetch 0 start
2010-03-27 02:01:04,106 fetch 0 end
2010-03-27 02:01:04,107 fetch 1 start
2010-03-27 02:01:05,203 download 0 end
2010-03-27 02:01:05,204 download 1 start
2010-03-27 02:01:06,110 fetch 1 end
2010-03-27 02:01:06,111 fetch 2 start
2010-03-27 02:01:08,115 fetch 2 end
2010-03-27 02:01:08,381 download 1 end
2010-03-27 02:01:08,383 join end


-- Daniele