Re: improving async support in psycopg

On Fri, Mar 26, 2010 at 2:58 PM, Jan Urbański <[email protected]> wrote:
> Daniele Varrazzo wrote:

>> On top of what you've done, I'd like to add a hook on the connection
>> in order to call automatically an user-provided function after execute
>> (i.e. like the `wait_for_query()` in your test case): I'd like to test
>> with it to see if it would be useful to use async psycopg in a "green"
>> environment. It seems easy enough and I'll try to do something in the
>> afternoon.
>
> Great! I already started a Twisted wrapper around it, so we might
> actually get two real usecases and see if the interface is good enough
> for both of them.

I've written a first version of the wait_callback idea: code is
available from the 'async' branch of
http://piro.develer.com/psycopg2.git (gitweb:
https://www.develer.com/gitweb/pub?p=users/piro/psycopg2.git;a=summary).

The connection has a new read/write wait_callback attribute: if set it
should be a callback receiving a cursor argument. The idea is that an
async connection calls the callback at each operation that would
block. If the callback is set to None nothing is invoked, so the
connection is left with the need to be poll()ed. Currently is it
called by execute() and callproc().

I've checked interoperability with the stdlib select() and with the
greenlet-based libraries gevent and eventlet: test cases are included.

The current interface is a proof of concept: it is not complete
because the queries issued during connect() can't take advantage of
the wait_callback. In order to overcome this limitation, I think there
should be a module attribute or function, e.g.
psycopg.extensions.register_wait_callback() so that a callback can be
inherited by every newly created connection. Currently the cb takes a
cursor, which is necessary to invoke poll(), so it wouldn't work e.g.
for connect() where a cursor doesn't exist. Probably the cb should
have a signature f(conn, cur=None) so that poll() can be invoked on
the right object.

I think with this callback automatically called by psycopg, the async
features could be extended to other operations:

- connection initialization
- begin, commit, rollback
- fetch*() for named cursors
- copy?
- lobjects?

Do you think this strategy could be useful?

-- Daniele
_______________________________________________
Psycopg mailing list
[email protected]
http://lists.initd.org/mailman/listinfo/psycopg