Re: improving async support in psycopg

On Wed, Mar 31, 2010 at 1:15 AM, Jan Urbański <[email protected]> wrote:
> On 28/03/10 03:35, Daniele Varrazzo wrote:
>>
>> On Sat, Mar 27, 2010 at 2:40 AM, Jan Urbański<[email protected]>  wrote:
>>
>>> Skimming over the patches I have a question: is it really necessary to
>>> add
>>> the callback feature into the C library? AFAICS the callback is executed
>>> on
>>> asynchronous cursors everytime there is a execute() attempt. Couldn't you
>>> just write a Python psycopg2.extensions.cursor subclass that overrides
>>> execute() by calling that callback?
>>
>> As things are now, execute and callproc being the only methods using
>> async features, this is the case. Probably it will be too extending
>> support to other currently blocking methods (begin/commit/rollback
>> etc).
>
> From the discussion it seems that Federico would prefer to shift the
> responsability of safely using async features to the framerworks instead of
> instrumenting more psycopg methods.
>
> OTOH I saw your post on the gevent mailing list and it indeed would be cool
> if you could change your apps to non-blocking without changing their code,
> by just dropping in the gevent module. I'm not convinced that's fully
> possible, though. I think gevent could provide connection and cursor
> subclasses that would emulate connection.commit() by issuing a COMMIT query
> and waiting (non-blockingly) for it to return. You could probably do the
> same thing for other methods and build up a fully async drop-in replacement
> that you could use with django or whatnot.
>
> I think it makes sense to only expose the raw stuff and let frameworks use
> it the way they see fit.
>
> All that sounds very interesting, please keep posting your results!

Hello, I have been working on the "other way". Couldn't write anything
in the last two days, but what I've done has contributed to clarify
myself things a little bit.

What I've understood is that what I want to do, after all... is not
async support :) In the sense that what I want to do doesn't allow,
for instance, to send 2 async queries and put both them in the same
select() call. Each call is still blocking, and still blocks on its
own. The only difference is that it gives the external environment a
possibility to use their way of blocking. For coroutine frameworks,
this block can yield to another waiting coroutine, enabling a
cooperative switch in the middle of a libpq blocking call. I think it
wouldn't work fine for "normal" system, where there is a single Python
stack (Twisted or an UI event loop): in these systems the call would
simply block.

So I think we are really doing two different things: with your
evolution psycopg will be usable with larger freedom and scheduled to
block whenever the caller decides, at the price of not being usable as
DBAPI adapter anymore, but more as a "low level tool" as described.
With my evolution psycopg would be usable from stackless environments
maintaining compatibility with everything built on top of it, but with
an imposed constraint: the select() is called when a psycopg function
is called, non later at users' will.

My patch is relatively little invasive: it only affects where PQexec
is called, i.e. only pq_execute_command_locked (there are also calls
in conn_setup but can probably be refactored to pass through the above
function) and where PQconnectdb is called. These calls are replaced by
the matching async calls, invoking the "wait hook" in the middle. It
is not very different from what e.g. PQexec does in the libpq (it
calls the async version and then blocks using a poll() or a select()).

My plan at this point is to stop calling what I'm doing "async" :) and
let you build the async path as is best for the frameworks you know
about. What I am really touching is the "sync" code path, adding a
"stackless friendly hook". I've currently implemented the hook call
and the connection, which I think was the trickiest part: as soon as i
can I shall implement the PQexec replacement, refactor the conn_setup
to use it and check that all the tests pass using an user supplied
hook (my target is to keep the full DBAPI semantics).

Thank you very much, have a nice day.

-- Daniele
_______________________________________________
Psycopg mailing list
[email protected]
http://lists.initd.org/mailman/listinfo/psycopg