Re: improving async support in psycopg

Daniele Varrazzo wrote:
> On Fri, Mar 26, 2010 at 3:20 AM, Jan Urbański <[email protected]> wrote:

Hi,

I noticed that you didn't CC: the list - is that intentional? I will CC:
the list on reply, I hope you don't mind.

> A few observations reading your patches:

> - is it necessary to expose all the states in the connection building,
> (CONN_STATUS_{SEND|SENT|GET}_DATESTYLE etc)? I suspect they are an
> implementation detail of the current psycopg version and I don't know
> if it would be safe to build Python code on top of them.

Oh, you're right, no need to expose those to Python. The connection
poll() method returns the same values as the cursor poll() method
(meaning psycopg2.extensions.POLL_{READ, WRITE, OK}) and only these need
to be exposed. Thanks for pointing that out, will fix.

> - it seems the async parameter on `cur.execute()` is unneeded now:
> what about dropping it? I don't think there is any production code
> using it (I may be wrong, in this case it is just to be pointed out in
> the docs).

Yes, the parameter on the cursor is now redundant, and actually there's
code that when you try to execute asynchronously from a sync connection
(or the other way around) will just raise an exception.
I left it for backwards compat, and to not have to touch the cursor
subclasses from psycopg2.extras. I'm afraid people might have cursor
subclasses that chain up to the superclass execute() method passing the
async parameter and this would break them. But of course I'm also fine
with just dropping it.

>> Documentation patches are missing from that repository, I'll wait with
>> learning Sphinx until you guys give me some feedback on this ;) But of
>> course if the feature would get accepted, I'll update the docs as well.
>
> Don't worry for the docs: I'll be happy to fix them as soon as the
> interface is settled.

Thanks, that would be awesome.

> On top of what you've done, I'd like to add a hook on the connection
> in order to call automatically an user-provided function after execute
> (i.e. like the `wait_for_query()` in your test case): I'd like to test
> with it to see if it would be useful to use async psycopg in a "green"
> environment. It seems easy enough and I'll try to do something in the
> afternoon.

Great! I already started a Twisted wrapper around it, so we might
actually get two real usecases and see if the interface is good enough
for both of them.

> Another question: how do you think async connections and named cursors
> mix? Basically in named cursors the fetch* methods are blocking too.
> Does your patch already address them?

Yes, it raises an exception when you try to create a named cursor from
an async connection :o)

I outlined a possible approach for named cursor in a different email,
but basically I think it might not be worth it. The problem is that the
cursor gets declared when you do execute() (and this would work
asynchronously), but then when you do fetch() it actually issues another
query (FETCH FROM) and this means that the fetch() method would have to
be async... I think it could be solved by adding yet another method that
would mediate between execute() and fetch(), which I proposed to call
load().

You can always issue your DECLARE CURSOR and FETCH FROM calls yourself.
For instance, to really use async connections you will have to turn off
the psycopg2 transaction management and issue BEGIN and COMMIT yourself,
because currently they both are blocking.

Cheers,
Jan

_______________________________________________
Psycopg mailing list
[email protected]
http://lists.initd.org/mailman/listinfo/psycopg