Re: Async support refactoring

Daniele Varrazzo ha scritto:
> On Fri, Apr 23, 2010 at 9:32 PM, Manlio Perillo
> <[email protected]> wrote:
> 
>> 1) I like the async branch, since it is very generic API.
>>   It can be used on frameworks like Twisted, or using greenlets.
>>
>>   I have only one doubt: can the async API be accessed by C code?
>>   This is rather important for me, since if there is a C API, I can
>>   implement event handling entirely in C, for my WSGI implementation
>>   for Nginx.
> 
> [...]
> 
> If you want to save something and directly call in C e.g. conn_poll()
> probably there is the need to provide a C api as explained in [1].

This is exactly the case :).

> [...]
>
>> 2) I don't like the green branch.
>>   The reason is that the wait_select callback is supposed to block the
>>   current thread.
>>
>>   In case greenlets are used, it will only block the current green
>>   thread.  But the problem is that the API is quite limited, it can not
>>   be used with a normal events API (like Twisted).
> 
> The "green branch" is the friendly name to the coroutine support.
> Twisted is not coroutine bases: it doesn't want to apply to it.
> 
> By the way, I also suspect you are not using enough fantasy.

No.

In order to work with servers where Python application **do not** have
control over the event loop, you need to return control to the event
loop, in order for events to be dispatched.

Instead, with the green branch, the function return only after all the
work has been done.

> A
> callback could consist in putting the fd in a list to be passed later
> to the twisted reactor. 

I'm not sure to follow you, here.

If the callback just puts the fd in a list, and then *return*, how will
the query be executed?

> [...]
>>   Instead, if one want to use greenlets, this can be done on top of the
>>   async branch.
> 
> It can't. Or, at least, it can't be psycopg. Nor it can be a dbapi
> module, which is designed around a blocking idea. The code can't make
> changes to the connection state after a query is sent, because the
> query result arrives after the function has returned. What remains is
> surely useful, but you have to write a program from scratch to work
> around these limitations [3].

Again, I'm not sure to follow you.

Isn't this what you do with the original async branch? As an example
with the suggested implementation for Twisted.

> It is probably good for twisted people
> because they are used to do everything their way, but there is a lot
> of people who may want to deploy Django using a even-driven wsgi web
> server, or use SQLAlchemy, or have a program already written and want
> to move from threads to greenlets (which is what happened to a project
> of ours: psycopg is currently the only piece of the puzzle that still
> forces us to use a thread and stops us to try the promising gevent).
> 

But this can be done by implementing greenlet support in a pure Python
wrapper.

As an example, you can write a custom dialect for SQLAlchemy, or a
custom plugin for Django.

You don't need to have direct support inside psycopg.

> [3] http://initd.org/psycopg/docs/advanced.html#asynchronous-support
> 
>>   The green branch is useful since it will "magically" allow to use the
>>   the normal DBAPI2 implementation in an asynchronous application, but
>>   I usually don't like the idea.
> 
> [...]
> Is there a reason for which psycopg would be better _without_
> coroutine support, which is obviously not useful to a
> non-coroutine-based library?
> 

My point is that you are adding a feature not strictly required in a C
extension module (and that needs to be maintained), and that will not
work with all kind of applications.


Manlio