Re: Async support refactoring

On Fri, Apr 23, 2010 at 9:32 PM, Manlio Perillo
<[email protected]> wrote:

> 1) I like the async branch, since it is very generic API.
>   It can be used on frameworks like Twisted, or using greenlets.
>
>   I have only one doubt: can the async API be accessed by C code?
>   This is rather important for me, since if there is a C API, I can
>   implement event handling entirely in C, for my WSGI implementation
>   for Nginx.

Aren't you mixing two distinct levels? psycopg is a Python wrapper
around the libpq. If you need access from the C level, can't you use
the libpq instead?

If you want to use psycopg to obtain Python object in a C layer you
may access it as regular python object, using PyObject_CallMethod and
such.

If you want to save something and directly call in C e.g. conn_poll()
probably there is the need to provide a C api as explained in [1].
There is a lot of burocracy in doing that, but probably if you provide
a patch Federico wouldn't mind :)

[1] http://docs.python.org/extending/extending.html#providing-a-c-api-for-an-extension-module

> 2) I don't like the green branch.
>   The reason is that the wait_select callback is supposed to block the
>   current thread.
>
>   In case greenlets are used, it will only block the current green
>   thread.  But the problem is that the API is quite limited, it can not
>   be used with a normal events API (like Twisted).

The "green branch" is the friendly name to the coroutine support.
Twisted is not coroutine bases: it doesn't want to apply to it.

By the way, I also suspect you are not using enough fantasy. A
callback could consist in putting the fd in a list to be passed later
to the twisted reactor. You could build coroutines using the python
generators: diesel [2] uses coroutines instead of green threads. Maybe
with some lateral thinking you could work out a way to write a
callback for twisted. I don't know twisted for that but I'm open to
suggestions.

[2] http://dieselweb.org/lib/

>   Instead, if one want to use greenlets, this can be done on top of the
>   async branch.

It can't. Or, at least, it can't be psycopg. Nor it can be a dbapi
module, which is designed around a blocking idea. The code can't make
changes to the connection state after a query is sent, because the
query result arrives after the function has returned. What remains is
surely useful, but you have to write a program from scratch to work
around these limitations [3]. It is probably good for twisted people
because they are used to do everything their way, but there is a lot
of people who may want to deploy Django using a even-driven wsgi web
server, or use SQLAlchemy, or have a program already written and want
to move from threads to greenlets (which is what happened to a project
of ours: psycopg is currently the only piece of the puzzle that still
forces us to use a thread and stops us to try the promising gevent).

[3] http://initd.org/psycopg/docs/advanced.html#asynchronous-support

>   The green branch is useful since it will "magically" allow to use the
>   the normal DBAPI2 implementation in an asynchronous application, but
>   I usually don't like the idea.

Too bad :) The green branch is not very invasive anyway: after the
refactoring following the merge, which was good in itself because
allowed me to clean up the code and iron out a lot of edge cases and
buglets, the green branch uses mostly the same code paths of the sync
connections and the same polling code of the async support.

Is there a reason for which psycopg would be better _without_
coroutine support, which is obviously not useful to a
non-coroutine-based library?

-- Daniele