Re: (long) improving async support in psycopg

On 24/03/2010 04:38, Jan Urbański wrote:
[snip]
> Another thing that I'm wondering about: why is the fileno() method doing
> PQflush? The comment says it's to make sure the user uses select()
> safely, but AFAICS it does not really do that. First of all one PQflush
> might not be enough to empty the write buffer and second of all, getting
> the file descriptor should not attempt to flush the outgoing buffer
> queue. Instead, the issent() method would be used for that.

The main idea was to avoid the need of issent(). Unfortunately a single
PQflush() call does not guarantee that the outgoing buffer if empty so
the call is not safe. I don't like the need to two methods (issent() and
isready()) and two loops just to one single async query.

> Next question is about pq_fetch and the loop where it does pq_is_busy
> and uses select() internally. Is this loop necessary at all? If the
> execute() call on the cursor was synchronous, there's no need to check
> the status of the connection, or to do a select. PQexec would block
> until all data has been transferred.

Right.

> If the execute() call was asynchronous, then a client should enter a
> isready() loop doing select() on his own, and eventually get all the
> result, so that loop would also not be necessary. I think it could just
> be dropped, maybe after making sure that pending NOTIFY messages are
> processed.

I agree. We can try to simplify the loop and see if anything breaks.

> That brings me back to the issue of doing a fetch() on an async cursor
> that has not received all data yet. The current code blocks until it
> gets all data, but I think that an easier solution would be to just
> error out if there's an async cursor that has not consumed all input
> yet. Anyone using async cursors would implement that select() loop
> anyway, so he will consume all input. That would make it possible to
> drop some things like needsfetch from the cursor structure and simplify
> the code around that quite a bit.

But then what happens to the connection? If we just throw an error then
we need to block and flush the buffers anyway because an important
assumption in all psycopg client code is that unless the connection is
broken you can recover from _any_ error by issuing a rollback() and then
you can start using the connection again. So, if we need to block/flush
buffers before raising the exception it does make sense to just use that
information, i.e., transform the fetchXXX() call into a blocking call
and let the user get its data.

> Another issue are server-side asynchronous cursors. ATM I don't see any
> sane way to support them, because after creating a named async cursor
> and executing something asynchronously with it, using isready() to loop
> will just make you wait for the DECLARE CURSOR to return. After that the
> only way to actually do the query is to call fetch*(), which will block.
> The problem is that the fetch*() method has to return the result, so
> it's too late to ask the user to use select() after that. The only thing
> I came up was making async server-side cursors throw errors for calls to
> fetch() that are not preceded by a call to a new method called load(n),
> that will issue the FETCH query and will make it possible to use
> isready() to wait for the completion. After that fetch() would have the
> data already on the client side and could throw errors if you try to
> access data that has not been loaded yet.

While I don't like to add yet another method I don't see an easy
solution with the current API.

> I won't bother you with the details of how an async connection building
> procedure could look like, I just want to raise one last issue. The
> connection class issues a couple of queries, like SHOW client_encoding
> or SHOW default_transaction_isolation. These are issues synchronously
> and there is no control over them. That would even be OK, hopefully an
> application would not create and close lots of connection during it's
> lifetime. The bad thing is the BEGIN query that gets issued
> synchronously and even less convenient is that COMMITs are synchronous,
> and these could take some time on systems with a certain configuration.
> 
> As you can see there are quite a few things to look at to be able to use
> psycopg2 as a fully async library (remember that my use case is Twisted,
> so every call that can potentially block is bad). I'm beginning to
> wonder if it's actually feasible...

True. I suppose we can have a flag on the connection saying "do
everything as async" so that even transaction control can be made
asynchronous but that will take a lot of work.

federico

-- 
Federico Di Gregorio                                       fog-NGVKUo/i/[email protected]
 In science one tries to tell people, in such a way as to be understood
  by everyone, something that no one ever knew before. But in poetry,
  it's the exact opposite.                                 -- Paul Dirac

_______________________________________________
Psycopg mailing list
Psycopg-IAPFreCvJWPBWskQ1e/[email protected]
http://lists.initd.org/mailman/listinfo/psycopg

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkuqekkACgkQvcCgrgZGjevv3ACdHmMU0u9iFHiCvGoOogVmqQwt
VlAAn0IrE1Gq9immQMvxCvkEThcIvc5P
=F/rn
-----END PGP SIGNATURE-----