Re: Re: Use generators instead of lists for resultsets?

On Sun, Jun 26, 2005 at 06:25:40PM +0200, PFC wrote:
> 	- some database drivers fetch the entire result set at once anyway  
> (postgres for instance) because they rightfully consider that you're using  
> a CURSOR anyway if you want to fetch a lot of stuff.

Yes, that seems to be true.  In fact, if my cursory reading of libpq
is accurate, then even pygresql, which constructs rows on demand (by
calling PQgetvalue) is really prefetching, too, because the rows are
already in the result set object after PQexec (I think).

> 	- delaying the processing on the client side will use precious 
> 	database  server resources for a longer time.

That would be true if the database server actually is delaying
returning the entire result set; but as you've just pointed out, at
least that isn't the case for the postgresql driver we're using.  It
would be true if FETCH were being used, presumably.

If I've understood this rightly, then an iterSome() method might
itself have to use FETCH (for drivers that prefetch results and
support server-side cursors) to significantly improve memory
consumption for very large result sets.  Would that be your
conclusion?

Incidentally, I'm for deferring implementing iterSome() (or whatever
arises from these discussions) until after 2.0 final.  There are
several things I need to learn and think about to do it with the
proper context, and I think tests for existing functionality are more
important right now.

Cheers,

js




-- 
Jacob Smullyan

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFCv1WHuqamFyFXXLIRAohTAJ9WHa9jIMgllJT0X4e/e53W3iUS/wCffJ3Q
JaTfC390/od3dtnJ5/80lgk=
=UJ9S
-----END PGP SIGNATURE-----