Re: Re: Use generators instead of lists for resultsets?

> Yes, that seems to be true.  In fact, if my cursory reading of libpq
> is accurate, then even pygresql, which constructs rows on demand (by
> calling PQgetvalue) is really prefetching, too, because the rows are
> already in the result set object after PQexec (I think).

	I believe the Postgres driver fetches the entire result set.
	I think, maybe, that mysql streams results, but I'm not sure at all.

	Also you should really really use psycopg instead of pygresql (maybe you  
alreasy do ?), it's something like 10x faster and the version 2 which is  
in beta now has amazing new features.

> If I've understood this rightly, then an iterSome() method might
> itself have to use FETCH (for drivers that prefetch results and
> support server-side cursors) to significantly improve memory
> consumption for very large result sets.  Would that be your
> conclusion?

	I agree with that.
	Also FETCH should be called with generous packet sizes like 100 or 1000  
to amortize the call, FETCH 1 is ridiculously slow, so it's the ORM's job  
to FETCH packets and then yield the results one by one.

> Incidentally, I'm for deferring implementing iterSome() (or whatever
> arises from these discussions) until after 2.0 final.  There are
> several things I need to learn and think about to do it with the
> proper context, and I think tests for existing functionality are more
> important right now.

	I think you're right.
	Keep up !

	PF Caillaud


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click