Re: Use generators instead of lists for resultsets?

On Fri, Jun 24, 2005 at 07:50:08PM +0100, [email protected] wrote:
> I've noticed that PyDO treats a resultset as a list, which for a
> large resultset could incur quite a memory overhead. How about using
> generators instead? Such a change would probably be transparent to
> most of the rest of the code; one of the places where it wouldn't be
> is the type test in getSome.

I've been wondering when someone would bring this up.  For some time
I've considered this option.  I've hesitated, however, because to
actually use less memory, you'd have to call fetchone() rather than
fetchall() on the cursor, which I feared might introduce more network
latency -- perhaps not significant in the ideal case when you have a
fast connection to the database, but nonetheless -- and time is a more
valuable resource than memory, in life and programming.  Whether or
not this concern is imaginary in most situations, I haven't wanted to
switch getSome() to be a generator primarily for this reason, and
secondly because of some squeamishness about the user api change.

I now think that both problems can be resolved by adding an additional
method, called, say, iterSome().  You could then choose whether to use
a generator which would call fetchone() on the cursor, or not.

As it happens, the fetch() function is a already wrapper around
another function, multifetch.iterfetch(), that is a generator, but it
currently calls fetchall(), so the memory hit is there anyway.  I
think this could bear refactoring.  In fact, the dbi execute() method,
which used to be the workhorse, now is a bit pointless (fetch doesn't
use it) and perhaps that should be refactored as well.

Thoughts?

js

-- 
Jacob Smullyan

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFCvGLPuqamFyFXXLIRAoEdAJoCujsKoyQ4gHLXUaOrB0Ulgq7KiwCgqGX2
AIZ/Yj4uQLNWunxhaEwXRmY=
=MVoN
-----END PGP SIGNATURE-----