Re: Use generators instead of lists for resultsets?

Hi Jacob

I agree that potential network latency probably makes it undesirable to switch
over completely to using 'fetchone' rather than 'fetchall', and so we will
probably need to have various fetching strategies. You expressed misgivings
about 'getSome' sometimes returning a generator and other times a list. Is
there any reason why it (and the functions it makes use of) can't always return
an iterator, even when 'fetchall' has been used? If this iterator [1] were to
implement a '__len__' method (returning the cursor's rowcount), then it would
be an almost seamless replacement for a list as far as other code was
concerned.

In the 'oracleconn' module I reimplemented 'execute' to use 'fetchone' [2]
rather than use 'fetchall'. But this wasn't so much motivated by an attempt to
reduce memory overhead; rather, Oracle's LOB locators get invalidated if you
don't use them before the next fetch (at least with cx_Oracle). But rather than
always use 'fetchone', it would be better if this was done only when wanted -
whether to reduce memory or because of the use of LOBs. This could be
controlled by an attribute of the associated table class.

So here's what I now propose:

1. Rewrite getSome and auxiliaries to return a ResultSet object that implements
iteration and __len__

2. Define a pydo.PyDO class attribute that controls whether to use fetchone or
fetchall for the associated table. (Leave fetchmany for the time being until
we've thought about it a bit more.)

3. For joins use the lowest ranked (fetchone < fetchall) of the strategies
specified by the constituent tables

[1] The provision of a __len__ method means that it can no longer actually be a
generator.

[2] I actually iterate over the cursor, which in turn uses 'fetchone'.


Hamish




-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click