Re: Use generators instead of lists for resultsets?

Before my horrendously long response to this, which no one in their
right mind will read, I should say again that my release plan is to
worry about this and other api enhancements for 2.1; I'm still a step
behind, concerned with assuring that the simple things work correctly.
If anyone has time to run the test suite in their environment, I'd be
very glad to get the results.

On Tue, Jun 28, 2005 at 08:27:14PM +0100, [email protected] wrote:

> I agree that potential network latency probably makes it undesirable
> to switch over completely to using 'fetchone' rather than
> 'fetchall', and so we will probably need to have various fetching
> strategies. You expressed misgivings about 'getSome' sometimes
> returning a generator and other times a list. Is there any reason
> why it (and the functions it makes use of) can't always return an
> iterator, even when 'fetchall' has been used? If this iterator [1]
> were to implement a '__len__' method (returning the cursor's
> rowcount), then it would be an almost seamless replacement for a
> list as far as other code was concerned.

There are operations that you could perform on a list that don't
involve iterating on it:

   cPickle.dump(obj.getSome(), fp)

or

   x=obj.getSome()
   x.append('hoho')

You'd need either to implement a lot of smarts in your ResultSet
iterator, or simply break backwards compatibility and make it clear
that it was now an iterator and you had to call list(x) to get a list.

I'd still prefer to provide just such an iterator method, but call it
something different, such as iterSome(), and having getSome() continue
to return a list -- it might be implemented as list(self.iterSome()).
Not only did I promise not to break backwards API compatibility with
the beta release, but I personally would like better a separate
method, if only because I have going on five years' history with
getSome(), and converting my old and crufty code to PyDO2 will be
difficult enough.

As for whether iterSome(), if you accept that name, should be
configurable to fetch iteratively on a per-class basis, my impression
was that per-query configurability would be more useful, but per-class
configurability would be better than none.

In summary, here is a start at a set of requirements and unsettled
issues:

 1. we need a way to get large result sets from the db in a stream.

 2. Some drivers may provide a means for doing this using dbapi methods like
    fetchone/fetchmany; others may require directly manipulating a sql
    cursor, because they prefetch the entire result set anyway.  It is
    possible that the first set of drivers is empty.

 3. we need an iterator that returns the same data as getSome() (whether 
    it is the same or a different method).

 4. It must be possible for the user to configure when the result sets are
    streamed and when they are fetched en bloc, either per query or
    per class, and preferably both.

Am I leaving out anything we've covered?

Here is a counter-proposal inspired by your implementation ideas:

A thorn is the fact that the getSome() api is rather rigid, in that it
can't be extended with keyword arguments without treading on potential
fields.  One solution would be to introduce an intermediate Query
object that has a getSome() method; the Query could have other
attributes, and PyDO.getSome() would simply use a default Query
object.  I can't say I'm really a fan of using that design pattern
just to get around a limitation of preexisting function signatures,
but this wouldn't be too bad; most of the time you'd never create a
Query object, doing so only when you have a particularly brutal result
set and need to solve a performance problem.  Using such an object
would also enable you to set a per-class default as well as override
it on a per-query basis.  As for how to deal with joins, if they are
being done as PyDO methods that only return one table, as in
joinTable, the table being returned is the one whose query settings
would be applied; in the case of fetch, with multiple tables, you
could get either use a default query object by calling fetch(), or
create a custom one and call .fetch() (or .iterfetch()) on it.
(Implicitly creating a lowest common denominator query obj may be OK,
unless it turns out that such an object can have other properties that
can't be reconciled in that way.)

Cheers,

js  
-- 
Jacob Smullyan

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFCwdN4uqamFyFXXLIRAnhLAJ9v1YEabS197JD9Nu6Fc/yigzUbFQCfZXWj
Wg+SoZUJAuNiy5Gy+L6QgcU=
=koZe
-----END PGP SIGNATURE-----