Re: Use generators instead of lists for resultsets?

On Sat, Jun 25, 2005 at 03:07:06AM +0200, Faber wrote:
> On Friday 24 June 2005 21:45, Jacob Smullyan wrote:
> 
> [cut]
> 
> > I now think that both problems can be resolved by adding an additional
> > method, called, say, iterSome().  You could then choose whether to use
> > a generator which would call fetchone() on the cursor, or not.
> 
> I think that there could be better solutions to this problem; for example, 
> adding an optional paramether to the getSome() method that specifies if the 
> result should be a generator or a list (default to list). Moreover, there 
> could be an attribute of PyDO derived classes that specifies the default 
> behaviour.

You are certainly right to question adding a new method and to
consider options for getting the behavior desired out of the api
already in existence, but in the end I'm not convinced by how it would
work, and think a new method would be cleaner.  Both options, the
getSome() return type parameter and the class attribute that
determines the default return type, have some problems which seem
pretty decisive for me. 

First, adding parameters to getSome() would conflict with the fact
that keyword parameters to it are column names, *except* when they are
sql keywords and hence can reasonably be reused to mean other things.
It may not be incredibly likely that someone will have a table with a
column called returntype, but the existence of such a problem invites
some policy decision about it, and my policy decision is not to do it
in the first place if I can avoid it.  Especially since there is
really nothing gained by having people specify an extra keyword
argument rather than call a different method. 

Also, having the return type of a method as essential to an api as
getSome() fluctuate according to a method parameter, or anything else,
feels less clean to me than having the method always return the same
kind of thing.  Methods that vary return type that way have their
place, but I feel are best confined to internal workhorse methods;
admittedly this is a matter of taste.

The unpredictability of return type is also my reason for objecting to
the idea of controlling it by a class attribute.  At least, when the
return type is being passed in explicitly by the method's user, it is
transparent to the caller what they are getting; if you have to check
a class attribute to find out, that is unnecessarily opaque and
inconvenient.  What would happen is that you would either end up
calling list(obj.getSome()) everywhere to normalize the behavior, or
you'd write code that implicitly depended on the customized behavior
of the method for particular objects, which could result in
inscrutibly tight coupling ("why did he call 'list(obj1.getSome())'
here?  He didn't with obj2!").  (Of course, much of the time you'd be
iterating over the return value anyway and it wouldn't matter, but
there would be times it would -- perhaps getSome() is nested in a
generator expression, which can have weird effects, or is being
pickled, etc.)

A weaker objection is that getSome() is such a general method that it
would be hard, at a class level, to say, "most queries of this table
will have huge result sets".  That might be true of a table which
stored bloated Microsoft Word documents as blobs, so that even one row
was a memory hit, but that isn't very typical.  Choosing on a
per-query basis seems more generally useful than on a per-class basis.

Explicit is better than implicit; iterSome() is dead simple and
doesn't have any of these problems, so I like it better. 

Of course, the bottom line is that both mechanisms should be
available, so a PyDO subclass could use either in its own api.  If you
wanted to make all queries against your object yield an iterator, but
only some of them call fetchone(), and all queries to go through one
method, you could do it easily:

   @classmethod
   def query(cls, *args, **kwargs):
       if cls._isLikelyToBeAHugeQuery(args, kwargs):
           meth=cls.iterSome
       else:
           meth=cls.getSome
       for r in meth(*args, **kwargs):
           yield r

The return type would be consistent, however.	

That's my point of view, at any rate.  I am susceptible to persuasion,
given enough time, so don't hesitate to argue if you feel like it!

Cheers,

js

-- 
Jacob Smullyan

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFCvM9euqamFyFXXLIRArSZAJ4j2g++zTDR0ldbG7XR9s0IGA71NQCbBf+f
q54bi8RW1jsDiTYpYRx3QTQ=
=RGFf
-----END PGP SIGNATURE-----