Re: Patterns for large resultsets of repositories - ideas?

On Fri, Sep 6, 2013 at 1:04 AM, Michael Brown <[email protected]> wrote:
> Most languages/environments have solved this problem. The Repository just acts as an abstraction over the given solution. Unless you have abnormally wide records, you can handle millions of records without much problem in memory on most desktop environments.

interfacing with magento using moose and perl, about 2000 results on a
product list resulted in about 2G of memory used, given some of this
was the XML payoad, some of this was the crappy Pure Perl Parser, and
some of this was me aggressively creating Moose objects from the
parsed structures so a double copy was in memory at once. The gist is
I do not agree.


>  From an OO point of view, what would be the best option to implement large resultsets of repositories, for instance a few hundred objects returned by a search operation? I want to implement some kind of iterator pattern to fetch entities one by one. This should avoid an out of memory problem, but will also introduce side effects with unexpected behavior, like an open and probably not safely closed database cursor.
> Introducing a "close" method on the iterator would also introduce some kind of leakage into the API, and the caller must be aware of closing the iterator in every case. An other option would be to fully iterate over the whole resultset, and the iterator implementation will free resources after the last entry was fetched. But the caller might not want to iterate the whole resultset. The last option would be to use a destructor/finalizer to free up resources, but in a managed environment, their execution cannot be predicted, so we would again have some strange side effects as the worst case.
>
> Any ideas?

My (untested thought) is an iterator that will have a fetch with a
limit, so you get, say, 50 records at once (which would be far
healthier for a SQL database, than getting one at a time. Once your
iterator runs out of in memory records it can do another fetch for the
next page. This allows you to iterate infinitely, without pounding
your memory or your db, I would make the limit and the current page
(offset) iterator constructor configurable (meaning you can pass them
as params optionally)

I'm not sure what you mean by close? you mean the DB connection? the
correct answer there is to have a pool of DB Connections that you
inject into your Repository/Mapper/ActiveRecord/... it's ok to have
many connections open concurrently.

-- 
Caleb Cushing

http://xenoterracide.com

Calendar:
https://www.google.com/calendar/embed?src=xenoterracide%40gmail.com&ctz=America/Chicago


------------------------------------