Re: Large queries

I think there was something similar to this in progress/in plans, but not
sure. My idea would be to get all the ids for the database so that you'll
have the required when you access them. Then user may specift chunk size,
the data is retrieved depending on their ids when required. Data is stored
as weakreference, so they can be collected by garbage collector when it runs
out of memory. I think this is what db4o currently uses.

Regards,

On Wed, Jul 23, 2008 at 12:38 PM, Jan Oravec <[email protected]> wrote:

> Hi,
>
>
> It would be interesting to decrease memory consumption with very large
> queries (i.e. millions to billions of returned objects), especially when
> returned data are processed one-by-one and thrown away (and
> CacheMode.Ignore is used).
>
> Currently, ICriteria.List() first loads results into List, which is then
> returned. First result is not usable by application before all results
> are loaded, increasing both latency and memory consumption.
>
> I propose conversion of all internal code paths using Lists to
> IEnumerable (either explicitly defining enumerator, or using 'yield
> return ...').
>
> External APIs should be backward compatible (i.e. ICriteria.Enumerable()
> would return IEnumerable and ICriteria.List would use .Enumerable() to
> create List.
>
> I expect this change would greatly decrease memory consumption and
> latency (even for smaller queries), while very slightly increasing
> required CPU time. Overall single-thread throughput should be also
> improved, as application could process data while they are still on the
> way from database server.
>
> If this change is acceptable, I would be interested in working on this.
>
>
> Jan
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Nhibernate-development mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nhibernate-development
>
>


-- 
Tuna Toksöz

Typos included to enhance the reader's attention ...

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
Nhibernate-development mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nhibernate-development