Re: Large queries

Yep, something like ScrollableResult. I think LINQ provider should use
it by default (there is no point in keeping any result lists for LINQ),
like:

foreach (Foo x in session.Linq<Foo> ().Where(...))
{
  ...
}

While thinking more about it, there could be a problem when code inside
foreach perform other database operations and database driver doesn't
support multiple queries. Your solution with keeping all IDs would solve
that, but with huge number of records the amount of memory required can
be still large. Also, processing must wait for initial query, so latency
is not decreased.


Jan


On Wed, 2008-07-23 at 13:37 +0300, Tuna Toksöz wrote:
> The feature in Hibernate is called "ScrollableResult". I think this is
> what you need?
> 
> On Wed, Jul 23, 2008 at 1:20 PM, Tuna Toksöz <[email protected]>
> wrote:
>         I think there was something similar to this in progress/in
>         plans, but not sure. My idea would be to get all the ids for
>         the database so that you'll have the required when you access
>         them. Then user may specift chunk size, the data is retrieved
>         depending on their ids when required. Data is stored as
>         weakreference, so they can be collected by garbage collector
>         when it runs out of memory. I think this is what db4o
>         currently uses.
>         
>         Regards,
>         
>         
>         On Wed, Jul 23, 2008 at 12:38 PM, Jan Oravec
>         <[email protected]> wrote:
>         
>                 
>                 Hi,
>                 
>                 
>                 It would be interesting to decrease memory consumption
>                 with very large
>                 queries (i.e. millions to billions of returned
>                 objects), especially when
>                 returned data are processed one-by-one and thrown away
>                 (and
>                 CacheMode.Ignore is used).
>                 
>                 Currently, ICriteria.List() first loads results into
>                 List, which is then
>                 returned. First result is not usable by application
>                 before all results
>                 are loaded, increasing both latency and memory
>                 consumption.
>                 
>                 I propose conversion of all internal code paths using
>                 Lists to
>                 IEnumerable (either explicitly defining enumerator, or
>                 using 'yield
>                 return ...').
>                 
>                 External APIs should be backward compatible (i.e.
>                 ICriteria.Enumerable()
>                 would return IEnumerable and ICriteria.List would
>                 use .Enumerable() to
>                 create List.
>                 
>                 I expect this change would greatly decrease memory
>                 consumption and
>                 latency (even for smaller queries), while very
>                 slightly increasing
>                 required CPU time. Overall single-thread throughput
>                 should be also
>                 improved, as application could process data while they
>                 are still on the
>                 way from database server.
>                 
>                 If this change is acceptable, I would be interested in
>                 working on this.
>                 
>                 
>                 Jan
>                 
>                 
>                 
>                 -------------------------------------------------------------------------
>                 This SF.Net email is sponsored by the Moblin Your Move
>                 Developer's challenge
>                 Build the coolest Linux based applications with Moblin
>                 SDK & win great prizes
>                 Grand prize is a trip for two to an Open Source event
>                 anywhere in the world
>                 http://moblin-contest.org/redirect.php?banner_id=100&url=/
>                 _______________________________________________
>                 Nhibernate-development mailing list
>                 [email protected]
>                 https://lists.sourceforge.net/lists/listinfo/nhibernate-development
>                 
>         
>         
>         
>         -- 
>         Tuna Toksöz
>         
>         Typos included to enhance the reader's attention ...
> 
> 
> 
> -- 
> Tuna Toksöz
> 
> Typos included to enhance the reader's attention ...
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________ Nhibernate-development mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nhibernate-development

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
Nhibernate-development mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nhibernate-development

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEABECAAYFAkiHE18ACgkQQy/4lyhWjrWF1QCdHbv9IkCbm0Bx2O0gcSEC/9eE
gsQAn1TyOG427hzo+R8+Koul4+tZ0pDY
=6DGQ
-----END PGP SIGNATURE-----