Re: Large queries
"Tuna Toksöz" <[email protected]> Wed, 23 Jul 2008 15:36:26 +0300
| Newsgroups | gmane.comp.windows.dotnet.nhibernate.devel |
|---|---|
| Message-ID | <[email protected]> |
Not actually N+1 but admit that it is inefficient. How does scrollable result work? Cursors in sql server? On Wed, Jul 23, 2008 at 3:30 PM, Ayende Rahien <[email protected]> wrote: > That is actually the worst way to handle this that I can think of. > You turn all queries into N+1 > > Note that you can use Enumerate() to do just that today. > > > On Wed, Jul 23, 2008 at 2:17 PM, Jan Oravec <[email protected]> wrote: > >> Yep, something like ScrollableResult. I think LINQ provider should use >> it by default (there is no point in keeping any result lists for LINQ), >> like: >> >> foreach (Foo x in session.Linq<Foo> ().Where(...)) >> { >> ... >> } >> >> While thinking more about it, there could be a problem when code inside >> foreach perform other database operations and database driver doesn't >> support multiple queries. Your solution with keeping all IDs would solve >> that, but with huge number of records the amount of memory required can >> be still large. Also, processing must wait for initial query, so latency >> is not decreased. >> >> >> Jan >> >> >> On Wed, 2008-07-23 at 13:37 +0300, Tuna Toksöz wrote: >> > The feature in Hibernate is called "ScrollableResult". I think this is >> > what you need? >> > >> > On Wed, Jul 23, 2008 at 1:20 PM, Tuna Toksöz <[email protected]> >> > wrote: >> > I think there was something similar to this in progress/in >> > plans, but not sure. My idea would be to get all the ids for >> > the database so that you'll have the required when you access >> > them. Then user may specift chunk size, the data is retrieved >> > depending on their ids when required. Data is stored as >> > weakreference, so they can be collected by garbage collector >> > when it runs out of memory. I think this is what db4o >> > currently uses. >> > >> > Regards, >> > >> > >> > On Wed, Jul 23, 2008 at 12:38 PM, Jan Oravec >> > <[email protected]> wrote: >> > >> > >> > Hi, >> > >> > >> > It would be interesting to decrease memory consumption >> > with very large >> > queries (i.e. millions to billions of returned >> > objects), especially when >> > returned data are processed one-by-one and thrown away >> > (and >> > CacheMode.Ignore is used). >> > >> > Currently, ICriteria.List() first loads results into >> > List, which is then >> > returned. First result is not usable by application >> > before all results >> > are loaded, increasing both latency and memory >> > consumption. >> > >> > I propose conversion of all internal code paths using >> > Lists to >> > IEnumerable (either explicitly defining enumerator, or >> > using 'yield >> > return ...'). >> > >> > External APIs should be backward compatible (i.e. >> > ICriteria.Enumerable() >> > would return IEnumerable and ICriteria.List would >> > use .Enumerable() to >> > create List. >> > >> > I expect this change would greatly decrease memory >> > consumption and >> > latency (even for smaller queries), while very >> > slightly increasing >> > required CPU time. Overall single-thread throughput >> > should be also >> > improved, as application could process data while they >> > are still on the >> > way from database server. >> > >> > If this change is acceptable, I would be interested in >> > working on this. >> > >> > >> > Jan >> > >> > >> > >> > >> ------------------------------------------------------------------------- >> > This SF.Net email is sponsored by the Moblin Your Move >> > Developer's challenge >> > Build the coolest Linux based applications with Moblin >> > SDK & win great prizes >> > Grand prize is a trip for two to an Open Source event >> > anywhere in the world >> > >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> > _______________________________________________ >> > Nhibernate-development mailing list >> > [email protected] >> > >> https://lists.sourceforge.net/lists/listinfo/nhibernate-development >> > >> > >> > >> > >> > -- >> > Tuna Toksöz >> > >> > Typos included to enhance the reader's attention ... >> > >> > >> > >> > -- >> > Tuna Toksöz >> > >> > Typos included to enhance the reader's attention ... >> > >> ------------------------------------------------------------------------- >> > This SF.Net email is sponsored by the Moblin Your Move Developer's >> challenge >> > Build the coolest Linux based applications with Moblin SDK & win great >> prizes >> > Grand prize is a trip for two to an Open Source event anywhere in the >> world >> > http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> > _______________________________________________ Nhibernate-development >> mailing list [email protected] >> https://lists.sourceforge.net/lists/listinfo/nhibernate-development >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's >> challenge >> Build the coolest Linux based applications with Moblin SDK & win great >> prizes >> Grand prize is a trip for two to an Open Source event anywhere in the >> world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Nhibernate-development mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/nhibernate-development >> >> > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win great > prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Nhibernate-development mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/nhibernate-development > > -- Tuna Toksöz Typos included to enhance the reader's attention ... ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Nhibernate-development mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nhibernate-development