Re: Large lists

Hi George,

On Monday 28 November 2005 05.58, George Karousis wrote:
> Hi to all,
> I'm a new ozone user and I'm trying to understand and solve the problems
> with
> large lists (Balance between RAM cost and time performance).
> I read in faqs that one solution to this problem -in case of  using
> standard java
> collections- is to replace the collection with a linked list whose nodes
> are ozone objects.
>
> Question1 : In case of such a linked list, how many ozone objects can I
> have without
> affecting time performance?

Actually what you put in the list are proxies to ozone objects no the object 
themselves. In a normal collection the whole content is loaded into memory 
upon access which means that your list of proxies will be deserialized from 
disk. I cant say how many you can fit and the performance implications but 
for me less than 100.000 entities works seamlessly without performance 
problems.

>            Question2 : I have a table of 300 million records(you see, I'
> m classic RDBMS developer)
> and each record has a size of 10 bytes. How could I design the objects
> in order to use
> ozone in this case?
300 million are a lot of objects! I think that this is beyond the scope of 
storing ozone proxies in a standard collection. In general the design should 
be focused on implementing the object model rather than a relational data 
model however in this case you have to think about how to access and query 
this structure so that you never have to keep all 300 million objects in 
memory unless you request that on purpose.

>            Request   : Please give me some hints or ideas on how to deal
> with the problems of large
> lists in order to use ozone.
>            Thanks in advance.
>
> George Karousis
I think you need three things that are under development but not done yet:
1. Persistent collections
2. Indexable query system
3. Gammastore

Of these, Persistant collections works but for very large lists such as yours 
you need Gammastore for operations to be performant.

When it comes to indexable query system, you need this to avoid having to 
traverse your entire list of objects if you are looking to select list items 
based on certain criterias but do optimized index scans instead. The quest 
for query system for Ozone has just begun for real and we are looking at / 
discussing various implementation options.

Gammastore is more than half way done but Leo, who has been driving its 
development, has been quiet for some time so I have no idea when it will be 
finished.

If you can contribute some time please help us out with any of these to get 
tem done earlier!

Best regards,
Per


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click