Re: Performance/Scalability

I recently asked the same question. Here is a copy of Leo's response:

On Wed, 2004-05-05 at 23:01, Kurt Marasco wrote:
> Below is a small test application, QkbApp.java, that is the beginnings
> of a questionnaire system. Also included are two Ozone persistent
> classes (I did not include the interfaces), ResponseImpl and
> ResponseSetImpl.    The newResponse() method has a for loop that
> allows multiple responses to be added to a ResponseSet collection. If
> the number of new items added here is increased, performance
> dramatically decreases. Processor utilization pegs at 100%, but memory
> utilization remains low.
> 
> Just for fun, I tried to create a million records. Four days later I
> killed the process. I had about 2.5 MB written to disk at that point.
> This has to be the result of some simple stupid mistake on my part.
> Please help to point out my error that is resulting in this
> performance issue. 

It is not such much of an error, as well as a direct result of _where_
you are creating new database objects.

> Also, when I killed the million record process, many responses were
> persisted to disk. Should the records not have been reversed since the
> transaction was never completed? 

There is no 'the transaction', there have been a lot of transactions.
Each iteration implicitly starts a new transaction. To have only 1
transaction (and a hell of a lot better performance) try to create your
objects server side, i.e. create something like an ObjectCreatorImpl or
something, and put the 'million objects iteration' in one of its
methods. Then call that method from a client.

Also, please note that a using a java.util.HashMap inside an ozone
object is not very scalable; every time an ozone object with such a
HashMap is changed, it has to be persisted to disk at least one time for
every transaction, which means serializing the whole HashMap... If you
use org.ozoneDB.collections.NodeTreeMap from ozoneCollections*.jar, you
avoid these scalability issues. There is no HashMap-like ozone
counterpart, because of how a hashtable works inside.

> BTW, the machine is a 1Ghz Duron, with 1GB ram.

I'm mostly developing on a 1.3GHz Thunderbird with 1GB, so no problems
there.

Cheers,
Leo

On Sat, 2004-05-08 at 22:08, David Mirly wrote:

> Hi,
> 
>  
> 
> I’ve been playing around with Ozone and the Garage/Car tutorial.
> 
>  
> 
> This is my first experience with ODB’s but the concepts and theory are
> very intriguing!
> 
> Nice work!
> 
>  
> 
>  
> 
> I have a real world case where the data structure of the Garage/Car
> tutorial fits nicely with.
> 
> That is a single collection (Garage) stored by name that includes many
> objects.  In my
> 
> case I will not have a known key to associate with my “Car” objects. 
> Instead my objects
> 
> will appear from clients using my webapp and will have a small number
> of fields that can 
> 
> only have a few different values and only be really unique by the
> timestamp that they are received by.
> 
> But of course I won’t know these timestamps so I wouldn’t be able to
> retrieve off this “key” later.
> 
>  
> 
> Also, I want to store millions of these objects.
> 
>  
> 
> If I modify the tutorial to have a big loop adding cars called “1”,
> “2”, “3”, etc. then the loop 
> 
> runs very slowly.  I’m guessing because with every object that is
> added, the entire data
> 
> structure has to be rewritten?  On my machine (700mhz, 512 mb, SuSE
> 8.1, JVM 1.4.2)
> 
> I was writing 1000 records every 20 minutes.  This is very much slower
> than I require.
> 
>  
> 
> Also, I ran out of memory as I approached 20000 records.  I assume
> this is because the entire
> 
> data structure was in RAM.
> 
>  
> 
> I am positive that I am not using this in the proper way to accomplish
> this objective of fast
> 
> write/read operations as well as running within a reasonable memory
> constraint.  Do you have
> 
> suggestions on how to code something like this?
> 
>  
> 
> Thank you for your suggestions!
> 
>  
> 
> David.
> 
>