Threads and findUsingPrototype

I tried to post the message below on Oct 30 but I sent it from the wrong address, so it didn't get through.

Since then, we haven't had any ideas on how to replace findUsingPrototype, and would welcome any thoughts on it.  (It's deprecated, so there must be some thoughts to be shared...)

However, we have made some progress on understanding the threading.  It seems the design is to have one session per thread, and one (or zero) session per dataset, and one current dataset per session.  Is that correct?  

If that is the design, then we'd like to have two threads that have their own sessions, that create datasets containing batches of records.  The batches of records (and their associated datasets) would be disassociated from the session, then put in a queue for the main thread to process, and ultimately destroy.  

We don't want to create sessions for each batch, just datasets for each batch.  So we'd need to create a new dataset for the session after the current dataset has been disassociated from the session (haven't figured out how to do this yet.)
 
We'd appreciate any comments or suggestions on how to replace findUsingPrototype (it's too slow) and on how to best use SimpleORM in a multithreaded application.

--
John Abraham
[email protected]

*************
** Original message (that didn't make it to the list on Friday Oct 30) **

Hello there.  We have been working at speeding up our SimpleORM project which processes 65 million records in sequence, in about 4000 batches.  We've profiled the code (using hprof) and would like to do two things now to increase performance:

1) we'd like to spin off two threads to pre-fetch batches of records (average batch size 65,000,000/4000 = 16,250) and put the resulting ArrayLists of SRecordInstances in a FIFO queue. That way we can process previous batches of records off the head of the queue on a separate processor while the database is engaged fetching the next batches for the tail of the queue. The Java program spends about 30% of its time waiting for the database to return the next batch and even more time (about 8%) for SimpleORM to create the resulting ArrayList; we might as well ask for a few batches in advance.  Question:  is SimpleORM thread safe for this type of operation?

2) findUsingPrototype is quite slow for repeated finds of prefetched records:  SDataSet.findOrCreate() has to create the prototype object, then SDataSet.finder() has to populate its fields, then there is the processing of SRecordInstance.equals() and SRecordInstance.hashCode() to find the record in the cache.  All of these together are, I think, about 30% of our runtime.  Is there a way to speed this up?  I notice findUsingPrototype() is deprecated, suggesting that someone already has a new plan for how to do this.  For some of our tables we know the underlying data is static, so our current plan is to build our own faster hashmaps for certain types of records (especially those with a single integer primary key), and check our own cache first before checking SimpleORM's cache.  But I'd rat
 her speed up SimpleORM than work around it.

So those are my two questions: 1) Threadsafe?  2) are there already ideas or bits of code to replace findUsingPrototype() with something faster?





------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/SimpleORM/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/SimpleORM/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:[email protected] 
    mailto:[email protected]

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/