Re: Threads and findUsingPrototype

This is just a note to let you know that this multi-threaded design did work.  All indications are that this *should* be much faster, but for some reason it isn't.  We're blaming SQL Server, Microsoft's JDBC driver, and/or the database configuration, because the code is now spending most of it's time in socketRead() waiting for the query result.  Investigations are continuing.  With PostgreSQL (instead of SQL Server) it is faster.  

I had some other thoughts about alternative designs.  What would be really cool is if the query methods in SimpleORM could immediately return an iterator to a List object that is actually populated by another thread.  The getNext() method in the iterator would block if the other thread hadn't pulled the data from the database yet. But I think the whole design of one-dataset-per-session and one-session-per-dataset wouldn't permit this?  Also we don't want to make SimpleORM into NotSimpleORM.

We had to add a method to SSessionJdbc which attaches a new dataset to a session, to replace the one that is detached and handed over to the processing thread.

--
John Abraham


--- In [email protected], jabraham@... wrote:
>
> I tried to post the message below on Oct 30 but I sent it from the wrong address, so it didn't get through.
> 
> Since then, we haven't had any ideas on how to replace findUsingPrototype, and would welcome any thoughts on it.  (It's deprecated, so there must be some thoughts to be shared...)
> 
> However, we have made some progress on understanding the threading.  It seems the design is to have one session per thread, and one (or zero) session per dataset, and one current dataset per session.  Is that correct?  
> 
> If that is the design, then we'd like to have two threads that have their own sessions, that create datasets containing batches of records.  The batches of records (and their associated datasets) would be disassociated from the session, then put in a queue for the main thread to process, and ultimately destroy.  
> 
> We don't want to create sessions for each batch, just datasets for each batch.  So we'd need to create a new dataset for the session after the current dataset has been disassociated from the session (haven't figured out how to do this yet.)
>  
> We'd appreciate any comments or suggestions on how to replace findUsingPrototype (it's too slow) and on how to best use SimpleORM in a multithreaded application.
> 
> --
> John Abraham
> jabraham@...
> 
> *************
> ** Original message (that didn't make it to the list on Friday Oct 30) **
> 
> Hello there.  We have been working at speeding up our SimpleORM project which processes 65 million records in sequence, in about 4000 batches.  We've profiled the code (using hprof) and would like to do two things now to increase performance:
> 
> 1) we'd like to spin off two threads to pre-fetch batches of records (average batch size 65,000,000/4000 = 16,250) and put the resulting ArrayLists of SRecordInstances in a FIFO queue. That way we can process previous batches of records off the head of the queue on a separate processor while the database is engaged fetching the next batches for the tail of the queue. The Java program spends about 30% of its time waiting for the database to return the next batch and even more time (about 8%) for SimpleORM to create the resulting ArrayList; we might as well ask for a few batches in advance.  Question:  is SimpleORM thread safe for this type of operation?
> 
> 2) findUsingPrototype is quite slow for repeated finds of prefetched records:  SDataSet.findOrCreate() has to create the prototype object, then SDataSet.finder() has to populate its fields, then there is the processing of SRecordInstance.equals() and SRecordInstance.hashCode() to find the record in the cache.  All of these together are, I think, about 30% of our runtime.  Is there a way to speed this up?  I notice findUsingPrototype() is deprecated, suggesting that someone already has a new plan for how to do this.  For some of our tables we know the underlying data is static, so our current plan is to build our own faster hashmaps for certain types of records (especially those with a single integer primary key), and check our own cache first before checking SimpleORM's cache.  But I'd r
 ather speed up SimpleORM than work around it.
> 
> So those are my two questions: 1) Threadsafe?  2) are there already ideas or bits of code to replace findUsingPrototype() with something faster?
>




------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/SimpleORM/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/SimpleORM/join
    (Yahoo! ID required)

<*> To change settings via email:
    [email protected] 
    [email protected]

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/