Speeding up loading snapshots

Please bear with me here, I'm really just thinking out loud so it may or may not make any sense.

I'm sort of tired of having to wait for my 5GB application to start up. 

To me the bottleneck seems to be that only one thread can be used to load a snapshot. But if one can separate the instance graph from root in entity objects (those with an identity) and store aggregated entity objects as stubs with just an identity in the snapshot, then it would be possible to have one snapshot per primary entity object index. One could then load as many primary indices at the same time as one have hardware cores. A secondary iteration would then replace the aggregated entity stubs with coupling the true instances, which also could be multithreaded.

A bit of annotations, a bit of reflection and an ad hoc serialization format.

According to my head it could be some 10-16 times faster to load such a decoupled set of snaphots and then recouple it using my 8 hyperthreading cores compared to just using a single thread to load the single snapshot as I do now.

Any smart comments on this?

I'll try to implement something experimental for kodapan-entity-store.

(I suppose the best thing to do is to implement a VM implementation dependent memory dump rather than using java.io.Serializable. But I assume that's too much work.)


	karl
------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires 
February 28th, so secure your free ArcSight Logger TODAY! 
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org