Some thoughts on serialization

This is me thinking out loud.

It must be the fifth time I bring this up here, and I still haven't done anything about but to complain about it. I didn't even get in touch with the people previously recommended here that might have good ideas regarding to how to solve it.

Serialization is terrible slow. 

This one application of mine contains some 12GB heap worth of domain data. Due to the complexity of my domain model and the amount of data it contains now I'm starting to hit stack overflows both while marshalling and unmarshalling unless I increase my -Xss. Given the amout of threads I usually run this is not a good solution for me. Thus in deployed environment I don't use snapshots at all, and since I have loads of changes to the persistent data every second the journals grows huge. It can take an hour to restart. 

Theoretically this means that in a year from now it can take me days to load the journals.

Generally speaking this is not a problem, there are many ways around this such as starting the new VM before bringing down the previous. But it's really annoying.

I have a few ideas. For starters all serialization I've seen use a single thread. Theoretically my computer could do it 32x faster if all CPU resources was used. (This is of course only applicable for loading snapshots.) So the other day tried to implement a proof of concept where the marhsaller would associate all instances with an identity, decouple all instances in my graph and then have the unmarshaller first instantiate them in multiple threads and then in a second iteration recouple all instances, but due to how ObjectIn/OutputStream works (in conjunction with my crappy code) I didn't get it working yet. In the end I'll probably have to instantiate some 3-4 helper object for each object to be unmarshalled. But even without trying it out for real I'd say that with my 32 threads it sho
 uld still be a lot faster than running a single thread. It will also cost quite a bit of RAM to keep track of what to be coupled with what in the second iteration, but I don't think anyone here mind spending a bit of RAM for greater speed...

One sort of neat thing that hit me while writing the code is that it's very easy to intern all immutable objects at marshalling time. I for one would have tons of reoccurring strings and dates in my domain data if it wasn't for that intern them in my transactions, this does however make me keep track of them in my root which consumes RAM and require me to think of this every time I write something new. But I only do this for places where I know there is a lot of reoccurring values. There are places in the system I currently don't care about since it would be too much of a hassle to keep track of. My guess is that I can cut heap by 20% or perhaps even more if I interned everything that could be, and perhaps generalized my dates a bit so they are on a minute level rather than milliseconds.

It would be really cool if someone here had some thoughts about this. I think very few people that implemented serialization have had system prevalence in mind when doing it. It's really not that complicated, just a bit of reflection and keeping track of instances as byte array postings. But it would not hurt with a few more brains processing this before implementing it.



			karl
------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org