Re: GC for Large Heaps

Prevayler mirroring is demoed with demo2 inside the Prevayler distro.
Run it with mirroring and let me know.

It is only a proof of concept, though. Some simple things would still
have to be implemented: if the master crashes and restarts, for
example, the clients will not attempt to reconnect.

Klaus


On Thu, Oct 13, 2011 at 2:50 PM, Naveen Chawla <[email protected]> wrote:
> That's brilliant Klaus, thanks! This'll choose the G1 collector in Java 7 by
> default, right?
>
> I have no idea how big my heap will become. But I'd like to prepare for sky
> being the limit. Which is also why I desperately want to be able to
> hot-swap/upgrade machines to add RAM seamlessly while my app is still
> running. This was planned for Prevayler 3, right? Should I wait for this or
> should I do clustering with an app server to do replication? Do these work
> ok for this purpose (i.e. just being able to literally switch off one of the
> computers any time, chuck some RAM in, and carry on, repeat with the other
> computer, or just swap for a new computers, one at a time, with the right
> configuration)? Which Java app server is best for this? Or should I wait for
> Prevayler 3?
>
> On 13 October 2011 15:45, Klaus Wuestefeld <[email protected]> wrote:
>>
>> The Java VM since Java5 has simplified GC options that they call
>> "ergonomic". You can choose to minimize any two out of the following
>> three things, according to your needs:
>>
>>  1) Pause time for exclusive GC.
>>  2) % CPU for GC em parallel with your application.
>>  3) Use of memory.
>>
>> Java5:
>> http://download.oracle.com/javase/1.5.0/docs/guide/vm/gc-ergonomics.html
>> Java7:
>> http://download.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html
>>
>> My choice of VM options for prevalent system servers is:
>>  -server
>>  -XX:MaxGCPauseMillis=500
>>  -XX:GCTimeRatio=3
>>
>> MaxGCPauseMillis=500 gives the VM a target of not pausing for more
>> than 500 millis for GC. You can use less. It will simply increase the
>> pressure on CPU usage and RAM.
>>
>> GCTimeRatio=3 will give the parallel garbage collector 25% of CPU (75%
>> for the app) before it starts allocating more heap. The default is 99
>> which gives only 1% of CPU to the collector and 99% to the app. With
>> prevalent systems the problem is normally not CPU but RAM, so it is
>> good to give the collector ample CPU.
>>
>> Given the above, the VM will apply a pretty cool adaptive heap
>> generation allocation strategy:
>>
>> "1. If the GC pause time is greater than the pause time goal then
>> reduce the generations sizes to better attain the goal.
>> 2. If the pause time goal is being met then consider the application's
>> throughput goal. If the application's throughput goal is not being
>> met, then increase the sizes of the generations to better attain the
>> goal.
>> 3. If both the pause time goal and the throughput goal are being met,
>> then the size of the generations are decreased to reduce footprint."
>>
>> ;)
>>
>> Klaus
>> PS: Naveen, did you answer Ralph's question on how big your heap will be?
>>
>>
>> On Thu, Oct 13, 2011 at 9:05 AM, Diego Fincatto
>> <[email protected]> wrote:
>> > Maybe terracotta can help!
>> > http://www.terracotta.org/products/bigmemory
>> >
>> >
>> > On Thu, Oct 13, 2011 at 8:53 AM, Naveen Chawla <[email protected]>
>> > wrote:
>> >> I agree, there's no way forcing garbage collection is "pauseless" as
>> >> such,
>> >> I'm just unsure if it could be a cheap workaround for "slow down now,
>> >> save
>> >> later". It turns out ordinary Java has garbage collector options G1 and
>> >> CMS,
>> >> either of which may altogether avoid long pauses, I don't know.
>> >>
>> >> So I'm thinking G1 or CMS gc, with heap size fixed to the maximum
>> >> comfortable RAM (unless G1 or CMS still triggers long pauses with large
>> >> memories). Anyone recommend different?
>> >>
>> >> On 13 October 2011 10:54, Ralph Johnson <[email protected]> wrote:
>> >>>
>> >>> If you want to collect more frequently, just make the size of your
>> >>> memory partitions smaller.   The only reason to ask for a collection
>> >>> is because you know that this is a good time to do the work, i.e. you
>> >>> aren't doing anything else.
>> >>>
>> >>> If you could get a pauseless g.c. by manual intervention then they
>> >>> wouldn't be so hard to make.
>> >>>
>> >>> -Ralph
>> >>>
>> >>>
>> >>>
>> >>> ------------------------------------------------------------------------------
>> >>> All the data continuously generated in your IT infrastructure contains
>> >>> a
>> >>> definitive record of customers, application performance, security
>> >>> threats, fraudulent activity and more. Splunk takes this data and
>> >>> makes
>> >>> sense of it. Business sense. IT sense. Common sense.
>> >>> http://p.sf.net/sfu/splunk-d2d-oct
>> >>> _______________________________________________
>> >>> To unsubscribe go to the end of this page:
>> >>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>> >>> _______________________________________________
>> >>> "Databases in Memoriam" -- http://www.prevayler.org
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> All the data continuously generated in your IT infrastructure contains
>> >> a
>> >> definitive record of customers, application performance, security
>> >> threats, fraudulent activity and more. Splunk takes this data and makes
>> >> sense of it. Business sense. IT sense. Common sense.
>> >> http://p.sf.net/sfu/splunk-d2d-oct
>> >> _______________________________________________
>> >> To unsubscribe go to the end of this page:
>> >> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>> >> _______________________________________________
>> >> "Databases in Memoriam" -- http://www.prevayler.org
>> >>
>> >>
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > All the data continuously generated in your IT infrastructure contains a
>> > definitive record of customers, application performance, security
>> > threats, fraudulent activity and more. Splunk takes this data and makes
>> > sense of it. Business sense. IT sense. Common sense.
>> > http://p.sf.net/sfu/splunk-d2d-oct
>> > _______________________________________________
>> > To unsubscribe go to the end of this page:
>> > http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>> > _______________________________________________
>> > "Databases in Memoriam" -- http://www.prevayler.org
>> >
>>
>>
>> ------------------------------------------------------------------------------
>> All the data continuously generated in your IT infrastructure contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2d-oct
>> _______________________________________________
>> To unsubscribe go to the end of this page:
>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>> _______________________________________________
>> "Databases in Memoriam" -- http://www.prevayler.org
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> To unsubscribe go to the end of this page:
> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org
>
>

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org