Re: Difference in active/non active objects distribution in caches of zserver-threads between Plone 3/Plone 4

My understanding was that plone 3 didn't leak memory but rather python 2.4
never releases memory back to operating system. Combine that with blobs
being stored in the zodb cache (unlike plone4) meant that the plone process
would grow large to fit those blobs and never shrink again.
On 28 Dec 2013 03:43, "Héctor Velarde" <[email protected]> wrote:

> I really enjoy always these posts from Hanno; thank you very much for
> explaining this kind of things to all of us, mere mortals :-)
>
> I want to make a clarification: Plone 3.0 used Zope 2.10.4 and not "an
> older version" as Hanno states in his answer.
>
> http://plone.org/products/plone/releases/3.0
>
> perhaps what Fred mentions is more related with the Python version
> supported at that time (2.4.4)?
>
> for years I've heard terror stories about Python 2.4 memory leaks so today
> I just checked the release changelog and found that the last bug fix
> release of that branch (2.4.6 ) included something that seems to be of
> interest:
>
> - Issue #2620: Overflow checking when allocating or reallocating memory
>   was not always being done properly in some python types and extension
>   modules.  PyMem_MALLOC, PyMem_REALLOC, PyMem_NEW and PyMem_RESIZE have
>   all been updated to perform better checks and places in the code that
>   would previously leak memory on the error path when such an allocation
>   failed have been fixed.
>
> http://www.python.org/download/releases/2.4.6/NEWS.txt
>
> being a bug release seems to me there is no reason not to upgrade.
>
> as a bonus, and for the hopeless optimist inside each one of us, I found
> this post from 5 years ago:
>
> http://blog.hannosch.eu/2008/12/plone-versus-python-3.html
>
> happy new year!
>
> Héctor Velarde
>
> On 22-10-2013 10:20, Hanno Schlichting wrote:
>
>> Fred van Dijk <f.van.dijk@...> writes:
>>
>>> It seems that one thread is 'preferred' and gets most of the traffic and
>>> fills up its cache, where thread 2 and 3 are barely used, at least
>>> looking
>>> at the cache values.
>>>
>>
>> I'm guessing the answer to this is two-fold. First there was a ZODB level
>> optimization first introduced in ZODB 3.9. Plone 3 used an older version
>> and
>> only Plone 4.0 upgraded to ZODB 3.9 (and 4.1+ upgraded to ZODB 3.10).
>>
>> The relevant entry from the ZODB 3.9.0 changelog:
>>
>> "Previously, database connections were managed as a stack. This tended to
>> cause the same connection(s) to be used over and over. For example, the
>> most
>> used connection would typically be the only connection used. In some rare
>> situations, extra connections could be opened and end up on the top of the
>> stack, causing extreme memory wastage. Now, when connections are placed on
>> the stack, they sink below existing connections that have more active
>> objects."
>>
>> The second part to this story is Plone 4 indeed being much faster.
>>
>> With Plone being so much faster, it's most often able to completely handle
>> a request, before any second request gets sent to the same server. This
>> means
>> there's in practice only ever one thread actually active and this thread
>> uses
>> the same ZODB connection over and over again.
>>
>> In the more uncommon scenario, where indeed two requests are handled in
>> parallel by two threads, each of them will use a different ZODB
>> connection.
>>
>> In Plone 3 both the often used or less used connection could end up on
>> top of
>> the connection pool stack. Effectively leading to both (or multiple)
>> connections to be used and their caches to grow. In Plone 4 the connection
>> with the largest cache is always returning to the top of the connection
>> pool
>> and will be used most of the time.
>>
>> This theory is unproven, so I might be wrong :)
>>
>>  Increasing zserver-threads in Plone 4 to say 6 only lists more thread
>>> caches that are not used, 200-500 objects active at most.
>>>
>>
>> There are only very few types of requests that can actually be executed in
>> parallel by one Zope process. Except for I/O, Python doesn't actually
>> execute
>> threads in parallel, but there's always just one thread active at a time.
>>
>> All the benchmarks I've seen suggest that using either one or two threads
>> is best for performance and the zope2instance recipe defaults to 2
>> threads.
>> If you need to handle more concurrent load, you need to add more
>> processes.
>>
>> Hope this helps,
>> Hanno
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
>> from
>> the latest Intel processors and coprocessors. See abstracts and register >
>> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=
>> /4140/ostg.clktrk
>> _______________________________________________
>> Plone-developers mailing list
>> Plone-developers-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>> https://lists.sourceforge.net/lists/listinfo/plone-developers
>>
>>
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> _______________________________________________
> Plone-developers mailing list
> Plone-developers-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/plone-developers
>
>

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT 
organizations don't have a clear picture of how application performance 
affects their revenue. With AppDynamics, you get 100% visibility into your 
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk

_______________________________________________
Plone-developers mailing list
Plone-developers-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/plone-developers