Re: Difference in active/non active objects distribution in caches of zserver-threads between Plone 3/Plone 4

Fred van Dijk <f.van.dijk@...> writes:
> It seems that one thread is 'preferred' and gets most of the traffic and
> fills up its cache, where thread 2 and 3 are barely used, at least looking
> at the cache values.

I'm guessing the answer to this is two-fold. First there was a ZODB level
optimization first introduced in ZODB 3.9. Plone 3 used an older version and
only Plone 4.0 upgraded to ZODB 3.9 (and 4.1+ upgraded to ZODB 3.10).

The relevant entry from the ZODB 3.9.0 changelog:

"Previously, database connections were managed as a stack. This tended to
cause the same connection(s) to be used over and over. For example, the most
used connection would typically be the only connection used. In some rare
situations, extra connections could be opened and end up on the top of the
stack, causing extreme memory wastage. Now, when connections are placed on
the stack, they sink below existing connections that have more active objects."

The second part to this story is Plone 4 indeed being much faster.

With Plone being so much faster, it's most often able to completely handle
a request, before any second request gets sent to the same server. This means
there's in practice only ever one thread actually active and this thread uses
the same ZODB connection over and over again.

In the more uncommon scenario, where indeed two requests are handled in
parallel by two threads, each of them will use a different ZODB connection.

In Plone 3 both the often used or less used connection could end up on top of
the connection pool stack. Effectively leading to both (or multiple)
connections to be used and their caches to grow. In Plone 4 the connection
with the largest cache is always returning to the top of the connection pool
and will be used most of the time.

This theory is unproven, so I might be wrong :)

> Increasing zserver-threads in Plone 4 to say 6 only lists more thread
> caches that are not used, 200-500 objects active at most.

There are only very few types of requests that can actually be executed in
parallel by one Zope process. Except for I/O, Python doesn't actually execute
threads in parallel, but there's always just one thread active at a time.

All the benchmarks I've seen suggest that using either one or two threads
is best for performance and the zope2instance recipe defaults to 2 threads.
If you need to handle more concurrent load, you need to add more processes.

Hope this helps,
Hanno


------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk