Re: Question about 2.6.5 kernel scalability

A common bottle neck for systems is, memory bandwidth.

It could be that by adding more processors your maxing out the memory 
bus. Remember each processor must go through the same ASIC when 
accessing memory. This is where NUMA is supposed to enhance memory 
bandwidth, also Unisys ES7000 systems have 4 ASIC's to increase memory 
bandwidth and reduce memory contention.

There are many ways to reduce memory contention but it usually comes in 
the form of special hardware(i.e. large ThirdLevelCache or even 
FourthLevelCache) and/or applications that are specifically designed  
for this purpose (i.e. special programming techniques like processor 
affinity for bound IO). 

This is also the same reason why you don't generally see intel servers 
with more than 8 physical CPU's. Intel actually usues a technology 
called "Fusion Bar" to connect 2 groups of 4 processors to reach a total 
of 8 processors and its primary purpose is to reduce contention on IO 
buses (primarily the memory bus).

Here is link about measuring memory bandwidth,    
http://www.streambench.org/

Just my 2 cents,
Earle

Sang s Lim wrote:

>I am currently measuring the performance of a directory server, OpenLDAP
>on IBM xSeries 455(8-way).
>First see the performance result(operations/sec) with increase in the 
>number of CPUs.
>
>1 : 6,100
>2: 10,000
>4: 17,600
>5: 14,200
>6: 13,200
>7: 13,100
>8: 11,500
>
>As you can see, the throughput starts degrading when the number of CPUs is 
>4.
>when the number of CPUs is 8, the throughput is about 65% of 4 CPUs.
>During experiment, CPU utilization of all cases are about 93%.
>The directory server are multithreaded and in the experiment 16~32 work 
>threads were used.
>I am currently investigating to find the reason of degradation.
>I guess that there might be scheduling anomaly on SMP.
>If any one knows any strange behavior of Linux 2.6.5 scheduler, please let 
>share the knowledge.
>It'll will help me a lot.
>
>Sang Seok
>-
>To unsubscribe from this list: send the line "unsubscribe linux-smp" in
>the body of a message to [email protected]
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
>  
>