SUMMARY: linux scalability

Thanks for the great responses. Several of you related real world positive
experiences with high end intel servers with >4GB memory. One person
indicated that PAE comes with a very significant performance hit. It
didn't seem that anyone was able to directly compare a high end intel
linux based system with a any RISC/[Solaris HPUX or AIX] combinations.

One person indicated that the ext3 filesystem had a bug causing degraded
filesystem performance with 4-way systems. The recommendation was to use
reiser filesystem. Anyone else experience problems with ext3 on 4-way
machines?

We've decided to go ahead and do some testing with RHAS and probably at
least an ML370 or preferrably a DL740 with 8GB memory.

All the responses follow.

Thanks,
Vic Engle



Subject: Re: linux scalability   View Full Header
View Printable Version
From: Jochen Schmidt <jochen.schmidt-hvmQA13wJmtWk0Htik3J/[email protected]>
Date: Fri, May 23, 2003 10:57 am
To: sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]
Priority:  Normal



-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Vic,

we're currently running RedHat Advanced Server 2.1 on 8 way Fujitsu
Siemens Computers as File- and Printserver (samba) or Database-Server
(oracle).
The machines performes very well, but it depends on the software you want
to run on the machines. If the software is only able to use one you won't
be happy with those machines...

Greets

Jochen





Subject: RE: linux scalability   View Full Header
View Printable Version
From: "Rivera, Angel R" <[email protected]>
Date: Fri, May 23, 2003 10:15 am
To: <sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]>
Priority:  Normal



we have done some testing on a 4-way xeon dell w/ 16GB of memory.





Subject: Re: linux scalability   View Full Header
View Printable Version
From: "Seth Rothenberg" <[email protected]>
Date: Fri, May 23, 2003 10:04 am
To: <sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]>
Priority:  Normal



Great question!  I manage Sun systems at work and I keep
telling my boss about Linux (my desktop is Linux)....

Gotta go now, meeting to discuss a $400K Purchase order with Sun :-)

Looking forward to the summary...






Subject: Re: linux scalability   View Full Header
View Printable Version
From: Scott Croft <[email protected]>
Date: Fri, May 23, 2003 9:53 am
To: sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]
Priority:  Normal



Vic,
   I have done work on sizes up to a 580 with really excellent
performance. I did benchmarking on 4-way 700MHz xeon's as well as a
4-way 1.4GHz. Both had 6GB of memory and I saw no issues at all. These
of course were running 32-bit kernels. We tested a product, that
comparing it to running the same version of software on windows and
Solaris, was light years ahead in capability with Linux. It just did a
better job overall. The only drawback at this point would be the disk
sub-system. Unless you are running HW raid behind the scenes, I would
suggest using NAS or SAN attached storage.

Just some thoughts.

Scott



Subject: Re: linux scalability   View Full Header
View Printable Version
From: "Robert L. Harris" <[email protected]>
Date: Fri, May 23, 2003 9:53 am
To: sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]
Cc: linuxmanagers-35TzE1X9F6582KRnZfj+bdi2O/[email protected]
Priority:  Normal




I've got a couple 4 way boxes with 16 Gigs of ram..  It definitely scales
nicely there.  Havne't heard much about larger though.  One thing to
keep in mind, if you want a journaling filesystem with something like
this go reiser as there's a bug with ext3 and 4+ proc systems, the
machine just about comes to a crawl.  My 4x with a 200Gig SCSI Software
RAID5 mailserver is a very sweet machine though.



Subject: Re: linux scalability   View Full Header
View Printable Version
From: Aly Dharshi <aly.dharshi-/[email protected]>
Date: Fri, May 23, 2003 10:59 am
To: sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]
Cc: linuxmanagers-35TzE1X9F6582KRnZfj+bdi2O/[email protected]
Priority:  Normal



Hello,

        I have no direct experience with such powerful boxes, but the RedHat
Advanced Server 2.1 should be able - theoretically atleast - be able to
handle the specs you are suggesting.

        Oracle had a paper on this with some of these issues raised and
mentions that RH AS 2.1 seems to be able to handle these sort of specs,
Dell has some high end PE8000 series with about 32GB of RAM and that is
certified for something like RH AS 2.1 and Oracle 9i.

        You may want to look into the papers from Oracle's Linux site, I
think
its called Oracle9iR2 on Linux.

        Hope that helps,

        Cheers,

        Aly.



Subject: Re: linux scalability   View Full Header
View Printable Version
From: Mike Brodbelt <m.brodbelt-ExSj+31/Nztaa/[email protected]>
Date: Fri, May 23, 2003 11:10 am
To: sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]
Priority:  Normal



sunmanager-vO8Gr2FDZLx9uiUsa/[email protected] wrote:

> I read in the kernel documentation that support for 64GB was possible but
> required a kernel with large memory support which in turn would use an
> intel processor feature called PAE (Physical Address Extension). Does this
> work well and is there a significant memory performance hit?

PAE is an Intel hack to allow the IA-32 architecture to access > 4Gb of
RAM. The address space of a 32 bit CPU is only 4Gb, so it's required to
support larger amounts of memory, no matter what OS you have on the
machine. You will need to compile a kernel with HIGHMEM support, but
that's all. Linux needs HIGHMEM in the kernel if you have amore than 1Gb
of physical RAM, due to the way the kernel splits the 4Gb address space
between kernel and user mode tasks.

It works fine, on the whole. You do need a motherboard chipset that
caches all the memory in the machine, but that shouldn't be a problem on
 a machine spec like yours. In an ideal world, PAE would be unnecessary,
but until we all move to IA-64 or Opteron, you're stuck with it.

> Have any of you done any testing with 4 way and 8 way intel boxes with
linux?

No, but with a 2.4 kernel you should have no trouble scaling well to 8
processors.

Mike.





Subject: Re: linux scalability   View Full Header
View Printable Version
From: Randy L Schrickel <[email protected]>
Date: Fri, May 23, 2003 12:36 pm
To: sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]
Priority:  Normal



sunmanager-vO8Gr2FDZLx9uiUsa/[email protected] wrote:
>
> I read in the kernel documentation that support for 64GB was possible but
> required a kernel with large memory support which in turn would use an
> intel processor feature called PAE (Physical Address Extension). Does this
> work well and is there a significant memory performance hit?

Hi Vic. We do a lot with Linux on 4 processor/16 GB Pentium/Xeon
boxes. Almost exclusively using Red Hat Advanced Server. The newer
processors with hyper-threading actually show up as double the number
of physical cpus (in "top" or "cat /proc/cpuinfo"). The OS sees all
the RAM with no problem.

BUT, individual processes are still limited to 3 GB each (maybe 4GB?).
That's a bummer for what we want to do. So that may be something to
think about if you have a similar need. I haven't really benchmarked
the performance against anything else, so I can't comment on any
effect the PAE might have.

Hope that helps,
randy



Subject: RE: linux scalability   View Full Header
View Printable Version
From: "Mike Kercher" <[email protected]>
Date: Fri, May 23, 2003 1:23 pm
To: <sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]>
Priority:  Normal



I run Redhat AS 2.1 and Redhat 7.2/3 on Proliants of all flavors...from Quad
Ppro's up to Quad Xeon's and it runs like a dream!

Mike





Subject: Re: linux scalability -- do you remember EMS under DOS?   View
Full Header
View Printable Version
From: "Bryan J. Smith" <[email protected]>
Date: Fri, May 23, 2003 10:14 am
To: sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]
Priority:  Normal



Quoting sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]:
> Hi,
> Does anyone have any experience with deploying linux on really high
> end intel boxes like maybe an 8 way Proliant DL760 with 16GB ram?

The problem with IA32 (aka 32-bit x86) is that it doesn't scale much past 4
processors, and even
beyond 2 processors isn't very efficient either.  A lot has to do with the
SMP
approach of Intel, and
Intel's lack of good I/O design in its system logic -- although their recent
partnership with
ServerWorks has addressed some of it.

APINetworks/AMD is addressing a lot of the I/O issues with HyperTransport,
which is
the native
system interconnect in its new x86-64 processors -- Opteron and Athlon 64.
 AMD's
reference
designs use Opterons in upto 4 and even 8 way, although Cray and others
are scaling
it to
hundreds of processors using an optical version of HyperTransport.

> I read in the kernel documentation that support for 64GB was possible
> but required a kernel with large memory support which in turn would use an
> intel processor feature called PAE (Physical Address Extension). Does
> this work well and is there a significant memory performance hit?

PAE has a _massive_performance_hit_, which is why many Linux developers
have come
out and
lambasted Intel over saying 32-bit should be commodity until 2007-2009.  Many
commercial
developers who use Linux as a desktop development platform who need >4GB
have been
early
adopters of AMD Opteron for this very reason.

Do you remember EMS under does?  PAE works _exactly_ like EMS.

EMS under DOS used a small 64KB window under 1MB to page 64KB windows of
another
15MB of
RAM in and out.

Using the same principle, PAE uses a window under 4GB to page in windows
of another
60GB.
Again, it sux.

> Have any of you done any testing with 4 way and 8 way intel boxes with
> linux?

If you keep your memory 4GB or under, you're okay.

If you need more than 4GB, either stick with Solaris/SPARC, or look at
Linux/x86-64
(now that
Alpha is basically dead).  Opteron is still expensive, although AMD will be
releasing 2GHz+
Opterons later this year, and they will be more commodity (because the
Athlon 64
will be released
at the same time).


-- 
Bryan J. Smith, E.I.   [email protected]   http://thebs.org


Subject: Re: linux scalability   View Full Header
View Printable Version
From: [email protected]
Date: Fri, May 23, 2003 1:15 pm
To: sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]
Priority:  Normal

Hey Vic,

I have a 2-way intel box with 6G of Mem(once above the 4G, its all the
same i guess)

I have had some(if not lots) of problems  with 2.4.20 and not many with
2.4.18

I have heard that 4 and 8 way arent very "stable" though....have no
practical experience.

I would also buy from a vendor who provides "real" linux support like
californiadigital.com penguincomputing.com

So that i have someone to help with the linux stuff...

Good Luck.




Subject: RE: linux scalability   View Full Header
View Printable Version
From: "Carroll, Jim P [Contractor]" <jcarro10-R4EMBtibEryFMUSaP/[email protected]>
Date: Fri, May 23, 2003 1:12 pm
To: <sunmanager-vO8Gr2FDZLx9uiUsa/[email protected]>
Priority:  Normal



I don't have details, but you might want to check out the specs over at
www.egenera.com which definitely fall into the definition of 'high end'.

jc







> Hi,
>
> Does anyone have any experience with deploying linux on really high end
> intel boxes like maybe an 8 way Proliant DL760 with 16GB ram?
>
> I read in the kernel documentation that support for 64GB was possible
> but required a kernel with large memory support which in turn would use
> an intel processor feature called PAE (Physical Address Extension). Does
> this work well and is there a significant memory performance hit?
>
> Have any of you done any testing with 4 way and 8 way intel boxes with
> linux?
>
> Thanks,
> Vic
> _______________________________________________
> LinuxManagers mailing list - http://www.linuxmanagers.org
> submissions: LinuxManagers-35TzE1X9F6582KRnZfj+bdi2O/[email protected]
> subscribe/unsubscribe:
> http://www.linuxmanagers.org/mailman/listinfo/linuxmanagers
_______________________________________________
LinuxManagers mailing list - http://www.linuxmanagers.org
submissions: LinuxManagers-35TzE1X9F6582KRnZfj+bdi2O/[email protected]
subscribe/unsubscribe: http://www.linuxmanagers.org/mailman/listinfo/linuxmanagers