Re: [suse-oracle] 10gR2 RAC on SUSE10

Hi Martin,

Thank you so much for the reply, it is much appreciated.

I've done more reading on HugePages, this is what I found so far:

First of all, kernel.shm-bigpages-per-file and vm.hugetlb_pool are not applicable to kernel 2.6.

Secondly, it seems adding hugepages=xx into /boot/grub/menu.lst is the same as adding vm.nr_huge_pages=xx into /etc/sysctl.

But I am still confused by:
  kernel.mmap-use-hugepages
  kernel.shm-use-hugepages
  SHM_USE_BIGPAGES in /etc/sysconfig/oracle

A while ago, I did a quick test on NUMA with a different HW, it seems in kernel 2.6, Linux will allocate HugePages equally from each CPU, but SGA won't.  For example, set 1000 HugePages on a machine with 4 CPUs, 250 HugePages will allocated to each CPU.  But, if the SGA is 300MB, the SGA is all allocated from HugePages on one CPU.  If I plan both HugePages and SGA carefully, I can still get SGA evenly distributed on all CPUs.  But, as Oracle is not NUMA aware on x86_64, does it give me any performance benefit at all?  Who knows.

From Oracle side, there is no Validated Configurations on 10gR2 on SUSE 10:
http://www.oracle.com/technology/tech/linux/validated-configurations/index.html

From Novell side, there is no YES certified Oracle 10gR2 on SUSE 10:
http://www.novell.com/partnerguide/company/41934.html

Even though Martin and others had done 10gR2 RAC on SUSE 10 before, I still feel like a guinea pig as there are so much uncetainty.

Regards,
Han

-----Original Message-----
From: Klier, Martin (A.T.U) [mailto:[email protected]]
Sent: Friday, 30 May 2008 5:21 PM
To: Xie, Han; [email protected]
Subject: AW: [suse-oracle] 10gR2 RAC on SUSE10

Hi Han,

we've got several RACs on 10gR2 and SLES10 x86_64 in production, so neither you nor we are the guinea pigs ;)

Kindly see some comments within your mail body.


> -----Ursprüngliche Nachricht-----
> 1. During the SUSE installation, do I need to tick "High Availability"?
> (I will install Oracle 10.2.0.4 which does not use hangcheck-timer any
> more.)
No.

> 2. Is there any difference to implement HugePages by add hugepages=xx
> into /boot/grub/menu.lst or by adding vm.nr_huge_pages=xx into
> /etc/sysctl please?
I don't know for absolutely sure with grub, but I always set this kind of stuff in /etc/sysctl.conf for higher flexibility (careful changes at run time).

>   I found this subject confusing.  There are many things floating
> around, such as:
>           SHM_USE_BIGPAGES in /etc/sysconfig/oracle,
>           kernel.shm-bigpages-per-file,
>           kernel.shm-use-bigpages,
>           vm.hugetlb_pool,
>
>   What is SUSE/Oracle's recommendation?  For example, should I set
> both vm.nr_huge_pages and vm.hugetlb_pool?
Settings in /etc/sysconfig/oracle and sysctl.conf end up the same, the oracle start script does the kernel setting same way.

> 3. Do I still need to set Oracle's pre_page_sga=true and lock_sga=true?
> It looks like SUSE 10 fixed vm.disable_cup_mlock issue, but not sure
> if Oracle 10gR2 can see the fix automatically.
NEED is not the topic. pre_page_sga=true means that all SGA will be allocated in one piece, which has advantages and disadvantages. One disadvantage in my experience is, that if the RAC has crashed and has to do instance recovery, the CRS considers the database as dead very easily due to the long startup time. In my special case, I don't use it, but nevertheless I am using hugepages.

lock_sga=true means that the SGA is not to be pages out into swap space. I can only recommend this setting, but remember to set the
oracle          hard    memlock         -1
oracle          soft    memlock         -1
in /etc/security/limits.conf in order to allow user oracle to pin memory without limit.

> 6. The hardware supports NUMA, but Oracle on X86_64 does not, should
> NUMA be turned on or off please?
There are nice memory fragmentation (or context changing) tests in the net, shame that I can't find the one about NUMA in special at the moment.

Other paper on OS optimization:
http://www.redbooks.ibm.com/redpapers/pdfs/redp3862.pdf
But I guess Arun will explain it all in detail.


> 7. With RAC, the clusterware will startup all services upon reboot, do
> I need to manually disable /etc/init.d/oracle?  Or setting all entries
> in /etc/oratab to 'N' will be enough.
I am using /etc/init.d/oracle and /etc/sysconfig/oracle to set all the kernel parameters like SHMMAX and swith off all other stuff with NO.

Do you know Alejandro Vargas' blog?
http://blogs.oracle.com/AlejandroVargas/

Maybe you want to have a look at my oracle blog as well, there are several RAC topics.
http://www.usn-it.de/index.php/category/it/oracle/



Kind regards,
--
Freundliche Grüße

i.A. Martin Klier
Systemadministration / Datenbanken


A.T.U Auto-Teile-Unger
Handels GmbH & Co. KG
Dr.-Kilian-Str. 4
92637 Weiden i. d. OPf.

Tel.: +49 961 306-5663
Fax:  +49 961 306-5982

[email protected]
www.atu.eu
_______________________________________________
suse-oracle mailing list
[email protected]
http://listx.novell.com/mailman/listinfo/suse-oracle