pagecache and hugetlb tuning for Oracle DB operations

I am managing a new server configuration to support Oracle 9i v 9.2.0.
The SGA is approximately 7 GB and we are running on a pair of new HP
DL585's with 4 dual-core Opteron CPU's and 16GB of RAM on our test
system, 32GB on production. After going live with the system last week,
users started complaining almost immediately about slow performance.
Upon investigation, it was determined that we were thrashing swap pretty
heavily. I do not mean data just sitting in swap, vmstat reported
constant si and so numbers, sometimes over a thousand in a 5 second
sample period. We could tell from the output of the free command and top
that there was a large amount of RAM dedicated to disk caching, so we
decided to decrease the disk cache. I set the vm.pagecache parameter on
the test platform to "1 5 10" and it had absolutely no effect at all
that I could see. The developers were able to easily generate disk cache
that filled over 90% of RAM.  I thought that pagecache setting limited
cache to 10% of RAM?

We also tried setting the hugetlb_pool parameters to dedicate 8GB of
huge pages for the SGA. I set this with the command "sysctl -w
vm.hugetlb_pool=8192" and according to /proc/meminfo I got 8 pages of
2048K pages. That did not make sense to me, so I put the same setting in
sysctl.conf and rebooted. The server hung at boot as it was "Setting
kernel parameters." I simply commented out the hugetlb_pool setting at
it booted without error. Can anyone tell me what I'm missing here or
recommend some better setting for our environment?

Red Hat Enterprise Linux AS release 3 (Taroon Update 7)
Linux XXX.XXXX.com 2.4.21-40.ELsmp #1 SMP Thu Feb 2 22:13:55 EST 2006
x86_64 x86_64 x86_64 GNU/Linux

# free
             total       used       free     shared    buffers
cached
Mem:      16010356   13834384    2175972          0     287552
12683848
-/+ buffers/cache:     862984   15147372
Swap:      8388472          0    8388472

# cat /proc/sys/vm/pagecache
1       5       10

# cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 33
model name      : AMD Opteron (tm) Processor 880
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
stepping        : 2
cpu MHz         : 2396.886
cache size      : 1024 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm
3dnowext 3dnow
bogomips        : 4771.02
TLB size        : 1088 4K pages
clflush size    : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

(.... duplicated 7 more times)

Jim Williams
Linux Administrator
Desk: (407) 804-8185
Cell: (407) 687-4532
[email protected]

--
Taroon-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/taroon-list