Re: [suse-oracle] Oracle Upgrade ( OOM Killer Invoked )

2010/4/11 Austin Joseph <[email protected]>:
> Dear All ,
>
> One of our customer was upgrading Oracle 10gR2 patch on production server
> from ver 10.2.0.2  to version 10.2.0.4 , but the patch upgrade failed with
> OOM-killer getting invoked due to system consuming lot of memory and OOM
> function killing the oracle process.
>
> The server is running SLES 10SP2 ( 42.8-smp kernel )  has 128 GB RAM and 64
> GB swap space ( 32 dualcore cpu IBM x3950 system )
>
> The messages as seen in kernel logs are like
>
> ####
>
> Oracle upgrade messages Mar 14 15:25:47 insapr3pin01 kernel: Call Trace:
> <ffffffff80164676>{oom_kill_process+87} Mar 14 15:25:47 insapr3pin01 kernel:
> <ffffffff8017f3c8>{shmem_swapin_async+87} <ffffffff80168bda>{__pagevec_lru_add_active+228} Mar
> 14 15:25:47 insapr3pin01 kernel:
> <ffffffff8017f477>{shmem_swapin+136} <ffffffff801673c3>{__set_page_dirty_nobuffers+198} Mar
> 14 15:25:47 insapr3pin01 kernel:
> <ffffffff8017f5ba>{shmem_getpage+308} <ffffffff8013cfe0>{del_timer_sync+12} Mar
> 14 15:25:47 insapr3pin01 kernel: Node 7 Normal per-cpu: Mar 14 15:25:47
> insapr3pin01 kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 29 Cold: hi: 62,
> btch: 15 usd: 0 Mar 14 15:25:47 insapr3pin01 kernel: CPU 1: Hot: hi: 186,
> btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0
>
> ####
>
> Since the upgrade was unsucessful they had to revert back to old Oracle
> version ( 10.2.0.2 ) and we do not see any OOM issues on running Oracle
> system , so the OOM messages came up only during Oracle patch upgrade.
>
> The customer  opened a ticket with SAP and they have analysed the entire
> logs ( Oracle/OS ) and had suggested if we could tweak OS setting and
> OOM-Killer parameter , they could not see why the system consumed a lot of
> memory neither could we.
>
> I found  TID# 7002775 which explains  "Overcommit memory in SLES"
>
> and something on Postgres which discusses similar issue and recomends
> changing OOM killer paramater
>
> http://developer.postgresql.org/pgdocs/postgres/kernel-resources.html
>
>
> I would like to know if there are any tuning paramaters for OOM that can be
> used in such circumstances and if any one has experienced such an issue
> while upgrading Oracle 10gR2 on SLES 10SP2 ?
>
>
> Thanks
> Austin
>
> PS: If any further information is required on this I can provide the same.
>

It's weird to see that kind of machine getting OOM when aparently
nothing is running. Do you have SAR statistics?, enabling them is a
must, or better using collectl as it also saves per process data.

Regards,

-- 
Ciro Iriarte
http://cyruspy.wordpress.com
--
_______________________________________________
suse-oracle mailing list
[email protected]
http://listx.novell.com/mailman/listinfo/suse-oracle