Re: extreme system load [kswapd]

Hello,

Maybe, try to track IO on your machine (iostat) to see if it is an IO
problem (i.e. write to disk) or not.


On Tue, Mar 20, 2012 at 10:08, Karol =C5=A0ebesta <sebesta.karol@gmail.=
com> wrote:
>
> Hello @All,
>
> We have a problem on our production machine with high CPU utilization
> caused by kswapd3 daemon. Server is 128GB of physical memory and 81GB
> of SWAP.
>
>
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.6 (Tikanga)
> #
>
>
> # uptime
> =C2=A009:52:13 up 38 days, 17:01, 14 users, =C2=A0load average: 43.93=
, 50.02, 51.36
> #
>
>
> # free -m
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 total =C2=A0 =C2=A0 =C2=A0 =
used =C2=A0 =C2=A0 =C2=A0 free =C2=A0 =C2=A0 shared =C2=A0 =C2=A0buffer=
s =C2=A0 =C2=A0 cached
> Mem: =C2=A0 =C2=A0 =C2=A0 =C2=A0128989 =C2=A0 =C2=A0 =C2=A075577 =C2=A0=
 =C2=A0 =C2=A053412 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0416 =C2=A0 =C2=A0 =C2=A057131
> -/+ buffers/cache: =C2=A0 =C2=A0 =C2=A018029 =C2=A0 =C2=A0 110960
> Swap: =C2=A0 =C2=A0 =C2=A0 =C2=A081919 =C2=A0 =C2=A0 =C2=A031310 =C2=A0=
 =C2=A0 =C2=A050609
> #
>
> ### from TOP:
>
> Tasks: 1590 total, =C2=A0 3 running, 1578 sleeping, =C2=A0 1 stopped,=
 =C2=A0 8 zombie
> Cpu(s): =C2=A01.1%us, =C2=A01.9%sy, =C2=A00.0%ni, 74.4%id, 22.4%wa, =C2=
=A00.0%hi, =C2=A00.2%si, =C2=A00.0%st
> Mem: =C2=A0132085528k total, 77397544k used, 54687984k free, =C2=A0 4=
26472k buffers
> Swap: 83886064k total, 32043168k used, 51842896k free, 58475692k cach=
ed
>
> =C2=A0PID USER =C2=A0 =C2=A0 =C2=A0PR =C2=A0NI =C2=A0VIRT =C2=A0RES =C2=
=A0SHR S %CPU %MEM =C2=A0 =C2=A0TIME+ =C2=A0COMMAND
> =C2=A01816 root =C2=A0 =C2=A0 =C2=A010 =C2=A0-5 =C2=A0 =C2=A0 0 =C2=A0=
 =C2=A00 =C2=A0 =C2=A00 D 203.0 =C2=A00.0 =C2=A0 1124:39 kswapd3
> 10801 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 =C2=A0113m =C2=A023m 1680 =
S 158.6 =C2=A00.0 =C2=A0 1694:13 vxconfigd
> =C2=A03925 root =C2=A0 =C2=A0 =C2=A0RT =C2=A0 0 =C2=A0162m =C2=A015m =
3368 S 140.1 =C2=A00.0 =C2=A0 1313:32 multipathd
> 21399 wsy5067 =C2=A0 16 =C2=A0 0 22768 3192 1704 S 50.3 =C2=A00.0 =C2=
=A0 0:12.96 top
> 10583 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=
=A00 =C2=A0 =C2=A00 S 43.8 =C2=A00.0 =C2=A058:56.55 dmp_daemon
> 16673 root =C2=A0 =C2=A0 =C2=A016 =C2=A0 0 17720 1220 =C2=A0792 S 37.=
3 =C2=A00.0 732:23.49 cmafcad
> 24636 oracle =C2=A0 =C2=A015 =C2=A0 0 3314m =C2=A041m =C2=A040m S 26.=
6 =C2=A00.0 =C2=A029:31.84 oracle
> =C2=A0487 oracle =C2=A0 =C2=A015 =C2=A0 0 3301m =C2=A090m =C2=A088m S=
 21.4 =C2=A00.1 =C2=A0 1:05.68 oracle
> 14197 root =C2=A0 =C2=A0 =C2=A018 =C2=A0 0 =C2=A0315m =C2=A015m 3984 =
S 21.1 =C2=A00.0 346:15.70 vxsvc
> 26071 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 23900 6788 2092 S 19.8 =C2=
=A00.0 =C2=A012:31.44 MountAgent
> 16418 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 =C2=A05308 3656 =C2=A0592 =
S 19.1 =C2=A00.0 544:54.78 cmahostd
> =C2=A0485 oracle =C2=A0 =C2=A015 =C2=A0 0 3301m =C2=A087m =C2=A087m S=
 18.8 =C2=A00.1 =C2=A0 1:03.92 oracle
> 21456 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=
=A00 =C2=A0 =C2=A00 S 18.5 =C2=A00.0 =C2=A014:16.09 pdflush
> =C2=A04699 root =C2=A0 =C2=A0 =C2=A022 =C2=A0 0 65204 8420 1568 S 15.=
6 =C2=A00.0 137:15.75 vxdclid
> 16603 root =C2=A0 =C2=A0 =C2=A018 =C2=A0 0 =C2=A08820 6808 =C2=A0332 =
D 14.3 =C2=A00.0 =C2=A0 1290:32 cmaperfd
> 22047 tlmcron1 =C2=A018 =C2=A0 0 1227m 115m 6812 S 12.3 =C2=A00.1 554=
:55.86 java
> =C2=A06157 oracle =C2=A0 =C2=A016 =C2=A0 0 3302m 118m 115m D 11.4 =C2=
=A00.1 =C2=A0 0:01.29 oracle
> 30669 oracle =C2=A0 =C2=A016 =C2=A0 0 2066m 141m 134m D 10.7 =C2=A00.=
1 =C2=A0 0:20.19 oracle
> =C2=A0357 tlmtst3 =C2=A0 18 =C2=A0 0 =C2=A0101m =C2=A015m 9292 S =C2=A0=
9.7 =C2=A00.0 =C2=A0 0:29.43 stlfetch
> 15005 oracle =C2=A0 =C2=A016 =C2=A0 0 2046m =C2=A084m =C2=A075m D =C2=
=A09.7 =C2=A00.1 =C2=A0 0:01.75 oracle
> 32744 tlmtst3 =C2=A0 18 =C2=A0 0 =C2=A0101m =C2=A015m 9292 S =C2=A08.=
8 =C2=A00.0 =C2=A0 0:31.52 stlfetch
> 26077 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 12116 4228 1936 S =C2=A07.=
8 =C2=A00.0 =C2=A0 4:26.71 VolumeAgent
> 17955 oracle =C2=A0 =C2=A016 =C2=A0 0 3301m =C2=A028m =C2=A025m D =C2=
=A07.1 =C2=A00.0 =C2=A0 0:00.22 oracle
> 26076 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 23268 5284 2224 S =C2=A06.=
8 =C2=A00.0 =C2=A0 4:00.50 OracleAgent
> =C2=A0111 root =C2=A0 =C2=A0 =C2=A010 =C2=A0-5 =C2=A0 =C2=A0 0 =C2=A0=
 =C2=A00 =C2=A0 =C2=A00 S =C2=A06.5 =C2=A00.0 =C2=A0 4:57.32 events/13
> =C2=A05112 oracle =C2=A0 =C2=A016 =C2=A0 0 3314m =C2=A035m =C2=A035m =
S =C2=A06.5 =C2=A00.0 =C2=A010:38.55 oracle
> 10440 oracle =C2=A0 =C2=A016 =C2=A0 0 3314m =C2=A036m =C2=A036m S =C2=
=A06.5 =C2=A00.0 =C2=A010:54.65 oracle
> 26074 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 23012 4904 2136 S =C2=A06.=
5 =C2=A00.0 =C2=A0 3:09.46 NetlsnrAgent
> =C2=A08631 oracle =C2=A0 =C2=A016 =C2=A0 0 3315m =C2=A037m =C2=A036m =
S =C2=A06.2 =C2=A00.0 =C2=A012:04.57 oracle
> 26636 oracle =C2=A0 =C2=A015 =C2=A0 0 3316m =C2=A040m =C2=A039m S =C2=
=A06.2 =C2=A00.0 =C2=A013:07.75 oracle
> 10780 oracle =C2=A0 =C2=A016 =C2=A0 0 6386m =C2=A039m =C2=A039m D =C2=
=A05.8 =C2=A00.0 =C2=A010:57.01 oracle
> 13091 tlmtst5 =C2=A0 15 =C2=A0 0 =C2=A0423m =C2=A018m =C2=A013m S =C2=
=A05.8 =C2=A00.0 =C2=A0 0:02.76 gedit
> 17212 oracle =C2=A0 =C2=A015 =C2=A0 0 2050m =C2=A039m =C2=A039m S =C2=
=A05.8 =C2=A00.0 =C2=A011:30.82 oracle
> 20283 oracle =C2=A0 =C2=A015 =C2=A0 0 3314m =C2=A041m =C2=A040m S =C2=
=A05.8 =C2=A00.0 =C2=A010:49.00 oracle
> 13147 oracle =C2=A0 =C2=A015 =C2=A0 0 3314m =C2=A043m =C2=A043m S =C2=
=A05.5 =C2=A00.0 =C2=A010:43.12 oracle
> 16875 oracle =C2=A0 =C2=A015 =C2=A0 0 3316m =C2=A042m =C2=A040m S =C2=
=A05.5 =C2=A00.0 =C2=A022:52.66 oracle
> 28143 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 =C2=A0111m 6700 3396 S =C2=
=A05.5 =C2=A00.0 143:25.46 opcmona
> =C2=A05718 oracle =C2=A0 =C2=A016 =C2=A0 0 3314m =C2=A038m =C2=A036m =
S =C2=A05.2 =C2=A00.0 =C2=A011:30.95 oracle
> 22163 oracle =C2=A0 =C2=A015 =C2=A0 0 3316m =C2=A039m =C2=A038m S =C2=
=A05.2 =C2=A00.0 =C2=A032:51.94 oracle
> 26065 root =C2=A0 =C2=A0 =C2=A015 =C2=A0 0 22960 4724 2164 S =C2=A04.=
5 =C2=A00.0 =C2=A0 1:53.23 ApplicationAgen
> 26612 oracle =C2=A0 =C2=A018 =C2=A0 0 3302m =C2=A022m =C2=A019m S =C2=
=A04.2 =C2=A00.0 =C2=A023:31.48 oracle
> =C2=A08597 oracle =C2=A0 =C2=A018 =C2=A0 0 3302m =C2=A018m =C2=A016m =
S =C2=A03.9 =C2=A00.0 =C2=A023:30.62 oracle
> 10690 oracle =C2=A0 =C2=A018 =C2=A0 0 6373m =C2=A020m =C2=A016m S =C2=
=A03.9 =C2=A00.0 =C2=A023:28.74 oracle
> 11822 oracle =C2=A0 =C2=A018 =C2=A0 0 2038m =C2=A029m =C2=A025m S =C2=
=A03.9 =C2=A00.0 =C2=A0 0:05.17 oracle
>
> Why the kswapd daemon causing so high CPU utilization? Can it be
> caused by some process which try to access memory pages which are not
> in memory, but in swap? Or can it be cause by swapping for exapmple
> some inactive oracle database SGA from memory to swap?
>
> Thank you for help.
>
>
> Regards,
>
> Karol Sebesta
> --
> To unsubscribe from this list: send the line "unsubscribe linux-admin=
" in
> the body of a message to [email protected]
> More majordomo info at =C2=A0http://vger.kernel.org/majordomo-info.ht=
ml




--

Nicolas Maupu - SS2J - Excilys
T=C3=A9l :+33 (0) 1 41 24 43 27