SLES 11 SP2 High-CPU usage on migration/X threads
[email protected] Wed, 23 Jan 2013 11:41:52 +0100
| Newsgroups | gmane.linux.suse.oracle.general |
|---|---|
| Message-ID | <OFD1BC0943.B6CC6609-ONC1257AFC.0039AF1F-C1257AFC.003AC430@bgnet.de> |
Dies ist eine mehrteilige Nachricht im MIME-Format.
--===============0782566167==
Content-Type: multipart/alternative; boundary="=_alternative 003AC426C1257AFC_="
Dies ist eine mehrteilige Nachricht im MIME-Format.
--=_alternative 003AC426C1257AFC_=
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi Guys,
I have some trouble on my SLES 11 SP2 machines. The machines are installed =
on a VMWare ESXi (latest stable version).
machine:~ # ps aux | grep migration
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6 93.0 0.0 0 0 ? =
S 2012 93550:09 [migration/0]
root 8 96.5 0.0 0 0 ? =
S 2012 97016:21 [migration/1]
root 13 94.8 0.0 0 0 ? =
S 2012 95365:44 [migration/2]
root 17 95.6 0.0 0 0 ? =
S 2012 96182:33 [migration/3]
"top" shows the same stats (CPU Time+), but %CPU is 0.
Linux lb40016 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012=20
(d73692b) x86=5F64 x86=5F64 x86=5F64 GNU/inux
Nothing interesting in the logfiles (/var/log/messages, etc). The machine=20
was not vMotioned. It runs some hours, then the migration threads going to =
this point. The actual vmware tools are installed.
after execute "ps aux | grep migration"
machine:~ # mpstat -P ALL
Linux 3.0.13-0.27-default 01/23/13 =5Fx86=5F64=5F
11:36:23 CPU %usr %nice %sys %iowait %irq %soft %steal=20
%guest %idle
11:36:23 all 0.33 0.00 0.03 0.00 0.00 0.00 0.00 =20
0.00 99.63
11:36:23 0 0.30 0.00 0.05 0.01 0.00 0.00 0.00 =20
0.00 99.64
11:36:23 1 0.31 0.01 0.03 0.00 0.00 0.00 0.00 =20
0.00 99.65
11:36:23 2 0.37 0.00 0.03 0.00 0.00 0.00 0.00 =20
0.00 99.59
11:36:23 3 0.34 0.00 0.03 0.00 0.00 0.00 0.00 =20
0.00 99.63
machine:~ # sar -u 2 5
Linux 3.0.13-0.27-default 01/23/13 =5Fx86=5F64=5F
11:38:17 CPU %user %nice %system %iowait %steal=20
%idle
11:38:19 all 0.25 0.00 0.13 0.00 0.00=20
99.62
11:38:21 all 0.00 0.00 0.00 0.00 0.00=20
100.00
11:38:23 all 0.00 0.00 0.13 0.00 0.00=20
99.87
11:38:25 all 0.00 0.00 0.00 0.00 0.00=20
100.00
11:38:27 all 0.00 0.00 0.13 0.00 0.00=20
99.87
Average: all 0.05 0.00 0.08 0.00 0.00=20
99.87
machine:~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 20156 0 0 0 IO-APIC-edge timer
1: 8 0 0 0 IO-APIC-edge i8042
3: 0 1 0 0 IO-APIC-edge
4: 0 0 0 1 IO-APIC-edge
6: 5 0 0 0 IO-APIC-edge floppy
7: 0 0 0 0 IO-APIC-edge parport0
8: 17 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 137 0 0 0 IO-APIC-edge i8042
14: 0 0 0 0 IO-APIC-edge ata=5Fpiix
15: 1767460 2377356 45 2282 IO-APIC-edge ata=5Fpiix
17: 1866 1300755 21061 2746 IO-APIC-fasteoi ioc0
18: 9141485 0 0 11 IO-APIC-fasteoi eth0
40: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
41: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
42: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
43: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
44: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
45: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
46: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
47: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
48: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
49: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
50: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
51: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
52: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
53: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
54: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
55: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
56: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
57: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
58: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
59: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
60: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
61: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
62: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
63: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
64: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
65: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
66: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
67: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
68: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
69: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
70: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
71: 0 0 0 0 PCI-MSI-edge PCIe=20
PME, pciehp
72: 11 0 0 0 PCI-MSI-edge vmci
73: 0 0 0 0 PCI-MSI-edge vmci
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 52661861 53572304 61909856 60410143 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring=20
interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 6948002 7854897 5876118 5547225 Rescheduling interrupts
CAL: 855295 1999 136878 138016 Function call=20
interrupts
TLB: 178002 356505 228928 210113 TLB shootdowns
TRM: 0 0 0 0 Thermal event=20
interrupts
THR: 0 0 0 0 Threshold APIC=20
interrupts
MCE: 0 0 0 0 Machine check=20
exceptions
MCP: 20107 20107 20107 20107 Machine check polls
ERR: 0
MIS: 0
Does anyone have an idea?
Franz Kinader
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
BG-Phoenics GmbH
Abteilung Betrieb
Loristra=DFe 6 a
80335 M=FCnchen
Fon: +49 (0) 89-12179-92 48
Fax: +49 (0) 89-12179-9 99
Mobil: +49 (0) 173-618-14 19
www.bg-phoenics.de
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Sitz der Gesellschaft: Hannover
Handelsregistergericht: Amtsgericht Hannover
HRB Nr.: 59345
Gesch=E4ftsf=FChrer: Burkhard Wolf (Vorsitz), Walter Lerch
--=_alternative 003AC426C1257AFC_=
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable
<font size=3D2 face=3D"sans-serif">Hi Guys,</font>
<br>
<br><font size=3D2 face=3D"sans-serif">I have some trouble on my SLES 11 SP2
machines. The machines are installed on a VMWare ESXi (latest stable versio=
n).</font>
<br>
<br><font size=3D2 face=3D"sans-serif">machine:~ # ps aux | grep migration<=
/font>
<br><font size=3D2 face=3D"sans-serif">USER PID %CPU %=
MEM
VSZ RSS TTY STAT START TIME
COMMAND</font>
<br><font size=3D2 face=3D"sans-serif">root &nb=
sp;
6 93.0 0.0
0 0
? S
2012 93550:09 [migration/0]</font>
<br><font size=3D2 face=3D"sans-serif">root &nb=
sp;
8 96.5 0.0
0 0
? S
2012 97016:21 [migration/1]</font>
<br><font size=3D2 face=3D"sans-serif">root &nb=
sp;
13 94.8 0.0
0 0
? S
2012 95365:44 [migration/2]</font>
<br><font size=3D2 face=3D"sans-serif">root &nb=
sp;
17 95.6 0.0
0 0
? S
2012 96182:33 [migration/3]</font>
<br>
<br><font size=3D2 face=3D"sans-serif">"top" shows the same stats
(CPU Time+), but %CPU is 0.</font>
<br>
<br><font size=3D2 face=3D"sans-serif">Linux lb40016 3.0.13-0.27-default #1
SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86=5F64 x86=5F64 x86=5F64 GNU/i=
nux</font>
<br>
<br><font size=3D2 face=3D"sans-serif">Nothing interesting in the logfiles
(/var/log/messages, etc). The machine was not vMotioned. It runs some hours,
then the migration threads going to this point. The actual vmware tools
are installed.</font>
<br>
<br><font size=3D2 face=3D"sans-serif">after execute "ps aux | grep mi=
gration"</font>
<br>
<br><font size=3D2 face=3D"sans-serif">machine:~ # mpstat -P ALL</font>
<br><font size=3D2 face=3D"sans-serif">Linux 3.0.13-0.27-default &nb=
sp;
01/23/13 =5Fx86=5F64=5F</font>
<br>
<br><font size=3D2 face=3D"sans-serif">11:36:23 CPU &n=
bsp;%usr
%nice %sys %iowait %irq %soft  =
;%steal
%guest %idle</font>
<br><font size=3D2 face=3D"sans-serif">11:36:23 all &n=
bsp;0.33
0.00 0.03 0.00 0.00
0.00 0.00 0.00 99.63</font>
<br><font size=3D2 face=3D"sans-serif">11:36:23 0 &nbs=
p;
0.30 0.00 0.05 0.01
0.00 0.00 0.00 0.00
99.64</font>
<br><font size=3D2 face=3D"sans-serif">11:36:23 1 &nbs=
p;
0.31 0.01 0.03 0.00
0.00 0.00 0.00 0.00
99.65</font>
<br><font size=3D2 face=3D"sans-serif">11:36:23 2 &nbs=
p;
0.37 0.00 0.03 0.00
0.00 0.00 0.00 0.00
99.59</font>
<br><font size=3D2 face=3D"sans-serif">11:36:23 3 &nbs=
p;
0.34 0.00 0.03 0.00
0.00 0.00 0.00 0.00
99.63</font>
<br>
<br><font size=3D2 face=3D"sans-serif">machine:~ # sar -u 2 5</font>
<br><font size=3D2 face=3D"sans-serif">Linux 3.0.13-0.27-default &nb=
sp;
01/23/13 =5Fx86=5F64=5F</font>
<br>
<br><font size=3D2 face=3D"sans-serif">11:38:17 =
CPU
%user %nice %system %iowait
%steal %idle</font>
<br><font size=3D2 face=3D"sans-serif">11:38:19 =
all
0.25 0.00 0.13
0.00 0.00 99.62</font>
<br><font size=3D2 face=3D"sans-serif">11:38:21 =
all
0.00 0.00 0.00
0.00 0.00 100.00</font>
<br><font size=3D2 face=3D"sans-serif">11:38:23 =
all
0.00 0.00 0.13
0.00 0.00 99.87</font>
<br><font size=3D2 face=3D"sans-serif">11:38:25 =
all
0.00 0.00 0.00
0.00 0.00 100.00</font>
<br><font size=3D2 face=3D"sans-serif">11:38:27 =
all
0.00 0.00 0.13
0.00 0.00 99.87</font>
<br><font size=3D2 face=3D"sans-serif">Average: =
all
0.05 0.00 0.08
0.00 0.00 99.87</font>
<br>
<br><font size=3D2 face=3D"sans-serif">machine:~ # cat /proc/interrupts</fo=
nt>
<br><font size=3D2 face=3D"sans-serif"> &=
nbsp;CPU0
CPU1 CPU2
CPU3</font>
<br><font size=3D2 face=3D"sans-serif"> 0: 20156
0 0
0 IO-APIC-edge  =
;timer</font>
<br><font size=3D2 face=3D"sans-serif"> 1:
8 0
0 0 IO-APIC-edge
i8042</font>
<br><font size=3D2 face=3D"sans-serif"> 3:
0 1
0 0 IO-APIC-edge</font>
<br><font size=3D2 face=3D"sans-serif"> 4:
0 0
0 1 IO-APIC-edge</font>
<br><font size=3D2 face=3D"sans-serif"> 6:
5 0
0 0 IO-APIC-edge
floppy</font>
<br><font size=3D2 face=3D"sans-serif"> 7:
0 0
0 0 IO-APIC-edge
parport0</font>
<br><font size=3D2 face=3D"sans-serif"> 8:
17 0 0
0 IO-APIC-edge  =
;rtc0</font>
<br><font size=3D2 face=3D"sans-serif"> 9:
0 0
0 0 IO-APIC-fasteoi
acpi</font>
<br><font size=3D2 face=3D"sans-serif"> 12:  =
;137
0 0
0 IO-APIC-edge  =
;i8042</font>
<br><font size=3D2 face=3D"sans-serif"> 14:
0 0
0 0 IO-APIC-edge
ata=5Fpiix</font>
<br><font size=3D2 face=3D"sans-serif"> 15: 1767460
2377356 45 2282
IO-APIC-edge ata=5Fpiix</font>
<br><font size=3D2 face=3D"sans-serif"> 17: 1866
1300755 21061 2746
IO-APIC-fasteoi ioc0</font>
<br><font size=3D2 face=3D"sans-serif"> 18: 9141485
0 0
11 IO-APIC-fasteoi eth0</font>
<br><font size=3D2 face=3D"sans-serif"> 40:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 41:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 42:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 43:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 44:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 45:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 46:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 47:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 48:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 49:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 50:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 51:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 52:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 53:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 54:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 55:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 56:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 57:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 58:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 59:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 60:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 61:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 62:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 63:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 64:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 65:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 66:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 67:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 68:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 69:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 70:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 71:
0 0
0 0 PCI-MSI-edge
PCIe PME, pciehp</font>
<br><font size=3D2 face=3D"sans-serif"> 72:
11 0 0
0 PCI-MSI-edge  =
;vmci</font>
<br><font size=3D2 face=3D"sans-serif"> 73:
0 0
0 0 PCI-MSI-edge
vmci</font>
<br><font size=3D2 face=3D"sans-serif">NMI: &nb=
sp;0
0 0
0 Non-maskable interrupts</font>
<br><font size=3D2 face=3D"sans-serif">LOC: 52661861 53572304
61909856 60410143 Local timer interrupts</font>
<br><font size=3D2 face=3D"sans-serif">SPU: &nb=
sp;0
0 0
0 Spurious interrupts</font>
<br><font size=3D2 face=3D"sans-serif">PMI: &nb=
sp;0
0 0
0 Performance monitoring interrupt=
s</font>
<br><font size=3D2 face=3D"sans-serif">IWI: &nb=
sp;0
0 0
0 IRQ work interrupts</font>
<br><font size=3D2 face=3D"sans-serif">RES: 6948002 &nb=
sp;7854897
5876118 5547225 Rescheduling interrupts</f=
ont>
<br><font size=3D2 face=3D"sans-serif">CAL: 855295 &nb=
sp;
1999 136878 138016 Function call
interrupts</font>
<br><font size=3D2 face=3D"sans-serif">TLB: 178002 &nb=
sp;
356505 228928 210113 TLB shootdowns</fon=
t>
<br><font size=3D2 face=3D"sans-serif">TRM: &nb=
sp;0
0 0
0 Thermal event interrupts</font>
<br><font size=3D2 face=3D"sans-serif">THR: &nb=
sp;0
0 0
0 Threshold APIC interrupts</font>
<br><font size=3D2 face=3D"sans-serif">MCE: &nb=
sp;0
0 0
0 Machine check exceptions</font>
<br><font size=3D2 face=3D"sans-serif">MCP: 20107
20107 20107 20107
Machine check polls</font>
<br><font size=3D2 face=3D"sans-serif">ERR: &nb=
sp;0</font>
<br><font size=3D2 face=3D"sans-serif">MIS: &nb=
sp;0</font>
<br>
<br><font size=3D2 face=3D"sans-serif">Does anyone have an idea?</font>
<br><font size=3D2 face=3D"sans-serif"><br>
Franz Kinader</font>
<br><font size=3D1 color=3D#808080 face=3D"sans-serif">=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F<br>
<br>
BG-Phoenics GmbH<br>
Abteilung Betrieb<br>
Loristra=DFe 6 a<br>
80335 M=FCnchen<br>
<br>
Fon: +49 (0) 89-12179-92 48<br>
Fax: +49 (0) 89-12179-9 99<br>
Mobil: +49 (0) 173-618-14 19<br>
<br>
</font><a href=3D"www.bg-phoenics.de"><font size=3D1 color=3D#808080 face=
=3D"sans-serif">www.bg-phoenics.de</font></a><font size=3D1 color=3D#808080=
face=3D"sans-serif"><br>
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F</font>
<br><font size=3D1 color=3D#808080 face=3D"sans-serif">Sitz der Gesellschaf=
t:
Hannover<br>
Handelsregistergericht: Amtsgericht Hannover<br>
HRB Nr.: 59345<br>
Gesch=E4ftsf=FChrer: Burkhard Wolf (Vorsitz), Walter Lerch<br>
</font>
--=_alternative 003AC426C1257AFC_=--
--===============0782566167==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
suse-sles-e mailing list
[email protected]
http://listx.novell.com/mailman/listinfo/suse-sles-e
--===============0782566167==--