RE: FTO Standby TEMS high cpu during sync

<[email protected]>
Newsgroups gmane.comp.sysutils.tivoli.general
Message-ID <EA5FD7F404DE694BB310D050B12764960D982C8E__23798.3535197147$1360257546$gmane$org@WFS0127A.corp1.ad1.seb.net>
Hi.

Had sustained cpu at 100% but after I deleted deploy statistics it peaks at whatever available cpu during sync cycle.

/Gus

From: [email protected] [mailto:[email protected]] On Behalf Of Robin.Smith-U3F5sf8KFGKCmk9zJvdQaQC/[email protected]
Sent: den 6 februari 2013 17:23
To: [email protected]
Subject: RE: [TME10] FTO Standby TEMS high cpu during sync

Wasn't that fixed in 6.22?  It's worth having a look though and checking that there are no deployments clogging up the system. But the symptoms for that are sustained CPU, as I saw it out at a customer

________________________________
From: [email protected] [mailto:[email protected]] On Behalf Of John Alvord
Sent: 06 February 2013 14:01
To: Discussion list for Tivoli product and Tivoli Ready products.
Subject: RE: [TME10] FTO Standby TEMS high cpu during sync
The most recent case like that I worked on.... was leftover remote deploy status data.

Regards,




John Alvord - Ph: 1-720-396-2788    Cell: none
Customer Support - Tivoli Software - jalvord-r/[email protected]<mailto:[email protected]>
Advisory Engineer - Tivoli Monitoring - ITM infrastructure
Follow us on Twitter! @Tivolisupport<http://twitter.com/TivoliSupport> and Facebook<https://www.facebook.com/IBMTivoliSupport>
Personalize your support needs with the IBM Support Portal<http://www.ibm.com/support/electronicsupport/>
Use Service Request<http://www.ibm.com/software/support/servicerequest/quick_start.html> to get assistance!
For emails regarding a PMR, copy [email protected]<mailto:[email protected]>
For secure browser uploads: https://www.ecurep.ibm.com/app/upload
For Customer Support guidance: IBM Software Support Handbook<http://www14.software.ibm.com/webapp/set2/sas/f/handbook/home.html>









From:        <[email protected]>
To:        <[email protected]>,
Date:        02/05/2013 11:11 PM
Subject:        RE: [TME10] FTO Standby TEMS high cpu during sync
Sent by:        [email protected]
________________________________




John, it’s the standby HUB the acting is not having any cpu issues at all. And it’s 100% related to it pulling data from the acting.

/Gus

From: [email protected] [mailto:[email protected]] On Behalf Of John Alvord
Sent: den 5 februari 2013 18:13
To: Discussion list for Tivoli product and Tivoli Ready products.
Subject: RE: [TME10] FTO Standby TEMS high cpu during sync

High CPU is always a concern for a TEMS. TEMS is a real time server and should also run low utilization so it can respond promptly,

Two recent cases where a situation created 100% CPU

1) 6K Agents and customer created a MS_Online with sampling interval 5 minutes - wham 100% CPU
2) 14K Agents, MS_Offline with persist=2 and sampling interval 5 minutes - wham 100% CPU.

so I would seriously suggest a PMR and an investigation.

Regards,






John Alvord - Ph: 1-720-396-2788    Cell: none
Customer Support - Tivoli Software - jalvord-r/[email protected]<mailto:[email protected]>
Advisory Engineer - Tivoli Monitoring - ITM infrastructure
Follow us on Twitter! @Tivolisupport<http://twitter.com/TivoliSupport> and Facebook<https://www.facebook.com/IBMTivoliSupport>
Personalize your support needs with the IBM Support Portal<http://www.ibm.com/support/electronicsupport/>
Use Service Request<http://www.ibm.com/software/support/servicerequest/quick_start.html> to get assistance!
For emails regarding a PMR, copy [email protected]<mailto:[email protected]>
For secure browser uploads: https://www.ecurep.ibm.com/app/upload
For Customer Support guidance: IBM Software Support Handbook<http://www14.software.ibm.com/webapp/set2/sas/f/handbook/home.html>










From:        <Robin.Smith-U3F5sf8KFGKCmk9zJvdQaQC/[email protected]<mailto:[email protected]>>
To:        <[email protected]<mailto:[email protected]>>,
Date:        02/05/2013 04:21 AM
Subject:        RE: [TME10] FTO Standby TEMS high cpu during sync
Sent by:        [email protected]<mailto:[email protected]>

________________________________




It's never good to have a CPU situation, as it will generate false alarms due to the design of Unix.

It should either be for an extended period of time or for a combination of other factors

e.g

if CPU > 95% for over an hour then alert

or something smarter like if CPU > 95% and CPU run queue/divided number of processors > 5

________________________________

From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of [email protected]<mailto:[email protected]>
Sent: 05 February 2013 11:33
To: [email protected]<mailto:[email protected]>
Subject: RE: [TME10] FTO Standby TEMS high cpu during sync

Right, the worries I have is that our def. cpu situation fires on this one (-:
I have now excluded this hostname from the situation.

As long as the cpu drops and the sync completes I can probably forget about it.

/Gus

From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of [email protected]<mailto:Robin.Smith-U3F5sf8KFGKCmk9zJvdQaQC/[email protected]>
Sent: den 5 februari 2013 12:02
To: [email protected]<mailto:[email protected]>
Subject: RE: [TME10] FTO Standby TEMS high cpu during sync

I am not sure you have much to worry about to be honest, Unix is designed to give as much CPU out as possible, especially when the system is quiet.  If it was running at constant 100%, and the run queue (column marked q below) was high, then you may have an issue.  Unfortunately most of the CPU is User (Us Column below), which means its application bound, and there is not much you can do on the system to tweak this.


________________________________


From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of [email protected]<mailto:[email protected]>
Sent: 04 February 2013 15:57
To: [email protected]<mailto:[email protected]>
Subject: RE: [TME10] FTO Standby TEMS high cpu during sync
This sync it peaked around 75% But I’ve seen 98-99 all day.

[tems@mytems bin]$ vmstat 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
3  0  20768 228888 546120 2236612    0    0    11   650   17    3 56  1 32 11  0
2  0  20768 228756 546120 2236640    0    0     0   876 1280 1160 99  1  0  0  0
1  1  20768 228756 546124 2236652    0    0     0   912 1284 1158 100  0  0  0  0
2  0  20768 228756 546124 2236652    0    0     0   816 1273 1065 100  0  0  0  0
9  0  20768 165856 546124 2236652    0    0     0   680 1256 2113 64 36  0  0  0
5  0  20768 209272 546124 2236660    0    0     0     0 1081 2483 13 87  0  0  0
1  0  20768 228440 546124 2236664    0    0     0  1308 1202 1393 69 13  6 12  0
1  0  20768 228440 546124 2236664    0    0     0  1608 1246 1476 76  1  8 15  0
0  1  20768 228440 546124 2236664    0    0     0  1460 1205 1387 75  1  9 15  0
1  0  20768 228316 546124 2236668    0    0     0  1560 1209 1453 75  1  7 17  0
[tems@lsp2052a bin]$ iostat 1 4
Linux 2.6.32-220.el6.x86_64 (mytems)  02/04/2013      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
         56.38    0.01    0.74   11.16    0.00   31.71

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.00         7.05        11.96   12944788   21970748
sdb             157.53        15.10      1288.35   27741684 2367079160
dm-0              0.29         2.61         1.90    4788242    3491296
dm-1              0.01         0.02         0.04      36720      77336
dm-2            161.18        15.08      1288.19   27698026 2366780928
dm-3              0.02         0.02         0.16      42978     298232
dm-4              0.10         1.29         0.58    2363362    1056672
dm-5              0.15         0.11         1.14     195666    2101272
dm-6              0.81         2.70         6.33    4957554   11621328
dm-7              0.26         0.24         1.97     433922    3622520

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
         74.26    0.00    0.99   15.84    0.00    8.91

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.99        15.84         0.00         16          0
sdb             380.20         0.00      3104.95          0       3136
dm-0              0.00         0.00         0.00          0          0
dm-1              0.00         0.00         0.00          0          0
dm-2            388.12         0.00      3104.95          0       3136
dm-3              0.00         0.00         0.00          0          0
dm-4              0.00         0.00         0.00          0          0
dm-5              0.00         0.00         0.00          0          0
dm-6              0.00         0.00         0.00          0          0
dm-7              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
         75.76    0.00    1.01   16.16    0.00    7.07

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.00         0.00         0.00          0          0
sdb             363.64         0.00      2933.33          0       2904
dm-0              0.00         0.00         0.00          0          0
dm-1              0.00         0.00         0.00          0          0
dm-2            366.67         0.00      2933.33          0       2904
dm-3              0.00         0.00         0.00          0          0
dm-4              0.00         0.00         0.00          0          0
dm-5              0.00         0.00         0.00          0          0
dm-6              0.00         0.00         0.00          0          0
dm-7              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
         71.29    0.00    0.99   18.81    0.00    8.91

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.00         0.00         0.00          0          0
sdb             380.20         0.00      3065.35          0       3096
dm-0              0.00         0.00         0.00          0          0
dm-1              0.00         0.00         0.00          0          0
dm-2            383.17         0.00      3065.35          0       3096
dm-3              0.00         0.00         0.00          0          0
dm-4              0.00         0.00         0.00          0          0
dm-5              0.00         0.00         0.00          0          0
dm-6              0.00         0.00         0.00          0          0
dm-7              0.00         0.00         0.00          0          0

/Gus

From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of [email protected]<mailto:Robin.Smith-U3F5sf8KFGKCmk9zJvdQaQC/[email protected]>
Sent: den 4 februari 2013 11:21
To: [email protected]<mailto:[email protected]>
Subject: RE: [TME10] FTO Standby TEMS high cpu during sync

CPU running peaking is a bit vague, can you run the following command and post the out here next time it peaks, might help identify the bottleneck a bit further

vmstat 1 10

iostat 1 4



________________________________


From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of [email protected]<mailto:[email protected]>
Sent: 01 February 2013 15:40
To: [email protected]<mailto:[email protected]>
Subject: [TME10] FTO Standby TEMS high cpu during sync
Hi.

Everytime between

(510BDF05.0000-10:kdyctrl.cpp,3932,"rTEMSSynchThread") TEMS is MIRROR, Synchronizing with HUB TEMS tables.
And:
q(510BE00D.0000-10:kdyctrl.cpp,4045,"rTEMSSynchThread") Sychronization complete. Going to sleep.

The cpu peaks at 99% an nothing unusual in the *_ms_* logs. HUB on Linux RHEL6 6.2.3 FP02.
As soon as sync is complete it drops to close to nothing within seconds.

I need tips on tracing FTP sync, anyone?

/Gus



Lloyds TSB Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales, number 2065. Telephone: 020 7626 1500.

Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 327000.  Telephone: 020 7626 1500.

Lloyds TSB Scotland plc. Registered Office: Henry Duncan House, 120 George Street, Edinburgh EH2 4LH. Registered in Scotland, number 95237. Telephone: 0131 225 4555.

Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales, number 2299428. Telephone: 01452 372372.

Lloyds TSB Bank plc, Lloyds TSB Scotland plc, Bank of Scotland plc and Cheltenham & Gloucester plc are authorised and regulated by the Financial Services Authority.

Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds TSB Bank plc.

HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 218813.  Telephone: 020 7626 1500.

Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 95000. Telephone: 0131 225 4555

This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it  (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments.

Telephone calls may be monitored or recorded.

Lloyds TSB Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales, number 2065. Telephone: 020 7626 1500.

Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 327000.  Telephone: 020 7626 1500.

Lloyds TSB Scotland plc. Registered Office: Henry Duncan House, 120 George Street, Edinburgh EH2 4LH. Registered in Scotland, number 95237. Telephone: 0131 225 4555.

Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales, number 2299428. Telephone: 01452 372372.

Lloyds TSB Bank plc, Lloyds TSB Scotland plc, Bank of Scotland plc and Cheltenham & Gloucester plc are authorised and regulated by the Financial Services Authority.

Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds TSB Bank plc.

HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 218813.  Telephone: 020 7626 1500.

Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 95000. Telephone: 0131 225 4555

This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it  (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments.

Telephone calls may be monitored or recorded.

Lloyds TSB Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales, number 2065. Telephone: 020 7626 1500.

Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 327000.  Telephone: 020 7626 1500.

Lloyds TSB Scotland plc. Registered Office: Henry Duncan House, 120 George Street, Edinburgh EH2 4LH. Registered in Scotland, number 95237. Telephone: 0131 225 4555.

Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales, number 2299428. Telephone: 01452 372372.

Lloyds TSB Bank plc, Lloyds TSB Scotland plc, Bank of Scotland plc and Cheltenham & Gloucester plc are authorised and regulated by the Financial Services Authority.

Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds TSB Bank plc.

HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 218813.  Telephone: 020 7626 1500.

Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 95000. Telephone: 0131 225 4555

This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it  (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments.

Telephone calls may be monitored or recorded._______________________________________________
TME10 mailing list
[email protected]<mailto:[email protected]>
Unsubscribe:[email protected]_______________________________________________
TME10 mailing list
[email protected]
Unsubscribe:[email protected]

Lloyds TSB Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales, number 2065. Telephone: 020 7626 1500.

Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 327000.  Telephone: 020 7626 1500.

Lloyds TSB Scotland plc. Registered Office: Henry Duncan House, 120 George Street, Edinburgh EH2 4LH. Registered in Scotland, number 95237. Telephone: 0131 225 4555.

Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales, number 2299428. Telephone: 01452 372372.

Lloyds TSB Bank plc, Lloyds TSB Scotland plc, Bank of Scotland plc and Cheltenham & Gloucester plc are authorised and regulated by the Financial Services Authority.

Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds TSB Bank plc.

HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 218813.  Telephone: 020 7626 1500.

Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 95000. Telephone: 0131 225 4555

This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it  (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments.

Telephone calls may be monitored or recorded.

_______________________________________________
TME10 mailing list
[email protected]
Unsubscribe:[email protected]
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.