RE: FTO Standby TEMS high cpu during sync
John Alvord <jalvord-r/[email protected]>
| Newsgroups | gmane.comp.sysutils.tivoli.general |
|---|---|
| Message-ID | <OFF27D2AC6.BB1A64D4-ON88257B0A.004CE356-88257B0A.004CF35B__37672.0207430937$1360159337$gmane$org@us.ibm.com> |
The most recent case like that I worked on.... was leftover remote deploy status data. Regards, John Alvord - Ph: 1-720-396-2788 Cell: none Customer Support - Tivoli Software - jalvord-r/[email protected] Advisory Engineer - Tivoli Monitoring - ITM infrastructure Follow us on Twitter! @Tivolisupport and Facebook Personalize your support needs with the IBM Support Portal Use Service Request to get assistance! For emails regarding a PMR, copy [email protected] For secure browser uploads: https://www.ecurep.ibm.com/app/upload For Customer Support guidance: IBM Software Support Handbook From: <[email protected]> To: <[email protected]>, Date: 02/05/2013 11:11 PM Subject: RE: [TME10] FTO Standby TEMS high cpu during sync Sent by: [email protected] John, it?s the standby HUB the acting is not having any cpu issues at all. And it?s 100% related to it pulling data from the acting. /Gus From: [email protected] [ mailto:[email protected]] On Behalf Of John Alvord Sent: den 5 februari 2013 18:13 To: Discussion list for Tivoli product and Tivoli Ready products. Subject: RE: [TME10] FTO Standby TEMS high cpu during sync High CPU is always a concern for a TEMS. TEMS is a real time server and should also run low utilization so it can respond promptly, Two recent cases where a situation created 100% CPU 1) 6K Agents and customer created a MS_Online with sampling interval 5 minutes - wham 100% CPU 2) 14K Agents, MS_Offline with persist=2 and sampling interval 5 minutes - wham 100% CPU. so I would seriously suggest a PMR and an investigation. Regards, John Alvord - Ph: 1-720-396-2788 Cell: none Customer Support - Tivoli Software - jalvord-r/[email protected] Advisory Engineer - Tivoli Monitoring - ITM infrastructure Follow us on Twitter! @Tivolisupport and Facebook Personalize your support needs with the IBM Support Portal Use Service Request to get assistance! For emails regarding a PMR, copy [email protected] For secure browser uploads: https://www.ecurep.ibm.com/app/upload For Customer Support guidance: IBM Software Support Handbook From: <Robin.Smith-U3F5sf8KFGKCmk9zJvdQaQC/[email protected]> To: <[email protected]>, Date: 02/05/2013 04:21 AM Subject: RE: [TME10] FTO Standby TEMS high cpu during sync Sent by: [email protected] It's never good to have a CPU situation, as it will generate false alarms due to the design of Unix. It should either be for an extended period of time or for a combination of other factors e.g if CPU > 95% for over an hour then alert or something smarter like if CPU > 95% and CPU run queue/divided number of processors > 5 From: [email protected] [ mailto:[email protected]] On Behalf Of [email protected] Sent: 05 February 2013 11:33 To: [email protected] Subject: RE: [TME10] FTO Standby TEMS high cpu during sync Right, the worries I have is that our def. cpu situation fires on this one (-: I have now excluded this hostname from the situation. As long as the cpu drops and the sync completes I can probably forget about it. /Gus From: [email protected] [ mailto:[email protected]] On Behalf Of Robin.Smith-U3F5sf8KFGKCmk9zJvdQaQC/[email protected] Sent: den 5 februari 2013 12:02 To: [email protected] Subject: RE: [TME10] FTO Standby TEMS high cpu during sync I am not sure you have much to worry about to be honest, Unix is designed to give as much CPU out as possible, especially when the system is quiet. If it was running at constant 100%, and the run queue (column marked q below) was high, then you may have an issue. Unfortunately most of the CPU is User (Us Column below), which means its application bound, and there is not much you can do on the system to tweak this. From: [email protected] [ mailto:[email protected]] On Behalf Of [email protected] Sent: 04 February 2013 15:57 To: [email protected] Subject: RE: [TME10] FTO Standby TEMS high cpu during sync This sync it peaked around 75% But I?ve seen 98-99 all day. [tems@mytems bin]$ vmstat 1 10 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 20768 228888 546120 2236612 0 0 11 650 17 3 56 1 32 11 0 2 0 20768 228756 546120 2236640 0 0 0 876 1280 1160 99 1 0 0 0 1 1 20768 228756 546124 2236652 0 0 0 912 1284 1158 100 0 0 0 0 2 0 20768 228756 546124 2236652 0 0 0 816 1273 1065 100 0 0 0 0 9 0 20768 165856 546124 2236652 0 0 0 680 1256 2113 64 36 0 0 0 5 0 20768 209272 546124 2236660 0 0 0 0 1081 2483 13 87 0 0 0 1 0 20768 228440 546124 2236664 0 0 0 1308 1202 1393 69 13 6 12 0 1 0 20768 228440 546124 2236664 0 0 0 1608 1246 1476 76 1 8 15 0 0 1 20768 228440 546124 2236664 0 0 0 1460 1205 1387 75 1 9 15 0 1 0 20768 228316 546124 2236668 0 0 0 1560 1209 1453 75 1 7 17 0 [tems@lsp2052a bin]$ iostat 1 4 Linux 2.6.32-220.el6.x86_64 (mytems) 02/04/2013 _x86_64_ (1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 56.38 0.01 0.74 11.16 0.00 31.71 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 1.00 7.05 11.96 12944788 21970748 sdb 157.53 15.10 1288.35 27741684 2367079160 dm-0 0.29 2.61 1.90 4788242 3491296 dm-1 0.01 0.02 0.04 36720 77336 dm-2 161.18 15.08 1288.19 27698026 2366780928 dm-3 0.02 0.02 0.16 42978 298232 dm-4 0.10 1.29 0.58 2363362 1056672 dm-5 0.15 0.11 1.14 195666 2101272 dm-6 0.81 2.70 6.33 4957554 11621328 dm-7 0.26 0.24 1.97 433922 3622520 avg-cpu: %user %nice %system %iowait %steal %idle 74.26 0.00 0.99 15.84 0.00 8.91 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 0.99 15.84 0.00 16 0 sdb 380.20 0.00 3104.95 0 3136 dm-0 0.00 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0 dm-2 388.12 0.00 3104.95 0 3136 dm-3 0.00 0.00 0.00 0 0 dm-4 0.00 0.00 0.00 0 0 dm-5 0.00 0.00 0.00 0 0 dm-6 0.00 0.00 0.00 0 0 dm-7 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 75.76 0.00 1.01 16.16 0.00 7.07 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 0.00 0.00 0.00 0 0 sdb 363.64 0.00 2933.33 0 2904 dm-0 0.00 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0 dm-2 366.67 0.00 2933.33 0 2904 dm-3 0.00 0.00 0.00 0 0 dm-4 0.00 0.00 0.00 0 0 dm-5 0.00 0.00 0.00 0 0 dm-6 0.00 0.00 0.00 0 0 dm-7 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 71.29 0.00 0.99 18.81 0.00 8.91 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 0.00 0.00 0.00 0 0 sdb 380.20 0.00 3065.35 0 3096 dm-0 0.00 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0 dm-2 383.17 0.00 3065.35 0 3096 dm-3 0.00 0.00 0.00 0 0 dm-4 0.00 0.00 0.00 0 0 dm-5 0.00 0.00 0.00 0 0 dm-6 0.00 0.00 0.00 0 0 dm-7 0.00 0.00 0.00 0 0 /Gus From: [email protected] [ mailto:[email protected]] On Behalf Of Robin.Smith-U3F5sf8KFGKCmk9zJvdQaQC/[email protected] Sent: den 4 februari 2013 11:21 To: [email protected] Subject: RE: [TME10] FTO Standby TEMS high cpu during sync CPU running peaking is a bit vague, can you run the following command and post the out here next time it peaks, might help identify the bottleneck a bit further vmstat 1 10 iostat 1 4 From: [email protected] [ mailto:[email protected]] On Behalf Of [email protected] Sent: 01 February 2013 15:40 To: [email protected] Subject: [TME10] FTO Standby TEMS high cpu during sync Hi. Everytime between (510BDF05.0000-10:kdyctrl.cpp,3932,"rTEMSSynchThread") TEMS is MIRROR, Synchronizing with HUB TEMS tables. And: q(510BE00D.0000-10:kdyctrl.cpp,4045,"rTEMSSynchThread") Sychronization complete. Going to sleep. The cpu peaks at 99% an nothing unusual in the *_ms_* logs. HUB on Linux RHEL6 6.2.3 FP02. As soon as sync is complete it drops to close to nothing within seconds. I need tips on tracing FTP sync, anyone? /Gus Lloyds TSB Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales, number 2065. Telephone: 020 7626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 327000. Telephone: 020 7626 1500. Lloyds TSB Scotland plc. Registered Office: Henry Duncan House, 120 George Street, Edinburgh EH2 4LH. Registered in Scotland, number 95237. Telephone: 0131 225 4555. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales, number 2299428. Telephone: 01452 372372. Lloyds TSB Bank plc, Lloyds TSB Scotland plc, Bank of Scotland plc and Cheltenham & Gloucester plc are authorised and regulated by the Financial Services Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds TSB Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 218813. Telephone: 020 7626 1500. Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 95000. Telephone: 0131 225 4555 This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded. Lloyds TSB Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales, number 2065. Telephone: 020 7626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 327000. Telephone: 020 7626 1500. Lloyds TSB Scotland plc. Registered Office: Henry Duncan House, 120 George Street, Edinburgh EH2 4LH. Registered in Scotland, number 95237. Telephone: 0131 225 4555. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales, number 2299428. Telephone: 01452 372372. Lloyds TSB Bank plc, Lloyds TSB Scotland plc, Bank of Scotland plc and Cheltenham & Gloucester plc are authorised and regulated by the Financial Services Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds TSB Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 218813. Telephone: 020 7626 1500. Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 95000. Telephone: 0131 225 4555 This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded. Lloyds TSB Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales, number 2065. Telephone: 020 7626 1500. Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 327000. Telephone: 020 7626 1500. Lloyds TSB Scotland plc. Registered Office: Henry Duncan House, 120 George Street, Edinburgh EH2 4LH. Registered in Scotland, number 95237. Telephone: 0131 225 4555. Cheltenham & Gloucester plc. Registered Office: Barnett Way, Gloucester GL4 3RL. Registered in England and Wales, number 2299428. Telephone: 01452 372372. Lloyds TSB Bank plc, Lloyds TSB Scotland plc, Bank of Scotland plc and Cheltenham & Gloucester plc are authorised and regulated by the Financial Services Authority. Halifax is a division of Bank of Scotland plc. Cheltenham & Gloucester Savings is a division of Lloyds TSB Bank plc. HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 218813. Telephone: 020 7626 1500. Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland, number 95000. Telephone: 0131 225 4555 This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded. _______________________________________________ TME10 mailing list [email protected] Unsubscribe:[email protected] _______________________________________________ TME10 mailing list [email protected] Unsubscribe:[email protected] _______________________________________________ TME10 mailing list [email protected] Unsubscribe:[email protected]