Re: issues with pmlogger_daily via cron
"Sharif Islam" <[email protected]> Wed, 15 Feb 2017 14:46:00 +0100 (CET)
| Newsgroups | gmane.comp.sysutils.pcp |
|---|---|
| Message-ID | <[email protected]> |
I am still debugging this issue. I noticed there were two pmlogger processes that were launched via cron. The 13:45 was launched via pmlogger_check cron script (runs every 30 minutes) and the 14:08 was the pmlogger_daily (usually runs after midnight). pmlogger -h local: -c /var/lib/pcp/config/pmlogger/logger.config -r -l /var/log/pcp-storage/pmlogger.log -U root -m pmlogger_check 20170215.13.45 pmlogger -r -U root -m pmlogger_daily -l /var/log/pcp-storage/pmlogger.log -h local: -c /var/lib/pcp/config/pmlogger/logger.config 20170215.14.08 Here are the two -V -V output: [control:1] host="#" primary="" socks="" dir="" args="" [control:2] host="#" primary="archive" socks="logging" dir="configuration/control" args="" [control:3] host="#" primary="Copyright" socks="(C)" dir="2005-2013" args="Silicon Graphics, Inc. All Rights Reserved." [control:4] host="#" primary="$Id:" socks="logger.control.in,v" dir="1.5" args="2015/04/30 16:08:57 sudhas Exp $" [control:5] host="#" primary="" socks="" dir="" args="" [control:6] host="" primary="" socks="" dir="" args="" [control:7] host="#" primary="===" socks="VARIABLE" dir="ASSIGNMENTS" args="===" [control:8] host="#" primary="" socks="" dir="" args="" [control:9] host="#" primary="DO" socks="NOT" dir="REMOVE" args="OR EDIT THE FOLLOWING LINE" [control:10] host="$version=1.1" primary="" socks="" dir="" args="" [control:11] host="" primary="" socks="" dir="" args="" [control:12] host="#" primary="===" socks="LOGGER" dir="CONTROL" args="SPECIFICATIONS ===" [control:13] host="#" primary="" socks="" dir="" args="" [control:14] host="#Host" primary="P?" socks="S?" dir="directory" args="args" [control:15] host="" primary="" socks="" dir="" args="" [control:16] host="#" primary="local" socks="pcp-storage" dir="logger" args="" [control:17] host="local:" primary="n" socks="n" dir="/var/lib/pcp-storage/archives/localhost" args="-c /var/lib/pcp/config/pmlogger/logger.config -r -l /var/log/pcp-storage/pmlogger.log -U root" Check pmlogger -h local: ... in /var/lib/pcp-storage/archives/localhost ... === daily maintenance of PCP archives for host local: === ... try /var/lib/pcp/tmp/pmlogger/31992: match=2 pmlogger process 31992 identified, OK Looking for pmlogger (process 31992) ... found root 31992 1 0 14:08 ? 00:00:01 pmlogger -r -U root -m pmlogger_daily -l /var/log/pcp-storage/pmlogger.log -h loca Terminating pmlogger (process 31992) ... SIGUSR1 ... SIGTERM ... done Launching new pmlogger in directory "/var/lib/pcp-storage/archives/localhost" as ... pmlogger -r -U root -m pmlogger_daily -l /var/log/pcp-storage/pmlogger.log -h loca -c /var/lib/pcp/config/pmlogger/logger.config 20170215.14.26 Waiting for new pmlogger to start ... process exited! Contents of pmlogger output file "/var/log/pcp-storage/pmlogger.log" ... Log for pmlogger on dmf started Wed Feb 15 14:26:12 2017 pmlogger: Cannot connect to PMCD on host "loca": No route to host Second one: [control:1] host="#" primary="" socks="" dir="" args="" [control:2] host="#" primary="archive" socks="logging" dir="configuration/control" args="" [control:3] host="#" primary="Copyright" socks="(C)" dir="2005-2013" args="Silicon Graphics, Inc. All Rights Reserved." [control:4] host="#" primary="$Id:" socks="logger.control.in,v" dir="1.5" args="2015/04/30 16:08:57 sudhas Exp $" [control:5] host="#" primary="" socks="" dir="" args="" [control:6] host="" primary="" socks="" dir="" args="" [control:7] host="#" primary="===" socks="VARIABLE" dir="ASSIGNMENTS" args="===" [control:8] host="#" primary="" socks="" dir="" args="" [control:9] host="#" primary="DO" socks="NOT" dir="REMOVE" args="OR EDIT THE FOLLOWING LINE" [control:10] host="$version=1.1" primary="" socks="" dir="" args="" [control:11] host="" primary="" socks="" dir="" args="" [control:12] host="#" primary="===" socks="LOGGER" dir="CONTROL" args="SPECIFICATIONS ===" [control:13] host="#" primary="" socks="" dir="" args="" [control:14] host="#Host" primary="P?" socks="S?" dir="directory" args="args" [control:15] host="" primary="" socks="" dir="" args="" [control:16] host="#" primary="local" socks="pcp-storage" dir="logger" args="" [control:17] host="local:" primary="n" socks="n" dir="/var/lib/pcp-storage/archives/localhost" args="-c /var/lib/pcp/config/pmlogger/logger.config -r -l /var/log/pcp-storage/pmlogger.log -U root" Check pmlogger -h local: ... in /var/lib/pcp-storage/archives/localhost ... === daily maintenance of PCP archives for host local: === ... try /var/lib/pcp/tmp/pmlogger/238439: match=2 pmlogger process 238439 identified, OK Looking for pmlogger (process 238439) ... found root 238439 1 0 13:45 ? 00:00:01 pmlogger -h local: -c /var/lib/pcp/config/pmlogger/logger.config -r -l /var/log/pc Terminating pmlogger (process 238439) ... SIGUSR1 ... SIGTERM ... done Launching new pmlogger in directory "/var/lib/pcp-storage/archives/localhost" as ... pmlogger -r -U root -m pmlogger_daily -l /var/log/pcp-storage/pmlogger.log -h local: -c /var/lib/pcp/config/pmlogger/logger.config 20170215.14.08 Waiting for new pmlogger to start ... done New pmlogger status ... pmlogger [31992] on host local: is logging metrics from host dmf Here's what is currently running: root# $PCP_PS_PROG $PCP_PS_ALL_FLAGS | grep '[p]mlogger' pcp 71712 1 0 11:23 pts/13 00:00:07 pmlogger -P -r -T24h10m -c config.default -m pmlogger_check 20170215.11.23 pcp 79765 66857 0 11:25 ? 00:00:07 /usr/bin/pmlogger -c /var/log/pcp/pmmgr/dmf/config.pmlogger -h local: -r -l /var/log/pcp/pmmgr/dmf/pmlogger.log -y -T @Thu Feb 16 01:00:00 2017 /var/log/pcp/pmmgr/dmf/archive-20170215.102529 I would appreciate if someone can look at this. --sharif ----- Original Message ----- From: "Sharif Islam" <[email protected]> To: "pcp" <[email protected]> Sent: Tuesday, February 14, 2017 11:28:19 AM Subject: Re: [pcp] issues with pmlogger_daily via cron G'day Sharif, I'm not altogether sure of the problem here ... let me guess ... Thanks. > ... > + args='-r -U root -m pmlogger_daily -l /var/log/pcp-storage/pmlogger.log -h > loca -c /var/lib/pcp/config/pmlogger/logger.config ' the -c here is wrong? This is my cron line: /usr/lib64/pcp/bin/pmlogger_daily -V -c /var/lib/pcp/config/pmlogger/logger.control To debug this we'll need ... 1. output from $ grep PS /etc/pcp.conf # grep PS /etc/pcp.conf PCP_PS_PROG=ps PCP_PS_HAVE_BSD=false PCP_PS_ALL_FLAGS=-efw 2. your pmlogger control file root# cat /var/lib/pcp/config/pmlogger/logger.control # # archive logging configuration/control # Copyright (C) 2005-2013 Silicon Graphics, Inc. All Rights Reserved. # $Id: logger.control.in,v 1.5 2015/04/30 16:08:57 sudhas Exp $ # # === VARIABLE ASSIGNMENTS === # # DO NOT REMOVE OR EDIT THE FOLLOWING LINE $version=1.1 # === LOGGER CONTROL SPECIFICATIONS === # #Host P? S? directory args # local pcp-storage logger LOCALHOSTNAME n n /var/lib/pcp-storage/archives/localhost -c /var/lib/pcp/config/pmlogger/logger.config -r -l /var/log/pcp-storage/pmlogger.log -U root 3. can you rerun this with pmlogger_daily being run from cron with -V -V (very verbose)? The cron did not return anything for a while. So I restarted pcp and got this in the mail. Before the restart there was a defunct process. Here's the -V -V output: https://gist.github.com/anonymous/f4ae6f70e3d62ed9b28f436fae9b5a39#file-pmlogger 4. output from $ . /etc/pcp.conf $ $PCP_PS_PROG $PCP_PS_ALL_FLAGS | grep '[p]mlogger' # $PCP_PS_PROG $PCP_PS_ALL_FLAGS | grep '[p]mlogger' pcp 10361 66857 0 01:01 ? 00:00:36 /usr/bin/pmlogger -c /var/log/pcp/pmmgr/dmf/config.pmlogger -h local: -r -l /var/log/pcp/pmmgr/dmf/pmlogger.log -y -T @Wed Feb 15 01:00:00 2017 /var/log/pcp/pmmgr/dmf/archive-20170214.000157 pcp 140823 140801 0 00:10 ? 00:00:00 [pmlogger_daily] <defunct> pcp 142598 1 0 00:10 ? 00:00:39 pmlogger -r -T 24h10m -m pmlogger_daily -P -l pmlogger.log -c /etc/pcp/pmlogger/config.default 20170214.00.10 root 152385 1 0 00:15 ? 00:01:09 pmlogger -h local: -c /var/lib/pcp/config/pmlogger/logger.config -r -l /var/log/pcp-storage/pmlogger.log -U root -m pmlogger_check 20170214.00.15 I did notice a defunct process. We have this running at 00:10 /usr/lib/pcp-storage/logger_reduce -c /var/lib/pcp/config/pmlogger/logger.control I restarted pcp. Now: # $PCP_PS_PROG $PCP_PS_ALL_FLAGS | grep '[p]mlogger' pcp 25399 1 0 11:24 pts/11 00:00:00 pmlogger -P -r -T24h10m -c config.default -m pmlogger_check 20170214.11.24 pcp 38526 66857 0 11:26 ? 00:00:00 /usr/bin/pmlogger -c /var/log/pcp/pmmgr/dmf/config.pmlogger -h local: -r -l /var/log/pcp/pmmgr/dmf/pmlogger.log -y -T @Wed Feb 15 01:00:00 2017 /var/log/pcp/pmmgr/dmf/archive-20170214.102653 --sharif -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#15177): https://groups.io/g/pcp/message/15177 View All Messages In Topic (4): https://groups.io/g/pcp/topic/4365667 Mute This Topic: https://groups.io/mt/4365667?uid=174580 New Topic: https://groups.io/g/pcp/post -=-=- pcp mailing list [email protected] https://groups.io/g/pcp/messages -=-=- Change Your Subscription: https://groups.io/g/pcp/editsub?uid=174580 Group Home: https://groups.io/g/pcp Contact Group Owner: [email protected] Terms of Service: https://groups.io/static/tos Unsubscribe: https://groups.io/g/pcp/leave/354243/563757577/xyzzy -=-=-=-=-=-=-=-=-=-=-=-