Re: Metrics not collecting / No PMCD agent for domain of request messages in pmlogger.log file.
[email protected] Tue, 03 Jan 2017 08:07:07 -0800
| Newsgroups | gmane.comp.sysutils.pcp |
|---|---|
| Message-ID | <[email protected]> |
Below is my pmcd.log for a server. The metrics were not getting collected on the server. Log for pmcd on hostname.of.server started Thu Dec 29 13:54:35 2016 active agent dom pid in out ver protocol parameters ============ === ===== === === === ======== ========== root 1 14933 8 9 2 bin pipe cmd=/var/lib/pcp/pmdas/root/pmdaroot pmcd 2 2 dso i:6 lib=/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so entry=pmcd_init [0x7fd2a4777ba0] proc 3 14935 13 14 2 bin pipe cmd=/var/lib/pcp/pmdas/proc/pmdaproc -d 3 xfs 11 14936 15 16 2 bin pipe cmd=/var/lib/pcp/pmdas/xfs/pmdaxfs -d 11 linux 60 14937 17 18 2 bin pipe cmd=/var/lib/pcp/pmdas/linux/pmdalinux mmv 70 2 dso i:4 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=mmv_init [0x7fd2a4570950] jbd2 122 2 dso i:4 lib=/var/lib/pcp/pmdas/jbd2/pmda_jbd2.so entry=jbd2_init [0x7fd2a436d790] Host access list: 00 01 Cur/MaxCons host-spec host-mask lvl host-name == == =========== ======================================= ======================================= === ============== y y 0 0 IPAddress 255.255.255.255 0 localhost y y 0 0 / / 1 unix: n 0 0 0.0.0.0 0.0.0.0 4 .* n 0 0 :: :: 8 :* User access list empty: user-based access control turned off Group access list empty: group-based access control turned off pmcd: PID = 14930, PDU version = 2 pmcd request port(s): sts fd port family address === ==== ===== ====== ======= ok 6 unix /var/run/pcp/pmcd.socket ok 0 44321 inet INADDR_ANY [Thu Dec 29 14:30:58] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:30:58] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:30:59] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:30:59] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:01] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:01] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:02] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:05] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:06] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:06] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:07] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:08] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:08] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Thu Dec 29 14:31:08] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Fri Dec 30 00:52:03] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Fri Dec 30 00:52:03] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:45] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:46] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:46] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:49] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:49] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:50] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:51] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:53] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:53] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Sat Dec 31 21:25:53] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:45] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:45] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:45] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:46] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:48] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:48] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:48] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:48] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:48] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:48] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:48] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:51] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:51] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:51] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:52] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:52] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:52] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:52] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:53] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:54] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 14:25:54] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 15:57:02] pmcd(14930) Info: DoFetch: select timeout [Mon Jan 2 15:57:02] pmcd(14930) Info: CleanupAgent ... Cleanup "proc" agent (dom 3): protocol failure for fd=13 [Mon Jan 2 15:57:03] pmcd(14930) Info: CleanupAgent ... Cleanup "linux" agent (dom 60): protocol failure for fd=17, exit(0) [Mon Jan 2 16:25:45] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:45] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:47] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:49] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:49] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:50] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:50] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:50] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:50] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:51] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:52] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:52] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:52] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:52] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:53] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:53] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:53] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Mon Jan 2 16:25:54] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Tue Jan 3 00:51:30] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure [Tue Jan 3 00:51:30] pmcd(14930) Error: ClientLoop: error sending Conn ACK PDU to new client IPC protocol failure I also noticed the following in the pmcd/linux.log and pmcd/proc.log files * linux.log * Log for pmdalinux on hostname.of.server started Thu Dec 29 13:54:35 2016 Log finished Mon Jan 2 15:57:05 2017 * proc.log * Log for pmdaproc on hostname.of.server started Thu Dec 29 13:54:35 2016 Log finished Mon Jan 2 15:57:11 2017 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After restarting the pmcd and pmlogger services the metrics are getting collected again but I am sure that it will stop again. Below is the pmcd.log file with the metrics getting collected. Log for pmcd on hostname.of.server started Tue Jan 3 10:34:57 2017 active agent dom pid in out ver protocol parameters ============ === ===== === === === ======== ========== root 1 4614 8 9 2 bin pipe cmd=/var/lib/pcp/pmdas/root/pmdaroot pmcd 2 2 dso i:6 lib=/var/lib/pcp/pmdas/pmcd/pmda_pmcd.so entry=pmcd_init [0x7f106f070ba0] proc 3 4615 13 14 2 bin pipe cmd=/var/lib/pcp/pmdas/proc/pmdaproc -d 3 xfs 11 4616 15 16 2 bin pipe cmd=/var/lib/pcp/pmdas/xfs/pmdaxfs -d 11 linux 60 4617 17 18 2 bin pipe cmd=/var/lib/pcp/pmdas/linux/pmdalinux mmv 70 2 dso i:4 lib=/var/lib/pcp/pmdas/mmv/pmda_mmv.so entry=mmv_init [0x7f106ee69950] jbd2 122 2 dso i:4 lib=/var/lib/pcp/pmdas/jbd2/pmda_jbd2.so entry=jbd2_init [0x7f106ec66790] Host access list: 00 01 Cur/MaxCons host-spec host-mask lvl host-name == == =========== ======================================= ======================================= === ============== y y 0 0 IPADDRESS 255.255.255.255 0 localhost y y 0 0 / / 1 unix: n 0 0 0.0.0.0 0.0.0.0 4 .* n 0 0 :: :: 8 :* User access list empty: user-based access control turned off Group access list empty: group-based access control turned off pmcd: PID = 4611, PDU version = 2 pmcd request port(s): sts fd port family address === ==== ===== ====== ======= ok 6 unix /var/run/pcp/pmcd.socket ok 0 44321 inet INADDR_ANY