[performancecopilot/pcp] pmlogger memory leak and crash after Ctrl-C (#116)
Marko Kevac <[email protected]>
| Newsgroups | gmane.comp.sysutils.pcp |
|---|---|
| Message-ID | <performancecopilot/pcp/issues/[email protected]> |
Hi.
```
marko@fedora:~/tmp $ cat /etc/fedora-release
Fedora release 24 (Twenty Four)
marko@fedora:~/tmp $ sudo dnf info pcp
Last metadata expiration check: 1:54:49 ago on Mon Sep 26 16:02:17 2016.
Installed Packages
Name : pcp
Arch : x86_64
Epoch : 0
Version : 3.11.4
Release : 1.fc24
Size : 3.3 M
Repo : @System
From repo : updates
Summary : System-level performance monitoring and performance management
URL : http://www.pcp.io
License : GPLv2+ and LGPLv2.1+ and CC-BY
Description : Performance Co-Pilot (PCP) provides a framework and services to support
: system-level performance monitoring and performance management.
:
: The PCP open source release provides a unifying abstraction for all of
: the interesting performance data in a system, and allows client
: applications to easily retrieve and process any subset of that data.
```
With this config
```
$ cat /tmp/cppbig23.mlan-leak.conf
log mandatory on 10 msec {
cgroup
containers
disk
event
filesys
hinv
hotproc
ipc
jbd2
kernel
mem
mmv
network
nfs
nfs3
nfs4
pmcd
pmda
proc
quota
rpc
swap
swapdev
sysfs
tmpfs
vfs
xfs
}
```
pmlogger executed as
`$ /usr/libexec/pcp/bin/pmlogger -h localhost -r -c /tmp/cppbig23.mlan-leak.conf -m marko_check -l /tmp/pmlogger.log 123`
leaks memory.
You can see this with pidstat output
```
$ pidstat -r -p $(pgrep pmlogger) 1
Linux 4.7.3-200.fc24.x86_64 (fedora) 09/26/2016 _x86_64_ (8 CPU)
05:52:22 PM UID PID minflt/s majflt/s VSZ RSS %MEM Command
05:52:23 PM 1000 21394 270.00 0.00 76172 9656 0.03 pmlogger
05:52:24 PM 1000 21394 278.00 0.00 77284 10772 0.03 pmlogger
05:52:25 PM 1000 21394 282.00 0.00 78444 11780 0.04 pmlogger
05:52:26 PM 1000 21394 270.00 0.00 79492 12840 0.04 pmlogger
05:52:27 PM 1000 21394 280.00 0.00 80648 13992 0.04 pmlogger
05:52:28 PM 1000 21394 263.00 0.00 81664 15152 0.05 pmlogger
05:52:29 PM 1000 21394 265.00 0.00 82772 16276 0.05 pmlogger
05:52:30 PM 1000 21394 280.00 0.00 83920 17280 0.05 pmlogger
05:52:31 PM 1000 21394 276.00 0.00 85072 18440 0.06 pmlogger
05:52:32 PM 1000 21394 278.00 0.00 86076 19568 0.06 pmlogger
05:52:33 PM 1000 21394 276.00 0.00 87288 20536 0.06 pmlogger
05:52:34 PM 1000 21394 272.00 0.00 88284 21816 0.07 pmlogger
^C
Average: 1000 21394 274.17 0.00 82260 15676 0.05 pmlogger
```
If you keep it that way, it will allocate all available memory on server.
Additionaly, if you stop such process with Ctrl-C, it will crash with buffer overflow:
```
$ /usr/libexec/pcp/bin/pmlogger -h localhost -r -c /tmp/cppbig23.mlan-leak.conf -m marko_check -l /tmp/pmlogger.log 123
^C*** buffer overflow detected ***: /usr/libexec/pcp/bin/pmlogger terminated
======= Backtrace: =========
/lib64/libc.so.6(+0x7566b)[0x7f32c3a5766b]
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f32c3af6c57]
/lib64/libc.so.6(+0x112d80)[0x7f32c3af4d80]
/lib64/libc.so.6(+0x114bca)[0x7f32c3af6bca]
/lib64/libpcp.so.3(+0x5efed)[0x7f32c401efed]
/lib64/libpcp.so.3(+0x1dee2)[0x7f32c3fddee2]
/lib64/libpcp.so.3(__pmGetPDU+0x5f)[0x7f32c3fde8ff]
/usr/libexec/pcp/bin/pmlogger(+0x7f3b)[0x55b298668f3b]
/usr/libexec/pcp/bin/pmlogger(+0x8d1e)[0x55b298669d1e]
/usr/libexec/pcp/bin/pmlogger(+0x54e0)[0x55b2986664e0]
/lib64/libc.so.6(__libc_start_main+0xf1)[0x7f32c3a02731]
/usr/libexec/pcp/bin/pmlogger(+0x65e9)[0x55b2986675e9]
======= Memory map: ========
55b298661000-55b29867b000 r-xp 00000000 fd:00 2366568 /usr/bin/pmlogger
55b29887b000-55b29887c000 r--p 0001a000 fd:00 2366568 /usr/bin/pmlogger
55b29887c000-55b29887d000 rw-p 0001b000 fd:00 2366568 /usr/bin/pmlogger
55b29887d000-55b29887e000 rw-p 00000000 00:00 0
55b29a36c000-55b29b816000 rw-p 00000000 00:00 0 [heap]
7f32bfdf1000-7f32bfdfc000 r-xp 00000000 fd:00 2370037 /usr/lib64/libnss_files-2.23.so
7f32bfdfc000-7f32bfffb000 ---p 0000b000 fd:00 2370037 /usr/lib64/libnss_files-2.23.so
7f32bfffb000-7f32bfffc000 r--p 0000a000 fd:00 2370037 /usr/lib64/libnss_files-2.23.so
7f32bfffc000-7f32bfffd000 rw-p 0000b000 fd:00 2370037 /usr/lib64/libnss_files-2.23.so
7f32bfffd000-7f32c0003000 rw-p 00000000 00:00 0
7f32c0003000-7f32c0007000 r-xp 00000000 fd:00 2366396 /usr/lib64/libattr.so.1.1.0
7f32c0007000-7f32c0206000 ---p 00004000 fd:00 2366396 /usr/lib64/libattr.so.1.1.0
7f32c0206000-7f32c0207000 r--p 00003000 fd:00 2366396 /usr/lib64/libattr.so.1.1.0
7f32c0207000-7f32c0208000 rw-p 00000000 00:00 0
7f32c0208000-7f32c027a000 r-xp 00000000 fd:00 2370087 /usr/lib64/libpcre.so.1.2.7
7f32c027a000-7f32c0479000 ---p 00072000 fd:00 2370087 /usr/lib64/libpcre.so.1.2.7
7f32c0479000-7f32c047a000 r--p 00071000 fd:00 2370087 /usr/lib64/libpcre.so.1.2.7
7f32c047a000-7f32c047b000 rw-p 00072000 fd:00 2370087 /usr/lib64/libpcre.so.1.2.7
7f32c047b000-7f32c0491000 r-xp 00000000 fd:00 2360969 /usr/lib64/libgcc_s-6.2.1-20160916.so.1
7f32c0491000-7f32c0690000 ---p 00016000 fd:00 2360969 /usr/lib64/libgcc_s-6.2.1-20160916.so.1
7f32c0690000-7f32c0691000 r--p 00015000 fd:00 2360969 /usr/lib64/libgcc_s-6.2.1-20160916.so.1
7f32c0691000-7f32c0692000 rw-p 00016000 fd:00 2360969 /usr/lib64/libgcc_s-6.2.1-20160916.so.1
7f32c0692000-7f32c06a4000 r-xp 00000000 fd:00 2359774 /usr/lib64/libgpg-error.so.0.19.1
7f32c06a4000-7f32c08a4000 ---p 00012000 fd:00 2359774 /usr/lib64/libgpg-error.so.0.19.1
7f32c08a4000-7f32c08a5000 r--p 00012000 fd:00 2359774 /usr/lib64/libgpg-error.so.0.19.1
7f32c08a5000-7f32c08a6000 rw-p 00013000 fd:00 2359774 /usr/lib64/libgpg-error.so.0.19.1
7f32c08a6000-7f32c0982000 r-xp 00000000 fd:00 2370131 /usr/lib64/libgcrypt.so.20.0.6
7f32c0982000-7f32c0b82000 ---p 000dc000 fd:00 2370131 /usr/lib64/libgcrypt.so.20.0.6
7f32c0b82000-7f32c0b83000 r--p 000dc000 fd:00 2370131 /usr/lib64/libgcrypt.so.20.0.6
7f32c0b83000-7f32c0b8b000 rw-p 000dd000 fd:00 2370131 /usr/lib64/libgcrypt.so.20.0.6
7f32c0b8b000-7f32c0b8c000 rw-p 00000000 00:00 0
7f32c0b8c000-7f32c0b9d000 r-xp 00000000 fd:00 2366821 /usr/lib64/liblz4.so.1.7.1
7f32c0b9d000-7f32c0d9c000 ---p 00011000 fd:00 2366821 /usr/lib64/liblz4.so.1.7.1
7f32c0d9c000-7f32c0d9d000 r--p 00010000 fd:00 2366821 /usr/lib64/liblz4.so.1.7.1
7f32c0d9d000-7f32c0d9e000 rw-p 00000000 00:00 0
7f32c0d9e000-7f32c0dc3000 r-xp 00000000 fd:00 2366822 /usr/lib64/liblzma.so.5.2.2
7f32c0dc3000-7f32c0fc2000 ---p 00025000 fd:00 2366822 /usr/lib64/liblzma.so.5.2.2
7f32c0fc2000-7f32c0fc3000 r--p 00024000 fd:00 2366822 /usr/lib64/liblzma.so.5.2.2
7f32c0fc3000-7f32c0fc4000 rw-p 00000000 00:00 0
7f32c0fc4000-7f32c0fc8000 r-xp 00000000 fd:00 2366447 /usr/lib64/libcap.so.2.24
7f32c0fc8000-7f32c11c7000 ---p 00004000 fd:00 2366447 /usr/lib64/libcap.so.2.24
7f32c11c7000-7f32c11c8000 r--p 00003000 fd:00 2366447 /usr/lib64/libcap.so.2.24
7f32c11c8000-7f32c11c9000 rw-p 00004000 fd:00 2366447 /usr/lib64/libcap.so.2.24
7f32c11c9000-7f32c11ec000 r-xp 00000000 fd:00 2370089 /usr/lib64/libselinux.so.1
7f32c11ec000-7f32c13ec000 ---p 00023000 fd:00 2370089 /usr/lib64/libselinux.so.1
7f32c13ec000-7f32c13ed000 r--p 00023000 fd:00 2370089 /usr/lib64/libselinux.so.1
7f32c13ed000-7f32c13ee000 rw-p 00024000 fd:00 2370089 /usr/lib64/libselinux.so.1
7f32c13ee000-7f32c13f0000 rw-p 00000000 00:00 0
7f32c13f0000-7f32c13f2000 r-xp 00000000 fd:00 2360747 /usr/lib64/libfreebl3.so
7f32c13f2000-7f32c15f1000 ---p 00002000 fd:00 2360747 /usr/lib64/libfreebl3.so
7f32c15f1000-7f32c15f2000 r--p 00001000 fd:00 2360747 /usr/lib64/libfreebl3.so
7f32c15f2000-7f32c15f3000 rw-p 00002000 fd:00 2360747 /usr/lib64/libfreebl3.so
7f32c15f3000-7f32c1641000 r-xp 00000000 fd:00 2373020 /usr/lib64/libdbus-1.so.3.15.2
7f32c1641000-7f32c1840000 ---p 0004e000 fd:00 2373020 /usr/lib64/libdbus-1.so.3.15.2
7f32c1840000-7f32c1842000 r--p 0004d000 fd:00 2373020 /usr/lib64/libdbus-1.so.3.15.2
7f32c1842000-7f32c1843000 rw-p 0004f000 fd:00 2373020 /usr/lib64/libdbus-1.so.3.15.2
7f32c1843000-7f32c184a000 r-xp 00000000 fd:00 2370024 /usr/lib64/libcrypt-2.23.so
7f32c184a000-7f32c1a49000 ---p 00007000 fd:00 2370024 /usr/lib64/libcrypt-2.23.so
7f32c1a49000-7f32c1a4a000 r--p 00006000 fd:00 2370024 /usr/lib64/libcrypt-2.23.so
7f32c1a4a000-7f32c1a4b000 rw-p 00007000 fd:00 2370024 /usr/lib64/libcrypt-2.23.so
7f32c1a4b000-7f32c1a79000 rw-p 00000000 00:00 0
7f32c1a79000-7f32c1a90000 r-xp 00000000 fd:00 2370046 /usr/lib64/libresolv-2.23.so
7f32c1a90000-7f32c1c8f000 ---p 00017000 fd:00 2370046 /usr/lib64/libresolv-2.23.so
7f32c1c8f000-7f32c1c90000 r--p 00016000 fd:00 2370046 /usr/lib64/libresolv-2.23.so
7f32c1c90000-7f32c1c91000 rw-p 00017000 fd:00 2370046 /usr/lib64/libresolv-2.23.so
7f32c1c91000-7f32c1c93000 rw-p 00000000 00:00 0
7f32c1c93000-7f32c1c9a000 r-xp 00000000 fd:00 2370048 /usr/lib64/librt-2.23.so
7f32c1c9a000-7f32c1e99000 ---p 00007000 fd:00 2370048 /usr/lib64/librt-2.23.so
7f32c1e99000-7f32c1e9a000 r--p 00006000 fd:00 2370048 /usr/lib64/librt-2.23.so
7f32c1e9a000-7f32c1e9b000 rw-p 00007000 fd:00 2370048 /usr/lib64/librt-2.23.so
7f32c1e9b000-7f32c1eb0000 r-xp 00000000 fd:00 2367257 /usr/lib64/libz.so.1.2.8
7f32c1eb0000-7f32c20af000 ---p 00015000 fd:00 2367257 /usr/lib64/libz.so.1.2.8
7f32c20af000-7f32c20b0000 r--p 00014000 fd:00 2367257 /usr/lib64/libz.so.1.2.8
7f32c20b0000-7f32c20b1000 rw-p 00015000 fd:00 2367257 /usr/lib64/libz.so.1.2.8
7f32c20b1000-7f32c20b4000 r-xp 00000000 fd:00 2370099 /usr/lib64/libplds4.so
7f32c20b4000-7f32c22b3000 ---p 00003000 fd:00 2370099 /usr/lib64/libplds4.so
7f32c22b3000-7f32c22b4000 r--p 00002000 fd:00 2370099 /usr/lib64/libplds4.so
7f32c22b4000-7f32c22b5000 rw-p 00003000 fd:00 2370099 /usr/lib64/libplds4.so
7f32c22b5000-7f32c22b9000 r-xp 00000000 fd:00 2370098 /usr/lib64/libplc4.so
7f32c22b9000-7f32c24b8000 ---p 00004000 fd:00 2370098 /usr/lib64/libplc4.so
7f32c24b8000-7f32c24b9000 r--p 00003000 fd:00 2370098 /usr/lib64/libplc4.so
7f32c24b9000-7f32c24ba000 rw-p 00004000 fd:00 2370098 /usr/lib64/libplc4.so
7f32c24ba000-7f32c24e1000 r-xp 00000000 fd:00 2365346 /usr/lib64/libnssutil3.so
7f32c24e1000-7f32c26e0000 ---p 00027000 fd:00 2365346 /usr/lib64/libnssutil3.so
7f32c26e0000-7f32c26e7000 r--p 00026000 fd:00 2365346 /usr/lib64/libnssutil3.so
7f32c26e7000-7f32c26e8000 rw-p 0002d000 fd:00 2365346 /usr/lib64/libnssutil3.so
7f32c26e8000-7f32c26eb000 r-xp 00000000 fd:00 2370025 /usr/lib64/libdl-2.23.so
7f32c26eb000-7f32c28ea000 ---p 00003000 fd:00 2370025 /usr/lib64/libdl-2.23.so
7f32c28ea000-7f32c28eb000 r--p 00002000 fd:00 2370025 /usr/lib64/libdl-2.23.so
7f32c28eb000-7f32c28ec000 rw-p 00003000 fd:00 2370025 /usr/lib64/libdl-2.23.so
7f32c28ec000-7f32c29f4000 r-xp 00000000 fd:00 2370027 /usr/lib64/libm-2.23.so
7f32c29f4000-7f32c2bf3000 ---p 00108000 fd:00 2370027 /usr/lib64/libm-2.23.so
7f32c2bf3000-7f32c2bf4000 r--p 00107000 fd:00 2370027 /usr/lib64/libm-2.23.so
7f32c2bf4000-7f32c2bf5000 rw-p 00108000 fd:00 2370027 /usr/lib64/libm-2.23.so
7f32c2bf5000-7f32c2c05000 r-xp 00000000 fd:00 2373023 /usr/lib64/libavahi-client.so.3.2.9
7f32c2c05000-7f32c2e04000 ---p 00010000 fd:00 2373023 /usr/lib64/libavahi-client.so.3.2.9
7f32c2e04000-7f32c2e05000 r--p 0000f000 fd:00 2373023 /usr/lib64/libavahi-client.so.3.2.9
7f32c2e05000-7f32c2e06000 rw-p 00010000 fd:00 2373023 /usr/lib64/libavahi-client.so.3.2.9
7f32c2e06000-7f32c2e11000 r-xp 00000000 fd:00 2366078 /usr/lib64/libavahi-common.so.3.5.3
7f32c2e11000-7f32c3011000 ---p 0000b000 fd:00 2366078 /usr/lib64/libavahi-common.so.3.5.3
7f32c3011000-7f32c3012000 r--p 0000b000 fd:00 2366078 /usr/lib64/libavahi-common.so.3.5.3
7f32c3012000-7f32c3013000 rw-p 0000c000 fd:00 2366078 /usr/lib64/libavahi-common.so.3.5.3
7f32c3013000-7f32c302f000 r-xp 00000000 fd:00 2367032 /usr/lib64/libsasl2.so.3.0.0
7f32c302f000-7f32c322e000 ---p 0001c000 fd:00 2367032 /usr/lib64/libsasl2.so.3.0.0
7f32c322e000-7f32c322f000 r--p 0001b000 fd:00 2367032 /usr/lib64/libsasl2.so.3.0.0
7f32c322f000-7f32c3230000 rw-p 0001c000 fd:00 2367032 /usr/lib64/libsasl2.so.3.0.0
7f32c3230000-7f32c326b000 r-xp 00000000 fd:00 2370097 /usr/lib64/libnspr4.so
7f32c326b000-7f32c346a000 ---p 0003b000 fd:00 2370097 /usr/lib64/libnspr4.so
7f32c346a000-7f32c346b000 r--p 0003a000 fd:00 2370097 /usr/lib64/libnspr4.so
7f32c346b000-7f32c346d000 rw-p 0003b000 fd:00 2370097 /usr/lib64/libnspr4.so
7f32c346d000-7f32c346f000 rw-p 00000000 00:00 0
7f32c346f000-7f32c3590000 r-xp 00000000 fd:00 2392926 /usr/lib64/libnss3.so
7f32c3590000-7f32c378f000 ---p 00121000 fd:00 2392926 /usr/lib64/libnss3.so
7f32c378f000-7f32c3794000 r--p 00120000 fd:00 2392926 /usr/lib64/libnss3.so
7f32c3794000-7f32c3796000 rw-p 00125000 fd:00 2392926 /usr/lib64/libnss3.so
7f32c3796000-7f32c3798000 rw-p 00000000 00:00 0
7f32c3798000-7f32c37dd000 r-xp 00000000 fd:00 2392928 /usr/lib64/libssl3.so
7f32c37dd000-7f32c39dc000 ---p 00045000 fd:00 2392928 /usr/lib64/libssl3.so
7f32c39dc000-7f32c39e0000 r--p 00044000 fd:00 2392928 /usr/lib64/libssl3.so
7f32c39e0000-7f32c39e1000 rw-p 00048000 fd:00 2392928 /usr/lib64/libssl3.so
7f32c39e1000-7f32c39e2000 rw-p 00000000 00:00 0
7f32c39e2000-7f32c3b9b000 r-xp 00000000 fd:00 2370021 /usr/lib64/libc-2.23.so
7f32c3b9b000-7f32c3d9a000 ---p 001b9000 fd:00 2370021 /usr/lib64/libc-2.23.so
7f32c3d9a000-7f32c3d9e000 r--p 001b8000 fd:00 2370021 /usr/lib64/libc-2.23.so
7f32c3d9e000-7f32c3da0000 rw-p 001bc000 fd:00 2370021 /usr/lib64/libc-2.23.so
7f32c3da0000-7f32c3da4000 rw-p 00000000 00:00 0
7f32c3da4000-7f32c3dbb000 r-xp 00000000 fd:00 2370045 /usr/lib64/libpthread-2.23.so
7f32c3dbb000-7f32c3fba000 ---p 00017000 fd:00 2370045 /usr/lib64/libpthread-2.23.so
7f32c3fba000-7f32c3fbb000 r--p 00016000 fd:00 2370045 /usr/lib64/libpthread-2.23.so
7f32c3fbb000-7f32c3fbc000 rw-p 00017000 fd:00 2370045 /usr/lib64/libpthread-2.23.so
7f32c3fbc000-7f32c3fc0000 rw-p 00000000 00:00 0
7f32c3fc0000-7f32c403d000 r-xp 00000000 fd:00 2361064 /usr/lib64/libpcp.so.3
7f32c403d000-7f32c423c000 ---p 0007d000 fd:00 2361064 /usr/lib64/libpcp.so.3
7f32c423c000-7f32c4240000 r--p 0007c000 fd:00 2361064 /usr/lib64/libpcp.so.3
7f32c4240000-7f32c4241000 rw-p 00080000 fd:00 2361064 /usr/lib64/libpcp.so.3
7f32c4241000-7f32c4246000 rw-p 00000000 00:00 0
7f32c4246000-7f32c426a000 r-xp 00000000 fd:00 2392880 /usr/lib64/ld-2.23.so
7f32c43ab000-7f32c43b4000 rw-p 00000000 00:00 0
7f32c43b4000-7f32c4438000 r-xp 00000000 fd:00 2367449 /usr/lib64/libsystemd.so.0.14.0
7f32c4438000-7f32c4439000 ---p 00084000 fd:00 2367449 /usr/lib64/libsystemd.so.0.14.0
7f32c4439000-7f32c443c000 r--p 00084000 fd:00 2367449 /usr/lib64/libsystemd.so.0.14.0
7f32c443c000-7f32c443d000 rw-p 00087000 fd:00 2367449 /usr/lib64/libsystemd.so.0.14.0
7f32c443d000-7f32c4445000 rw-p 00000000 00:00 0
7f32c4467000-7f32c4469000 rw-p 00000000 00:00 0
7f32c4469000-7f32c446a000 r--p 00023000 fd:00 2392880 /usr/lib64/ld-2.23.so
7f32c446a000-7f32c446b000 rw-p 00024000 fd:00 2392880 /usr/lib64/ld-2.23.so
7f32c446b000-7f32c446c000 rw-p 00000000 00:00 0
7ffc38c28000-7ffc38c4a000 rw-p 00000000 00:00 0 [stack]
7ffc38dcc000-7ffc38dce000 r--p 00000000 00:00 0 [vvar]
7ffc38dce000-7ffc38dd0000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/performancecopilot/pcp/issues/116
_______________________________________________
pcp mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/pcp