Re: [performancecopilot/pcp] pmwebd impossibly slow when using grafana with 300 archives (#117)
"Frank Ch. Eigler" <[email protected]>
| Newsgroups | gmane.comp.sysutils.pcp |
|---|---|
| Message-ID | <performancecopilot/pcp/issues/117/[email protected]> |
You're roughly right. The hostselect.js dashboard's query is:
/graphite/render?format=json&target=*.pmcd.pmlogger.port.*&from=-1m&until=now
... which asks pmwebd to iterate through all archives (300*7), to pull out one metric value recorded in the last minute. Its goal is to enumerate those archive files that are currently being written to, so it can reverse-engineer host names etc. from them.
Those archives whose end-of-records timestamp doesn't include this moment will be rejected pretty quickly (after one `pmGetArchiveEnd` call). This involves reading little bits of the beginning and the end of the .0 / .index / .meta files. (Compare `strace pmloglabel -l $ARCHIVEFILE`.) ... though there are some thousands of them, and this is not cached within pmwebd (see that rhbz link above), so there is some wasted time but not that much I/O.
Those archives whose time intervals does include the last minute are probably those 300 that are currently being written to by a running `pmlogger`. In this case, `pmwebd` seeks to the near-end of the archive with `pmSetMode`, and tries to fetch that metric value. Unfortunately, things go badly here, because the metric in question turns out to be stored only one time, at the beginning of the archive file/timeline, and libpcp decides to go searching for it. During this search, libpcp reads, **backwards**, essentially the **whole** archive data file. For the sad tale, see your own strace, or the analogous:
strace pmval -t 60 -S "@`date -d '-1min' +'%y-%m-%d %H:%M'`" -T "@`date +'%y-%m-%d %H:%M'`" -a $ANY_ACTIVE_ARCHIVE_FILE pmcd.pmlogger.port
Sorry about that. That is pretty abysmal.
As a hack (and I bet you'll figure out why it works if it works), try changing the hostselect.js file thusly, and clear those browser caches:
- url: pmwebd + "/graphite/render?format=json&target=*.pmcd.pmlogger.port.*&from=-1m&until=now"
+ url: pmwebd + "/graphite/render?format=json&target=*.proc.nprocs&from=-1m&until=now"
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/performancecopilot/pcp/issues/117#issuecomment-251401622
_______________________________________________
pcp mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/pcp