Re: RFC: metric labels as first-class PCP concept
"Ken McDonell" <[email protected]>
| Newsgroups | gmane.comp.sysutils.pcp |
|---|---|
| Message-ID | <[email protected]> |
Tardiness is not lack of interest, I've been thinking ... at it takes a
little longer these days as my PCP engagement is pro bono.
I am not opposed to this proposal, although the motivational use cases
are outside my direct experience, except for the "extensible pmUnits"
one, but the proposal does not really address that because there is no
way to load code to handle the additional units, like "temperature".
But I do for see some tricky semantic issues that need to be resolved.
On 21/11/16 17:30, Nathan Scott wrote:
> ... Some other projects implemented
> this notion from the beginning, some examples (and more background):
>
> https://prometheus.io/docs/concepts/data_model/
> http://metrics20.org/
Couple of interesting side-notes:
- What & Why section of the Metrics 2.0 web page reads like the words I
was using to evangelize PCP in 1997 ... yep, 19 years ago!
- Prometheus Metric and Label Naming web page shows striking
similarities with PCP discussions about metric names vs instances =>
probably only subtle semantics separate metrics and instances (labels)
in some cases
> ...
> So, the proposed approach is:
> a/ keep the existing PCP metric and instance concepts unchanged - including
> naming and pmDesc metadata;
> b/ extend those ideas with an (optional) "label" concept, allowing name=value
> pairs to be associated with metric values. In a sense, labels would be
> similar to filesystem extended attributes;
> c/ see labels as primarily static and optimise for that case, while allowing
> for some dynamic change (much like we do with PCP instance names).
I think it is going to be really important to get straight in the
beginning the _scope_ for labels. I can see the following possible
taxonomy (it is not really a hierarchy as the metric and instance
"levels" are orthogonal):
1. context (the global stuff)
2A. metric (common to all instances of a given metric)
2B. one or more instances within an instance domain (common to those
instances across all metrics that share the instance domain)
3. metric-instance (unique to a specific metric-instance pair)
Before delving too deep into the PMAPI requirements, this needs to be
agreed to (are all 4 going to be supported?) and then what are the
semantics for a label that is redefined (with a different, or possibly
the same) value in another place? For example, cute="foo" with the
metric X (over the InDom I), and cute="bar" with instance i of InDom I,
and cute="no clue" with instance i of metric X.
And a somewhat related issue is points in time where the labels change
(a label is added or deleted, or a value for a label changes) ... what
happens for:
- pmlogextract and pmlogreduce?
- interpolate mode in libpcp?
- multi-archive support in libpcp?
- the "reconnect if I can" support in pmlogger?
- and probably some other curly cases I haven't thought of yet
These problems exist with static labels (due to evolutionary change),
but are magnified if labels are dynamic.
> ...
> $ pminfo -dfl cgroup.cpuacct.usage_percpu
>
> cgroup.cpuacct.usage_percpu
> Data Type: 64-bit unsigned int InDom: 3.22 0xc00016
> Semantics: counter Units: nanosec
> inst [0 or "/::cpu0"] value 7137875587296 labels cgroup="/" cpu="cpu0"
> inst [1 or "/::cpu1"] value 3539695017434 labels cgroup="/" cpu="cpu1"
> inst [2 or "/system.slice::cpu0"] value 407816293372 labels cgroup="/system.slice" cpu="cpu0"
> inst [3 or "/system.slice::cpu1"] value 202747983510 labels cgroup="/system.slice" cpu="cpu1"
I'm with Lukas here, this is nigh-on unreadable and difficult for shell
scripts to parse ... perhaps put the labels on a second line?
> ...
> - Context labels ("global") could be setup via /etc/pcp/pmcd/labels.d entries
> which are then exported via a pmcd.labels metric. This would provide support
> for both archive and host context labelling, and would be wrapped up via the
> pmGetContextLabels(3) API.
>
> - These labels.d files would be name=value (basename,= file contents) pairs,
> under sysadmin control (i.e. likely updated via puppet/chef/ansible/... etc).
I think it may be worth considering these files to contain 1 or more
name=value lines ... this avoids the label "name" being constrained to
this lowest common denominator of the filesystem names on the systems we
support (think /, \, ., :, <space> all of which _might_ be in a label
name, but should not be used in a filename).
> - We'll need rules around valid name and value characters (probably similar
> to PMNS naming conventions; no whitespace, limited special characters, etc,
> for the label names at least).
And for Prometheus, the name may begin with _ which is not quite the
same as a PMNS name component.
>
>
> Implementation notes
> ====================
> ...
Comments on pmapi.h patch
- pmLabels
+ probably should be pmLookupLabels (to follow the pm<op><thing>
naming convention for PMAPI methods
+ what are the semantics of the first in argument?
+ pmID rather than pmID * (like elsewhere)
+ how is an instance specified, or is that the first argument?
- pmLabelSet
+ why pmid here?
+ numinst why?
- pmLabels
+ why inst?
+ labslen -> numlabels?
+ labsbuf -> char **labname; char **labvalue;
This makes processing the labels much easier and clients above the
PMAPI do not need to parse a formatted buffer of
name=<quote>value<quote><separator-or-end-of-buffer> pairs. The whole
pmLabels thing can still be a single malloc()'d area, so free() works.
> Possible optimisations, extensions, and other random ideas:
> ...
> - once the initial functionality is in place, could extend the fetch
> profile concept such that labels could be used by client tools to
> filter fetch results sampled/returned.
The whole fetch profile thing is a minefield (that has been the site of
some nasty injuries in the past) ... there is no ACK PDU, it is
currently exposed at both ends of the IPC channel to pmUseContext()
changes, etc. So be very careful about "extending" this label-based
filtering.
Label-based filtering => label expressions involving values, value
ranges, value sets, regular expressions over values, relational
operators (==, !=, >, <, ...), boolean operators (&&, || and not), etc.
Once you have this, the existing instance profile could become a
instance=<expr> under the hood and exposed as an alternative
specification method at the PMAPI. This would promote the instance
profile to a first-class label concept, rather than presenting two
different filtering methods.
Some other thoughts:
Frank's comments on the archive format not being backwards compatible
... we should consider if there is some clever way to hide this ... one
wild-arse idea is to add a <basename>.labels file to go alongside
<basename>.{N,index,meta} which would work I think, but I have no idea
what the format/contents of <basename>.labels would be, particularly if
a specific label value might vary in time (over the life of the archive)
like a dynamic instance domain.
-=-=-=-=-=-=-=-=-=-=-=-
pcp mailing list
[email protected]
https://groups.io/g/pcp/messages
-=-=-
Groups.io Links:
You receive all messages sent to this group.
View/Reply Online (#14797): https://groups.io/g/pcp/message/14797
View All Messages In Topic (11): https://groups.io/g/pcp/topic/3188520
Mute This Topic: https://groups.io/mt/3188520?uid=174580
New Topic: https://groups.io/g/pcp/post
Change Your Subscription: https://groups.io/g/pcp/editsub?uid=174580
Group Home: https://groups.io/g/pcp
Contact Group Owner: [email protected]
Terms of Service: https://groups.io/static/tos
Unsubscribe: https://groups.io/g/pcp/leave/354243/563757577/xyzzy
-=-=-=-=-=-=-=-=-=-=-=-