Re: RFC: metric labels as first-class PCP concept

Hey Ken,

"Ken McDonell" <[email protected]> writes:
[...]
>> So, the proposed approach is:
>> a/ keep the existing PCP metric and instance concepts unchanged - including
>>    naming and pmDesc metadata;
>> b/ extend those ideas with an (optional) "label" concept, allowing name=value
>>    pairs to be associated with metric values.  In a sense, labels would be
>>    similar to filesystem extended attributes;
>> c/ see labels as primarily static and optimise for that case, while allowing
>>    for some dynamic change (much like we do with PCP instance names).
>
> I think it is going to be really important to get straight in the
> beginning the _scope_ for labels.  I can see the following possible
> taxonomy (it is not really a hierarchy as the metric and instance
> "levels" are orthogonal):
> 1. context (the global stuff)
> 2A. metric (common to all instances of a given metric)
> 2B. one or more instances within an instance domain (common to those
> instances across all metrics that share the instance domain)
> 3. metric-instance (unique to a specific metric-instance pair)

I think this makes sense, and we should be able to support all of these
cases.

> Before delving too deep into the PMAPI requirements, this needs to be
> agreed to (are all 4 going to be supported?) and then what are the
> semantics for a label that is redefined (with a different, or possibly
> the same) value in another place?  For example, cute="foo" with the
> metric X (over the InDom I), and cute="bar" with instance i of InDom
> I, and cute="no clue" with instance i of metric X.

I chatted with a colleague earlier today that works in a similar area
(hawkular).  Hawkular support all of the above cases, however, they do
it on a per metric basis only.  Recognition that a label is common
throughout an entire Instance, or exceptions of an instance, is done (for
them), after the fact.  I think it's also important to allow dynamic
changes of labels, and importantly, record when the change occured
(keeping the old value where it applicable over the existing/recorded
values).

A typical use case of this having a server/pod/whatever move from dev ->
staging -> production (and perhaps back again).  Currently they're
facing an issue where if the label is changed, it changes for the
entirety of the dataset (ie. if we go from staging -> prod, the entire
metric label is the new value).  We should probably be mindful of this
case when making our decisions.

> And a somewhat related issue is points in time where the labels change
> (a label is added or deleted, or a value for a label changes) ... what
> happens for:
> - pmlogextract and pmlogreduce?
> - interpolate mode in libpcp?
> - multi-archive support in libpcp?
> - the "reconnect if I can" support in pmlogger?
> - and probably some other curly cases I haven't thought of yet

Perhaps we can treat them similarily to how we treat instantaneous
metrics in these cases?

> These problems exist with static labels (due to evolutionary change),
> but are magnified if labels are dynamic.

Right, but I think we're going to run into a demand for dynamic labels
eventually.  Hopefully we can at least design a static label
implementation that won't hem ourselves in when implementing dynamic
labels later.

[...]
>> - Context labels ("global") could be setup via /etc/pcp/pmcd/labels.d entries
>> which are then exported via a pmcd.labels metric.  This would provide support
>> for both archive and host context labelling, and would be wrapped up via the
>> pmGetContextLabels(3) API.
>>
>> - These labels.d files would be name=value (basename,= file contents) pairs,
>> under sysadmin control (i.e. likely updated via puppet/chef/ansible/... etc).
>
> I think it may be worth considering these files to contain 1 or more
> name=value lines ... this avoids the label "name" being constrained to
> this lowest common denominator of the filesystem names on the systems
> we support (think /, \, ., :, <space> all of which _might_ be in a
> label name, but should not be used in a filename).
>> - We'll need rules around valid name and value characters (probably similar
>> to PMNS naming conventions; no whitespace, limited special characters, etc,
>> for the label names at least).

Just to add on here (from my discussion with a colleague).  He was
emphatically in agreement here with creating basic rules for a valid
label.  I think it makes sense (and we wouldn't be the first) to avoid
something like whitespace in key values.  Additionally, not leave things
open ended to develop into tricky corner cases later would be great.
He even mentioned designating a character (such as a period, but could be
anything else) that we could explicitly disallow in a label name, but
then use to query labels later on, would make it easier for
using array or map labels:

Metrics disk.dev.*
label: Vendors: [foo, bar, baz]
or
label: vendor_foo=True
label: vendor_bar=True
label: vendor_baz=True

and then looking for labels where 'vendor.foo=True' if we've reserved
the '.' from use within a label name.  In the latter case, with
individual labels for each vendor, I think it not only becomes more
verbose, but will end up taking up more room.

Aside from the fact I find the former label visually, I'd like to take a
chance to advocate for the array type again here.

Under the assumption we're using labels as a means to help search
through and help correlate/understand the metrics being reported, there
are a couple situations I can think of that we'd find it useful.

System Ownership/Purpose
ex.
System_ownership: [Devteam1, sysops2, oncall1]
various teams responsible for the maintanence of a box or have code running on that
box, which, again, could change, which would be another reason to have
dynamic values.  Could even be as specific as individuals themselves.

system_components: [product_v3, backend_v1, frontend_v1]
versions of different software bits on the machine that possibly don't
have corresponding metrics/pmdas.  Again, label values like these would
naturally change over time.

I'm sure there are more I haven't thought of yet.

Cheers,

Lukas





-=-=-=-=-=-=-=-=-=-=-=-
pcp mailing list
[email protected]
https://groups.io/g/pcp/messages
-=-=-
Groups.io Links:

You receive all messages sent to this group.

View/Reply Online (#14804): https://groups.io/g/pcp/message/14804
View All Messages In Topic (14): https://groups.io/g/pcp/topic/3188520
Mute This Topic: https://groups.io/mt/3188520?uid=174580
New Topic: https://groups.io/g/pcp/post

Change Your Subscription: https://groups.io/g/pcp/editsub?uid=174580

Group Home: https://groups.io/g/pcp
Contact Group Owner: [email protected]

Terms of Service: https://groups.io/static/tos

Unsubscribe: https://groups.io/g/pcp/leave/354243/563757577/xyzzy
-=-=-=-=-=-=-=-=-=-=-=-