Re: RFC: metric labels as first-class PCP concept

Hi Ken,

----- Original Message -----
> Tardiness is not lack of interest, I've been thinking ... at it takes a
> little longer these days as my PCP engagement is pro bono.

No problem at all - I'd take quality replies over quantity any day.

> I am not opposed to this proposal, although the motivational use cases
> are outside my direct experience, except for the "extensible pmUnits"
> one, but the proposal does not really address that because there is no
> way to load code to handle the additional units, like "temperature".

Keeping in mind a major consumer of labels will be software layered above
PCP (like Grafana & Prometheus, which will be using them for filtering).
So even with no use of labels in PCP itself (although I'd hope we do use
'em alot), there's still significant value.

> But I do for see some tricky semantic issues that need to be resolved.

*nod*, thanks for the insights.

> On 21/11/16 17:30, Nathan Scott wrote:
> > ...  Some other projects implemented
> > this notion from the beginning, some examples (and more background):
> >
> >     https://prometheus.io/docs/concepts/data_model/
> >     http://metrics20.org/
> 
> Couple of interesting side-notes:
> - What & Why section of the Metrics 2.0 web page reads like the words I
> was using to evangelize PCP in 1997 ... yep, 19 years ago!

Heh.

> I think it is going to be really important to get straight in the
> beginning the _scope_ for labels.  I can see the following possible
> taxonomy (it is not really a hierarchy as the metric and instance
> "levels" are orthogonal):
> 1. context (the global stuff)
> 2A. metric (common to all instances of a given metric)
> 2B. one or more instances within an instance domain (common to those
> instances across all metrics that share the instance domain)
> 3. metric-instance (unique to a specific metric-instance pair)
> 
> Before delving too deep into the PMAPI requirements, this needs to be
> agreed to (are all 4 going to be supported?) and then what are the

Yes, that's the goal (all 4) with the original proposal.

> semantics for a label that is redefined (with a different, or possibly
> the same) value in another place?  For example, cute="foo" with the
> metric X (over the InDom I), and cute="bar" with instance i of InDom I,
> and cute="no clue" with instance i of metric X.

I think a rule like "labels closest to the leaf nodes take precedence"
may be the way to go (i.e. instances trump all, and context labels are
lowest precedence) - simple, easy to remember and easy to apply.

> And a somewhat related issue is points in time where the labels change
> (a label is added or deleted, or a value for a label changes) ... what
> happens for:
> - pmlogextract and pmlogreduce?
> - interpolate mode in libpcp?
> - multi-archive support in libpcp?
> - the "reconnect if I can" support in pmlogger?
> - and probably some other curly cases I haven't thought of yet

Rule of thumb may be to use the same rules for labels as we do instances,
in these situations.  Not sure labels will interact with interp.c though.

> These problems exist with static labels (due to evolutionary change),
> but are magnified if labels are dynamic.

*nod* - that was behind my earlier thinking re logs that they'd need to
be treated in a similar fashion to instance names.

> I think it may be worth considering these files to contain 1 or more
> name=value lines ... this avoids the label "name" being constrained to
> this lowest common denominator of the filesystem names on the systems we
> support (think /, \, ., :, <space> all of which _might_ be in a label
> name, but should not be used in a filename).

+1

> > - We'll need rules around valid name and value characters (probably similar
> > to PMNS naming conventions; no whitespace, limited special characters, etc,
> > for the label names at least).
> 
> And for Prometheus, the name may begin with _ which is not quite the
> same as a PMNS name component.

*nod*

> >
> >
> > Implementation notes
> > ====================
> > ...
> 
> Comments on pmapi.h patch
> 
> - pmLabels
>    + probably should be pmLookupLabels (to follow the pm<op><thing>
> naming convention for PMAPI methods

Yes, heh - we're on the same page - the patch I sent doesn't compile
due to the function name being the data structure name too, and that
(pmLookupLabels) was exactly what I changed it to.

>    + what are the semantics of the first in argument?
>    + pmID rather than pmID * (like elsewhere)
>    + how is an instance specified, or is that the first argument?
> 
> - pmLabelSet
>    + why pmid here?
>    + numinst why?
> 
> - pmLabels
>    + why inst?
>    + labslen -> numlabels?
>    + labsbuf -> char **labname; char **labvalue;
>      This makes processing the labels much easier and clients above the
> PMAPI do not need to parse a formatted buffer of
> name=<quote>value<quote><separator-or-end-of-buffer> pairs. The whole
> pmLabels thing can still be a single malloc()'d area, so free() works.

In general, I attempted to follow pmFetch conventions (vaguely).  I'll
write up a man page and try to answer these questions there.

> The whole fetch profile thing is a minefield (that has been the site of
> some nasty injuries in the past) ... there is no ACK PDU, it is
> currently exposed at both ends of the IPC channel to pmUseContext()
> changes, etc.  So be very careful about "extending" this label-based
> filtering.

Yes.  I'm thinking along the lines of a new PDU at this stage.

> Label-based filtering => label expressions involving values, value
> ranges, value sets, regular expressions over values, relational
> operators (==, !=, >, <, ...), boolean operators (&&, || and not), etc.
> Once you have this, the existing instance profile could become a
> instance=<expr> under the hood and exposed as an alternative
> specification method at the PMAPI.  This would promote the instance
> profile to a first-class label concept, rather than presenting two
> different filtering methods.

Very nice.

> ... we should consider if there is some clever way to hide this ... one
> wild-arse idea is to add a <basename>.labels file to go alongside
> <basename>.{N,index,meta} which would work I think, but I have no idea

That's a clever idea.

> what the format/contents of <basename>.labels would be, particularly if
> a specific label value might vary in time (over the life of the archive)
> like a dynamic instance domain.

It would need similar properties to the .meta file handling of instances,
where timestamps are recorded along side the data.  If we do end up with a
archive format version bump I would vote for this to be a .meta addition,
with a TYPE_LABEL (alongside TYPE_DESC and TYPE_INDOM).

cheers.

--
Nathan

-=-=-=-=-=-=-=-=-=-=-=-
pcp mailing list
[email protected]
https://groups.io/g/pcp/messages
-=-=-
Groups.io Links:

You receive all messages sent to this group.

View/Reply Online (#14809): https://groups.io/g/pcp/message/14809
View All Messages In Topic (16): https://groups.io/g/pcp/topic/3188520
Mute This Topic: https://groups.io/mt/3188520?uid=174580
New Topic: https://groups.io/g/pcp/post

Change Your Subscription: https://groups.io/g/pcp/editsub?uid=174580

Group Home: https://groups.io/g/pcp
Contact Group Owner: [email protected]

Terms of Service: https://groups.io/static/tos

Unsubscribe: https://groups.io/g/pcp/leave/354243/563757577/xyzzy
-=-=-=-=-=-=-=-=-=-=-=-