Re: RFC: metric labels as first-class PCP concept

Hi -


> > If the effort were to involve changing the archive file format, it
> > would no longer be backward compatible in the sense that older clients
> > cannot manipulate newer files.  This is a property that we had
> > retained for quite some time.
> 
> What you're describing is "forwards compatibility" which is not something
> we have ever promised [...]

Not sure about our entire public statement history, but we have
retained that property nevertheless.


> [...]  Otherwise the argument boils down to "we can never change the
> PCP archive version"- which is clearly nonsense [...]

No one said that.  It would simply be an incompatible change, no more,
no less.


> >  (The better PMDAs take advantage of [instance profiles].)
>
> Well *all* PMDAs having metrics-with-instances do that - its
> implemented in the libpcp_pmda fetch code (like label-based
> filtering will be eventually - as long as we get the basics modeled
> well) - i.e. outside of the PMDA.

You're right to the extent that PMDAs are told of the instance
profiles via some of the fetch callbacks.  However, several PMDAs do
not use that knowledge to fully optimize their internal behaviour,
what I described as "not [wasting] effort collecting data on
uninteresting instances", never mind gathering data for uncalled-for
metrics.


> > > [...]  - Context labels ("global") could be setup via
> > > /etc/pcp/pmcd/labels.d entries [...]
> > Depending on machinery, you may not need any extra API for that --

> The implementation would be just like pmGetContextHostName (i.e. its
> a metric underneath, but having that clear, simple API is important
> too.

No, pmGetContextHostName uses a metric (pmcd.hostname) only in the
CONTEXT_HOST case.


> > - global labels == JSON string valued ordinary metrics (if necessary at all)
> 
> Global labels are clearly necessary.  They seem to be the most common use
> of labels from what I can tell, looking at how labels are being used by
> other tools.  More discussion on using JSON later, but I'm not totally
> against that aspect.

Global labels are clearly necessary **in other tools** because they
store intermingled results from several monitoring targets in a single
database, and must use those labels to disentangle them again.  PCP
archives and host connections do not intermingle.  

"global labels" could be (almost?) as easily assigned by the pcp
client based on its specific reason for initiating the connection.
For example, a hypothetical kubernetes-aware pmmgr would already know
labels from k8s for all the remote nodes/containers it's connecting
to.


> > - metric labels == JSON blobs in metric long-description strings

> [...]  Its clearly separate metadata, lets do it cleanly; abusing
> help text like this is just ... wow, no.
> We also do not log metric help text.  

That is a good point, but surmountable with the "peer metric"
alternative implementation.

> thought balloon *pop*

Please be serious.


> > - metric-instance labels == JSON blobs in indom instance-name strings
> 
> That's just so wrong as well.  Labels need to be a first-class concept
> so that we can implement and use them properly throughout PCP.

That is circular reasoning.


> [...]  The simple (non-JSON) name=value concept used by other tools
> has the advantage of being *super simple* - it is consistent, always
> easy to use - never any concern about parsing it [...]

When you have an over-simple notation, you lose the ability to express
complicated things.  "name=value name=value" loses if value is
something like the string "value name=".  Thus the suggestion to use
an existing, well-defined format that governs quotations,
character-set encodings, syntax, data types, and the like.  Of course
you can limit nesting etc. if they turn truly horrible, and your code
can't handle it.  Such data would come from PCP, and presumably we can
police ourselves not to make unnecessary complex labels.


> [...] [It's] possible we might want to add a libpcp_pmda label-merge
> helper facility too, ala "all metrics below oracle. are labeled with
> xxx" - becomes significantly complicated via JSON.

You barely discussed how you imagine this being implemented
operationally down at the PMDA level.  Would a client context have a
label selection predicate associated with it that would have to be
passed to all the pmdas for interpretation?  For each fetch?  (That
would be as inefficient as the container/pmdaroot implementation is
now.)

In my model, label filtering evaluation is entirely client-side, and
requires zero machinery at the PMDA, beyond instance profiles.  That
would put responsibility on the clients to adapt to new instances, to
reevaluate label selection predicates for them when they are seen, but
that's doable.  I can sketch out some pseudocode if required.


> > Filtering - [...] with existing instance-profile machinery.  A client would
> > have to have a way of enumerating the entire indom (which they already
> > can, even for the final state of an archive indom), parse the labels,
> > apply any filtering predicates, then pass the remaining selected
> > instances to the plain PMAPI machinery via vanilla instance profiles.
> 
> This works basically the same in the original proposal using first-class
> labels rather than monster instance names and overloading the metric help
> text.

Well, no - in my proposal, filtering is done entirely by the client,
not by passing label selection predicates to the server.


> [...]
> > Downsides: Some text would be uglified (which clients can hide), and
> 
> You can't hide those awful instance names

Yes, you can.

>  - pmchart, pmval, pmie, ...  lots of the tools expect some kind of
> meaningful instance name - and an instance name with a JSON document
> appended to it is not that, sorry.

These tools are all shipped with PCP.  We can give them a flag, or
even hard-code behaviour, to clip off the "@label ..." parts, as a UI
policy matter.


> [...] I pondered a way to do labels with what you call "sibling"
> metrics too, there's many (other) flaws - those metrics might not
> get logged

That is a very small matter of programming in pmlogger: it could
implicitly add the "sibling metrics" for all normal metrics.

> its heavyweight, & others.

I don't know what you mean.  A SEM_DISCRETE TYPE_STRING metric
updated almost never consumes almost no time / space.


> > Upsides: no change to network protocol nor archive file format, thus
> > complete compatibility.
> 
> And that's been shown to be incorrect.

No, it is still correct.  If metric labels were represented as values
in peer metrics, rather than help text, there is no format
archive/etc. change required.


> Labels are really a separate metadata concept.  They definitely
> should not be shoe-horned in this fashion IMO - let's do it cleanly.

They are not so separate - as your own use case identified labels as a
way to add extra dimensions to instance domains.  You also mentioned
the need to vary labels - that necessitates treating them as values
along a timeline, just like normal metrics.  The concepts are more
similar than they may seem.

One can skin this cat in several ways, with vastly different
quantities of change, one just needs to be selectively creative.


- FChE

-=-=-=-=-=-=-=-=-=-=-=-
pcp mailing list
[email protected]
https://groups.io/g/pcp/messages
-=-=-
Groups.io Links:

You receive all messages sent to this group.

View/Reply Online (#14795): https://groups.io/g/pcp/message/14795
View All Messages In Topic (10): https://groups.io/g/pcp/topic/3188520
Mute This Topic: https://groups.io/mt/3188520?uid=174580
New Topic: https://groups.io/g/pcp/post

Change Your Subscription: https://groups.io/g/pcp/editsub?uid=174580

Group Home: https://groups.io/g/pcp
Contact Group Owner: [email protected]

Terms of Service: https://groups.io/static/tos

Unsubscribe: https://groups.io/g/pcp/leave/354243/563757577/xyzzy
-=-=-=-=-=-=-=-=-=-=-=-