Re: RFC: metric labels as first-class PCP concept

tl;dr - alternate "labels as second-class PCP concept" approach fell
down; labels as JSON instead of simpler name=value pairs uncertain.

----- Original Message -----
> [...]
> > We do not have design luxuries of a designed-from-scratch project
> > and complete backward compatibility must be maintained.  However,
> > the approach outlined below seems to me like it strikes a good
> > balance in terms of added functionality while preserving
> > compatibility.
> 
> If the effort were to involve changing the archive file format, it
> would no longer be backward compatible in the sense that older clients
> cannot manipulate newer files.  This is a property that we had
> retained for quite some time.

What you're describing is "forwards compatibility" which is not something
we have ever promised (nor delivered - guess what came before V.2 logs?).
Backwards compatibility is retained completely in the earlier design.

The wire protocol has the same issue, FWIW.  And we know how to add in new
features like this now, it has been done it plenty of times.  We also know
an archive format bump is on the cards in the not too distant future (for
compression and indom deltas).  So that's not an overly concerning problem
and certainly not a reason for the radical design compromises suggested.

The future pmlogger(1) that supports labels will gain a flag saying "don't
generate V.3 archives".  New libpcp will support both V.2 and V.3 archives
and old libpcp will cleanly report lack of support for V.3 archives.  That
is backwards compatibility and that's why we have a version number stamped
into every PCP archive.

Otherwise the argument boils down to "we can never change the PCP archive
version"- which is clearly nonsense as its already been done once before.
We should not do it often, however, it has a big impact.  I also wonder if
we should introduce a feature bitfield in V.3 (thoughts Ken?) so that it's
not an all-or-nothing-affair when introducing new features.

> > [...]
> - in order to make it easier to find / search / filter by that metadata
> 
> That second part is crucial.

Yes, as mentioned (later on) in the original message - I should have added
an explicit use case for that I guess, it would have been clearer.

>  (The better PMDAs take advantage of [instance profiles].)

Well *all* PMDAs having metrics-with-instances do that - its implemented in
the libpcp_pmda fetch code (like label-based filtering will be eventually -
as long as we get the basics modeled well) - i.e. outside of the PMDA.

> > [...]
> > - Context labels ("global") could be setup via /etc/pcp/pmcd/labels.d
> > entries
> > which are then exported via a pmcd.labels metric.  This would provide
> > support
> > for both archive and host context labelling, and would be wrapped up via
> > the
> > pmGetContextLabels(3) API.
> 
> Depending on machinery, you may not need any extra API for that --

The implementation would be just like pmGetContextHostName (i.e. its a
metric underneath, but having that clear, simple API is important too.

Actually we probably need another API - to implement label merge rules.
i.e. take two name=value group strings & produce sort'd and uniq'd set
applying the precedence rules.

> Have you tried imagining this sort of facility as retrofitted into the
> existing pcp infrastructure?  Something like this:

Yes, albeit in different ways to this.  Setting aside the implementation
kludgey-ness there are unsolvable problems with your suggestion, esp. the
help-text part.  And those instance names - oh, the horror.

> - global labels == JSON string valued ordinary metrics (if necessary at all)

Global labels are clearly necessary.  They seem to be the most common use
of labels from what I can tell, looking at how labels are being used by
other tools.  More discussion on using JSON later, but I'm not totally
against that aspect.

> - metric labels == JSON blobs in metric long-description strings

*shudder*.  I had to re-read that several times, almost can't believe you
went there!

Its clearly separate metadata, lets do it cleanly; abusing help text like
this is just ... wow, no.

We also do not log metric help text.  So, this would require that format
change you were using to justify the kludge - which makes the "labels as
second-class PCP concept" thought balloon *pop*

> - metric-instance labels == JSON blobs in indom instance-name strings

That's just so wrong as well.  Labels need to be a first-class concept
so that we can implement and use them properly throughout PCP.

> % pminfo -f pmcd.labels
> @labels: {"datacenter":"sydney", "host":"acme1", "deployment":"production"}

OK, so returning to global labels & JSON - I could see JSON working for
the label values, possibly, maybe.

It does raise concerns that it will quickly get abused via deep levels
of nesting and inconsistencies between JSON tags, giving client tools
little to work with - in which case the value of adding labels could
become lost (if we use JSON, we'd have to have mandatory fields like
"name" and "value" at least I think to provide some basic client-side
certainty).

The simple (non-JSON) name=value concept used by other tools has the
advantage of being *super simple* - it is consistent, always easy to
use - never any concern about parsing it, no special parsers needed,
no real syntax errors, etc; can be easily managed in the shell.  But
... hmm ... maybe ... I'm a bit on the fence re JSON for this.

> etc.  So basically, the idea is to put labels as specially formatted
> substrings in existing places where longer pieces of text are already
> carried.  Clients can parse & pretty-print them.  It would let us
> present higher dimensional instance domains.

I don't like the overloading aspects at all, and I'm 50/50 on the use
of JSON for labels.  On the one hand, its more - alot more - complex
than simple name=value pairs, but on the other hand we could describe
the labels themselves more deeply, I guess; unless it becomes anarchy
in which case using JSON will have been a disastrous decision.

Another down side is that there is a need to merge the global and per-
value labels into a single set - that task is made more complex using
JSON over simple name=value pairs.  Its possible we might want to add
a libpcp_pmda label-merge helper facility too, ala "all metrics below
oracle. are labeled with xxx" - becomes significantly complicated via
JSON.

> Filtering - [...] with existing instance-profile machinery.  A client would
> have to have a way of enumerating the entire indom (which they already
> can, even for the final state of an archive indom), parse the labels,
> apply any filtering predicates, then pass the remaining selected
> instances to the plain PMAPI machinery via vanilla instance profiles.

This works basically the same in the original proposal using first-class
labels rather than monster instance names and overloading the metric help
text.

> Filtering at the metric and "global" level could be meaningful to
> restrict PMNS enumeration to metrics of interest, e.g.
> 
> % pminfo --label-matches foo=bar cgroup

Yep.  (no different to original proposal)

> One can identify a relatively rich filtering language, all implemented
> on the client side, within the tools and/or libpcp.

Totally, that's a big part of the whole labeling concept.  But that is
a secondary step, I'm mostly concerned about efficiently representing
labels at the lowest levels at this stage and we'll build higher-level
concepts like this later.

> Downsides: Some text would be uglified (which clients can hide), and

You can't hide those awful instance names - pmchart, pmval, pmie, ...
lots of the tools expect some kind of meaningful instance name - and an
instance name with a JSON document appended to it is not that, sorry.

> label tuple changes would be limited to instance domain changes, and
> the whole cartesian product of dynamic label keys/values would have to
> fit into the 32-bit instance id.  (If metric labels also had to be
> dynamic, that could be represented e.g. by a sibling metric carrying
> labels as sem_discrete string values rather than as metadata, just
> like the pmcd.labels global labels.  Contemplating the implementation
> consequences, maybe not worth it.)

Yeah, that's alot of downsides to start with.  I pondered a way to do
labels with what you call "sibling" metrics too, there's many (other)
flaws - those metrics might not get logged, its heavyweight, & others.

Happily, instead of compromising like this, there's no reason not to
implement labels as a first-class concept in PCP and all of those
problems go away.

> Upsides: no change to network protocol nor archive file format, thus
> complete compatibility.

And that's been shown to be incorrect.

Labels are really a separate metadata concept.  They definitely should
not be shoe-horned in this fashion IMO - let's do it cleanly.

cheers.

ps: another random thought bubble ... should PM_TYPE_STRING be split
into separate PM_TYPE_STRING and PM_TYPE_JSON? (as we're going to be
rev'ing archive format, protocol, libpcp_pmda version, etc)

--
Nathan

-=-=-=-=-=-=-=-=-=-=-=-
pcp mailing list
[email protected]
https://groups.io/g/pcp/messages
-=-=-
Groups.io Links:

You receive all messages sent to this group.

View/Reply Online (#14786): https://groups.io/g/pcp/message/14786
View All Messages In Topic (6): https://groups.io/g/pcp/topic/3188520
Mute This Topic: https://groups.io/mt/3188520?uid=174580
New Topic: https://groups.io/g/pcp/post

Change Your Subscription: https://groups.io/g/pcp/editsub?uid=174580

Group Home: https://groups.io/g/pcp
Contact Group Owner: [email protected]

Terms of Service: https://groups.io/static/tos

Unsubscribe: https://groups.io/g/pcp/leave/354243/563757577/xyzzy
-=-=-=-=-=-=-=-=-=-=-=-