Re: RFC: metric labels as first-class PCP concept

Hi -


> So, here's an initial design write-up on how this idea might make
> its way into PCP - not just as an add-on above the PMAPI, but
> built-in as a first class concept.

Thank you the sketching effort.

> We do not have design luxuries of a designed-from-scratch project
> and complete backward compatibility must be maintained.  However,
> the approach outlined below seems to me like it strikes a good
> balance in terms of added functionality while preserving
> compatibility.

If the effort were to involve changing the archive file format, it
would no longer be backward compatible in the sense that older clients
cannot manipulate newer files.  This is a property that we had
retained for quite some time.


> [...]
> Use Cases
> =========
> 
> 1 Additional metric metadata (e.g. for unsupported units, like temperature)
> 2 Describe compound instance domains more completely (cgroup example below)
> 3 End-user attributes for metrics, for higher level tools (e.g. Prometheus
>   and Grafana, and many others now) - useful for needs like identifying the
>   data centre a metric value came from, once a value has been exported into
>   the external tool (external tools to PCP, at least initially I expect).

While these are reasonable ways to start thinking about it, we'd also
have to think through the whole collection-analysis workflow.  Why do
we want to have labels on data?

- to have a little more metadata
- in order to make it easier to find / search / filter by that metadata

That second part is crucial.  Instance domains are useful not only
because they give us one extra metric dimension over the PMNS, but
because through instance profiles we can instruct pmdas not to waste
effort collecting data on uninteresting instances.  (The better PMDAs
take advantage of that.)  We can instruct libpcp to filter out
uninteresting instances from archive scans.


> [...]
> - In terms of data modeling, labels are inherently a server side
> concept - not a client side concept (i.e. we would not want to try
> to implement this along the lines of derived metrics, which are
> entirely client-side).

I'm not sure what that means.  Naturally, labels originate from the
server.  But what implementation option do you mean to exclude by
saying that it's "server side, not client side"?


> [...]
> - Context labels ("global") could be setup via /etc/pcp/pmcd/labels.d entries
> which are then exported via a pmcd.labels metric.  This would provide support
> for both archive and host context labelling, and would be wrapped up via the
> pmGetContextLabels(3) API.

Depending on machinery, you may not need any extra API for that --
maybe not even new PMDA nor PMCD code, as we have pmdapipe & pmdammv
to report on files from the server.


> Implementation notes
> ====================
> This requires changes everywhere of course (from PCP protocol, upward):
> [...]

Have you tried imagining this sort of facility as retrofitted into the
existing pcp infrastructure?  Something like this:

- global labels == JSON string valued ordinary metrics (if necessary at all)
- metric labels == JSON blobs in metric long-description strings
- metric-instance labels == JSON blobs in indom instance-name strings

e.g.:

% pminfo --desc --fetch --labels kernel.all.pswitch

... could generate the same output as your example, but instead of new
APIs, it would simply merge together the label JSON blobs from the
various sources, client side pmapi equivalents:

% pminfo -f pmcd.labels
@labels: {"datacenter":"sydney", "host":"acme1", "deployment":"production"}

% pminfo -T kernel.all.pswitch
Help:
context switches metric from /proc/stat
@labels: {"etc.":"etc."}

The instance label not applicable here, but for your
cgroup.cpuacct.usage_percpu example, all the above, plus the
decoding the instance name

    [0 -> '/::cpu0 @labels: {"cgroup":"/", "cpu"="cpu0"}'] 
    [1 -> '/::cpu1 @labels: {"cgroup":"/", "cpu"="cpu1"}'] 

etc.  So basically, the idea is to put labels as specially formatted
substrings in existing places where longer pieces of text are already
carried.  Clients can parse & pretty-print them.  It would let us
present higher dimensional instance domains.

Filtering - remember filtering? - can be done at the instance domain
level with the existing instance-profile machinery.  A client would
have to have a way of enumerating the entire indom (which they already
can, even for the final state of an archive indom), parse the labels,
apply any filtering predicates, then pass the remaining selected
instances to the plain PMAPI machinery via vanilla instance profiles.

Filtering at the metric and "global" level could be meaningful to
restrict PMNS enumeration to metrics of interest, e.g.

% pminfo --label-matches foo=bar cgroup

One can identify a relatively rich filtering language, all implemented
on the client side, within the tools and/or libpcp.

Downsides: Some text would be uglified (which clients can hide), and
label tuple changes would be limited to instance domain changes, and
the whole cartesian product of dynamic label keys/values would have to
fit into the 32-bit instance id.  (If metric labels also had to be
dynamic, that could be represented e.g. by a sibling metric carrying
labels as sem_discrete string values rather than as metadata, just
like the pmcd.labels global labels.  Contemplating the implementation
consequences, maybe not worth it.)

Upsides: no change to network protocol nor archive file format, thus
complete compatibility.


- FChE

-=-=-=-=-=-=-=-=-=-=-=-
pcp mailing list
[email protected]
https://groups.io/g/pcp/messages
-=-=-
Groups.io Links:

You receive all messages sent to this group.

View/Reply Online (#14782): https://groups.io/g/pcp/message/14782
View All Messages In Topic (4): https://groups.io/g/pcp/topic/3188520
Mute This Topic: https://groups.io/mt/3188520?uid=174580
New Topic: https://groups.io/g/pcp/post

Change Your Subscription: https://groups.io/g/pcp/editsub?uid=174580

Group Home: https://groups.io/g/pcp
Contact Group Owner: [email protected]

Terms of Service: https://groups.io/static/tos

Unsubscribe: https://groups.io/g/pcp/leave/354243/563757577/xyzzy
-=-=-=-=-=-=-=-=-=-=-=-