RFC: metric labels as first-class PCP concept

Hi all,

A month or two ago Lukas and I met up with our colleague Peter Portante,
and he described the increasing importance he's observed in metric value
labeling by some popular new system performance tools - e.g. Grafana and
Prometheus.  That prompted thinking about how we might use similar sorts
of concepts in PCP to solve some existing problems, extend the toolkit,
and to improve the integration between PCP and these new tools.

So, here's an initial design write-up on how this idea might make its way
into PCP - not just as an add-on above the PMAPI, but built-in as a first
class concept.  We do not have design luxuries of a designed-from-scratch
project and complete backward compatibility must be maintained.  However,
the approach outlined below seems to me like it strikes a good balance in
terms of added functionality while preserving compatibility.


Concepts
========
Time series data can be seen as a stream of timestamped values belonging
to the same metric and the same set of labeled dimensions.  A value time
series identity is directly tied to the metric name (as well as instance
name, in PCP) and its labels.  The order in which labels are presented is
not significant.  Labels can be used for many purposes, most commonly for
querying and filtering time series data.  Some other projects implemented
this notion from the beginning, some examples (and more background):

    https://prometheus.io/docs/concepts/data_model/
    http://metrics20.org/


PCP Today
=========
From that metrics20 spec, we clearly have a number of "labels" already; we
capture that as metric metadata (pmLookupDesc(3) - PMAPI pmDesc structure).
However, we wish to augment that metadata now with system and user-defined
metadata values.

In addition, we have some long-standing modeling problems in PCP that we
could use labelling to tackle (Use Cases #1 and #2 below).


Use Cases
=========

1 Additional metric metadata (e.g. for unsupported units, like temperature)
2 Describe compound instance domains more completely (cgroup example below)
3 End-user attributes for metrics, for higher level tools (e.g. Prometheus
  and Grafana, and many others now) - useful for needs like identifying the
  data centre a metric value came from, once a value has been exported into
  the external tool (external tools to PCP, at least initially I expect).


Possible approach in PCP
========================

Looking into the sorts of things people use labels for, it appears there are
two primary cases for PCP to capture.  Firstly, metadata associated with the
source of the values (hostname, data centre, and so on).  Secondly, there is
metadata that may be associated with the individual values (metrics and/or
instances).

So, the proposed approach is:
a/ keep the existing PCP metric and instance concepts unchanged - including
   naming and pmDesc metadata;
b/ extend those ideas with an (optional) "label" concept, allowing name=value
   pairs to be associated with metric values.  In a sense, labels would be
   similar to filesystem extended attributes;
c/ see labels as primarily static and optimise for that case, while allowing
   for some dynamic change (much like we do with PCP instance names).

$ pminfo --desc --fetch --labels kernel.all.pswitch

kernel.all.pswitch
    Data Type: 64-bit unsigned int  InDom: PM_INDOM_NULL 0xffffffff
    Semantics: counter  Units: count
    value 661089250 labels datacentre="sydney" host="acme1" deployment="production"

$ pminfo -dfl cgroup.cpuacct.usage_percpu

cgroup.cpuacct.usage_percpu
    Data Type: 64-bit unsigned int  InDom: 3.22 0xc00016
    Semantics: counter  Units: nanosec
    inst [0 or "/::cpu0"] value 7137875587296 labels cgroup="/" cpu="cpu0"
    inst [1 or "/::cpu1"] value 3539695017434 labels cgroup="/" cpu="cpu1"
    inst [2 or "/system.slice::cpu0"] value 407816293372 labels cgroup="/system.slice" cpu="cpu0"
    inst [3 or "/system.slice::cpu1"] value 202747983510 labels cgroup="/system.slice" cpu="cpu1"

That is where we're aiming to end up, in terms of pmcd (server) <-> pminfo
(client) interaction representing everything that happens "under the hood"
- these examples present use cases #3 and #2 respectively, as well.


Some design notes
=================
- See attached pmapi.h patch for proposed client-side PMAPI extension,
  regarding calling conventions for client tools needing access to labels.

- In terms of data modeling, labels are inherently a server side concept -
not a client side concept (i.e. we would not want to try to implement this
along the lines of derived metrics, which are entirely client-side).

- Want to ensure labels can be present at the level of individual instances
of metrics, not just per-metric and not just per-host.

- In-built metadata (pmDesc) should not be explicitly communicated as labels,
but could be turned into labels if the client tool consuming labels wanted to
use them in that way (as in the metrics20.org labels).  As we already have
protocols / disk formats for pmDesc, these should not be re-presented at the
network PDU / ondisk levels.

- To form full set of available labels for an individual metric, client tools
form the combined list from: context labels + metric/instance (value) labels.

- Context labels ("global") could be setup via /etc/pcp/pmcd/labels.d entries
which are then exported via a pmcd.labels metric.  This would provide support
for both archive and host context labelling, and would be wrapped up via the
pmGetContextLabels(3) API.

- These labels.d files would be name=value (basename,= file contents) pairs,
under sysadmin control (i.e. likely updated via puppet/chef/ansible/... etc).

- We'll need rules around valid name and value characters (probably similar
to PMNS naming conventions; no whitespace, limited special characters, etc,
for the label names at least).


Implementation notes
====================
This requires changes everywhere of course (from PCP protocol, upward):
  o  impl.h: new PDU type, PDU_FLAG_LABELS credentials feature bit, and
     client/server negotiation during pmNewContext (as with other recent
     optionally-available features)
  o  libpcp: implement pmLabels(3) API, in two stages:
  o  libpcp: new src/libpcp/src/p_labels.c sources; structure definition
     from attached pmapi.h patch needs to be over-the-wire transportable
  o  libpcp: archive support for storing the same, which means an on-disk
     version bump.  We'll possibly want to consider other on-disk changes
     (like compression) and make a single V3 on-disk change, in one step.
  o  pmdapmcd: export pmcd.labels as per above
  o  pmdapmcd: export pmcd.features.labels (0/1, off/on)
  o  pminfo, pmprobe: options for reporting labels
  o  libpcp_pmda: PMDA_INTERFACE_7 with a new labels interface, e.g.
	int     (*labels)(int, pmID *, pmAnnotation **, pmdaExt *);
  o  libpcp_pmda: helper routine(s) for PMDAs using labels, probably to
     build the name=value buffers via callback like is done for pmFetch:
typedef int (*pmdaFetchCallBack)(pmdaMetric *, unsigned int, pmAtomValue *);
typedef int (*pmdaLabelsCallBack)(pmdaMetric *, unsigned int, char **);
     (indom profiles work for metric annotations as for fetch results.)
  o  pmcd: add support for the new LABELS PDU, and support combining the
     requests from clients much like pmFetch (i.e. different pmIDs means
     different agents responding).
  o  libpcp: implement pmGetContextLabels(3) API:
  o  pmcd: add support for /etc/pcp/pmcd/labels.d and pmcd.labels metric
  o  export metric labels as metadata via pmwebd _metric operation.

Possible optimisations, extensions, and other random ideas:
  - pmDesc would ideally flag the presence of labels on a metric, so
    clients can tell if pmLabels(3) roundtrip is needed. One possible 
    place to indicate this is in the pmDesc.units struct, which has a
    zero-filled padding field available.
  - do we need to distinguish between labels set for an instance, to
    labels set (once) for a metric?  probably, as it'd be inefficient
    otherwise (repeatedly sending/storing dup labels).  The padding
    field in pmDesc.units struct could hold this second flag also.
  - once the initial functionality is in place, we could add helper
    routines to aid in making labels settable at the level of a PMNS
    sub-tree
  - we will need to think about how client tools (and possibly above
    PMNS sub-tree thing) form the labels set, in case of overlapping
    label names.  Precedence rules will need to be defined.
  - might want a verbose mode to client tools like pminfo/pmprobe to
    report pmDesc metadata as labels too, e.g. units="KByte / sec" and
    type="unsigned 32-bit integer" and so on.
  - might want a little meta-language whereby metrics can be substitued
    into values for context labels (e.g. pmcd.hostname, pmcd.container,
    pmcd.timezone, etc) - when building the pmcd.labels value, that is,
    parsing the /etc/pcp/pmcd.labels.d file contents.
  - once the initial functionality is in place, could extend the fetch
    profile concept such that labels could be used by client tools to
    filter fetch results sampled/returned.


cheers.

--
Nathan

-=-=-=-=-=-=-=-=-=-=-=-
pcp mailing list
[email protected]
https://groups.io/g/pcp/messages
-=-=-
Groups.io Links:

You receive all messages sent to this group.

View/Reply Online (#14756): https://groups.io/g/pcp/message/14756
View All Messages In Topic (1): https://groups.io/g/pcp/topic/3188520
Mute This Topic: https://groups.io/mt/3188520?uid=174580
New Topic: https://groups.io/g/pcp/post

Change Your Subscription: https://groups.io/g/pcp/editsub?uid=174580

Group Home: https://groups.io/g/pcp
Contact Group Owner: [email protected]

Terms of Service: https://groups.io/static/tos

Unsubscribe: https://groups.io/g/pcp/leave/354243/563757577/xyzzy
-=-=-=-=-=-=-=-=-=-=-=-

diff --git a/src/include/pcp/pmapi.h b/src/include/pcp/pmapi.h
index c0e59b0..7b8cb1d 100644
--- a/src/include/pcp/pmapi.h
+++ b/src/include/pcp/pmapi.h
@@ -66,9 +66,13 @@ typedef struct pmUnits {
     unsigned int	scaleSpace : 4;	/* one of PM_SPACE_* below */
     unsigned int	scaleTime : 4;	/* one of PM_TIME_* below */
     signed int		scaleCount : 4;	/* one of PM_COUNT_* below */
-    unsigned int	pad : 8;
+    unsigned int	pad : 6;
+    unsigned int	labelsSet : 1;	/* labels-are-present flag */
+    unsigned int	labelsAll : 1;	/* same labels on all instances */
 #else
-    unsigned int	pad : 8;
+    unsigned int	labelsAll : 1;	/* same labels on all instances */
+    unsigned int	labelsSet : 1;	/* labels-are-present flag */
+    unsigned int	pad : 6;
     signed int		scaleCount : 4;	/* one of PM_COUNT_* below */
     unsigned int	scaleTime : 4;	/* one of PM_TIME_* below */
     unsigned int	scaleSpace : 4;	/* one of PM_SPACE_* below */
@@ -478,6 +482,36 @@ PCP_CALL extern int pmFetch(int, pmID *, pmResult **);
 PCP_CALL extern int pmFetchArchive(pmResult **);
 
 /*
+ * Support for annotating metric values with labels (name=value pairs).
+ * The full set of labels for a given metric instance is the union of
+ * those found at the source (host/archive) level and those set at the
+ * metric instance level.
+ */
+typedef struct pmLabels {
+    int     		inst;		/* instance identifier or PM_IN_NULL */
+    unsigned int	labslen;	/* length of all name=value strings; */
+    char		labsbuf[1];	/* space separated, null terminated. */
+} pmLabels;
+
+typedef struct pmLabelsSet {
+    pmID		pmid;		/* metric identifier */
+    int			numinst;	/* number of instances or error code */
+    pmLabels		*illist[1];	/* set of instances and their labels */
+} pmLabelsSet;
+
+typedef struct pmAnnotation {
+    int			numpmid;
+    pmLabelsSet		*lset[1];
+} pmAnnotation;
+
+PCP_CALL extern int pmGetContextLabels(pmAnnotation **);
+PCP_CALL extern int pmLabels(int, pmID *, pmAnnotation **);
+PCP_CALL extern void pmFreeAnnotation(pmAnnotation *);
+
+/* test metric descriptor (pmDesc) for presence of labels */
+#define PM_HAS_LABELS(descp) ((descp)->units.labelsSet == 1)
+
+/*
  * struct timeval is sometimes 2 x 64-bit ... we use a 2 x 32-bit format for
  * PDUs, internally within libpcp and for (external) archive logs
  */