Re: unprivileged collection on containers

On Thu, Feb 2, 2017 at 3:58 PM, Nathan Scott <[email protected]> wrote:

> I set about finding out whether or not this works already.  Refer to
> the attached Dockerfile, based on Dan Walsh's systemd+httpd blog post
> https://developers.redhat.com/blog/2016/09/13/running-
> systemd-in-a-non-privileged-container/
>

The PCP containers infrastructure we have now was developed ~ two years
ago. At the time, the systemd / pid-1 discussions were in full swing but
there was no working init system based on systemd available. COntainers
were all built with the application being "pid-1", i.e. init.

So we made the design decision for one container per-PCP service, which is
what we have today. There are strong arguments for both approaches - it
really depends on whether the application needs an init system (for
multiple services and other infrastructure), or whether it just runs one
main process (pid-1) in isolation. In the case of PCP, having systemd
manage multiple services makes more sense and is likely the approach we
would have taken if the choice was available back then. With
one-container-per-service, we need bind mounts for all the important PCP
directories (so they are shared by all PCP services in different
containers), along with super-privileged access to all the namespaces
(especially pid=host) - we ended up effectively running all PCP services on
the host namespaces and filesystem .. which kind of defeats using
containers - PCP might as well just be installed directly on the host!


> Looks like it should work out-of-the-box.  Application instrumentation
> is available with the MMV PMDA there too, at no extra cost.
>
> I've setup the Dockerfile to present pmcd only via unix-domain socket,
> to avoid the port-numbering issues, but the container could expose the
> pmcd port (44321) alongside Apache (80) quite easily too - the naming
> is more clear using filesystem paths over port numbers.
>
> # pminfo -h unix:/run/containers/httpd/pcp/pmcd.socket -f proc.memory.rss
>
> proc.memory.rss
>     inst [1 or "000001 /sbin/init"] value 5120
>     inst [27 or "000027 /usr/lib/systemd/systemd-journald"] value 6512
>     inst [39 or "000039 /usr/bin/dbus-daemon"] value 3768
>     inst [41 or "000041 /usr/sbin/httpd"] value 9996
>     inst [110 or "000110 /usr/sbin/httpd"] value 7376
>     inst [111 or "000111 /usr/sbin/httpd"] value 7376
>     inst [115 or "000115 /usr/sbin/httpd"] value 7376
>     inst [120 or "000120 /usr/sbin/httpd"] value 7376
>     inst [139 or "000139 /usr/sbin/httpd"] value 7376
>     inst [149 or "000149 /usr/libexec/pcp/bin/pmcd"] value 5216
>

Ok so that seems to work well, and is fine provided you're only interested
in the pid namespace of the container rather than the entire system.


>
> > If someone were to need this in a hurry, what would you say?
>
> I'd say resourcing and scheduling of Red Hat projects are more suited
> to internal discussion, not public lists like this one.
>
> In terms of PCP community relevance though, I'd say (assuming Mark/anyone
> else sees no issues) we could add a "pcp-standalone" container into our
> current set, separating out PCP related configuration from the Apache stuff
> in the Dockerfile attached, adding Parfait and UoM for some examples beyond
> Apache, and that could be shipped on bintray... thoughts Mark?
>

All for a new suite of pcp-standalone containers, managed by systemd under
the hood. These can target specific application monitoring requirements
(httpd, parfait, etc etc). It'll be far more deployable without the need
for so many bind-mounts and scary privileges. Multiple containers running
the same PCP services should be able to run on the same host without
interference, though we may need some way of coordinating exported tcp/inet
ports, etc.  And containerized QA should be more or less the same as
running in a VM, which would be tremendous.

Nathan - let's discuss this one some more later today.

Regards
-- Mark