Re: OpenVZ containers - how can you manage /dev/shm sharing?

Forwarding this on Michael Eager's behalf, as there was some delivery 
problem.


Subject: Re: [svlug] OpenVZ containers - how can you manage /dev/shm sharing?
To: Robert Freiberger <[email protected]>,
 "[email protected]" <[email protected]>
From: Michael Eager <[email protected]>
Date: Wed, 26 Sep 2018 11:14:27 -0700

On 09/26/2018 10:19 AM, Robert Freiberger wrote:
> Hello everyone,
> 
> Hope your week is going well! I'm very new to containers and most
> of my past work has been in the virtualized environments, recently
> I'm working more with containers and ran into a weird issue.
> 
> We have OpenVZ containers at work and some of these are running
> Jenkins in slave mode. To speed things up, these were set up to use
> the Jenkins workspace as a mount in */dev/shm* (tmpfs memory). So
> while working on these systems, they reported that during a build,
> the slaves would report out of memory errors.

You don't mention how many containers are running at the same time.

> The issue is that Jenkins copies over GB's of files to the
> workspace mounted in memory and exceeds the physical memory
> allocation for the VE (guest container). The problem is the HN
> (hardware host) has 144GB of memory and it's correctly allocating
> the */dev/shm* as 71GB (please correct me if I'm wrong here), which
> is showing up for the guest VE's as well, even though they only
> have 20GB of allocated physical memory.

I've used OpenVZ in the past and currently use LXC containers
extensively and I share file systems between host and container by
having them mounted in both.  Most are mounted from a file server
using NFS.  I've never tried to share a tmpfs file system between host
and container.  But here are my thoughts:

Allocating 71Gb of virtual memory on the host doesn't affect how much
physical memory is allocated.  Physical memory is allocated when the
pages are touched.  Physical memory allocation limits are independent
for each container and the host.  When you exceed that limit in the
container, allocation (or access) will fail.  If you want a container
to be able to access more than 20Gb of physical memory, whether tmpfs
or program, you need to increase this limit.

> Being new to containers, what is the proper method to share
> resources like */dev/shm* with containers? Should you
> oversubscribe the containers (risking the host go OOM) or keep hard
> limits?

OOM is a rather blunt tool, killing processes which may or may not be
the cause of the memory allocation problem.  OOM will only be
triggered when most pages are allocated and cannot be spilled to disk.
Since tmpfs has no backing store, all of the physical pages would be
locked.
Over-commit is a problematic policy.  Then again, you currently have
memory limit failures.

Are the host and containers sharing the same tmpfs file system?  This
would sound problematic.  Most file systems do not support being
accessed or modified by two different systems at the same time.  Even
if one system is RW and the others are RO, you may get errors.  I
would not mount a file system on both host and container unless the
file system is designed for this.


-- 
Michael Eager    [email protected]
1960 Park Blvd., Palo Alto, CA 94306

----- End forwarded message -----

_______________________________________________
svlug mailing list
[email protected]
http://lists.svlug.org/lists/listinfo/svlug