Re: free: regression due to a different calculation of Used memory

Hi,
I am sorry for being persistent here but I would really appreciate some
further feedback on this. If there is no further action planned to
address the reported concern then I can certainly live with that. The
documentation (man page) should express that the value exludes shmem
which can lead to discrepancies when there is a lot of shared memory in
use, swap is not available globally or in a memory cgroup context and
likely some other cases I am not thinking of right now.

Thanks!

On Tue 06-04-21 08:57:45, Michal Hocko wrote:
> On Sat 03-04-21 16:26:58, Craig Small wrote:
> > Hi Michal,
> >   You probably need to follow through several changes to get to where we
> > are now.  The fact that something else is not reporting the same thing as
> > procps happens a lot because its both subjective and a lot of things
> > (including procps) have and are wrong about how to calculate certain memory
> > values.
> > 
> > I'd suggest reading the links at
> > https://gitlab.com/procps-ng/procps/-/commit/602b9a02ce2a3ed443b0b79fca457826da44d654
> > which discusses some of this issue.
> > It has a link to the freelist thread where this is change was made.
> 
> Thanks for the link. I have read through the discussion and I have to
> say I do agree with the reporter. It is good to see that the obviously
> broken slab accounting thing got fixed. I haven't checked the version we
> are using in our distribution but it likely has that fix as well. But
> Slab part was merely something I've spotted in the referenced commit and
> not something that is causing the main problem. That is indeed ths
> swap backed page cache. More on that below.
> 
> > On Thu, 1 Apr 2021 at 18:30, Michal Hocko <[email protected]>
> > wrote:
> > 
> > > From a semantic point of view it is quite hard to understand what the
> > > value is actually trying to represent. I assume that reducing the page
> > >
> > This is actually the main problem. What does used mean?
> 
> Well, kernel doesn't expose counter like that. It does provide and
> estimation of the usable memory which is likely what you want to
> be expressed in its complement.
> 
> If there is no clear semantic behind the value then it likely shouldn't
> try to be clever. I do understand that many people are confused by their
> system consuming a lot of memory for caching. This is not something new
> and I am afraid that hiding that fact from them is just going to
> continue confusion elsewhere. As you can see by this bug report.
>  
> [...]
> > The calculation is now (see
> > https://gitlab.com/procps-ng/procps/-/blob/master/proc/sysinfo.c#L789 )
> > 
> > Memory Used = Memory Total - Memory Free - ( Cache + Slab Reclaimable ) -
> > Main Buffers
> > 
> > The first two parts are easy enough, total - free = used
> > 
> > So the question is why are we, in effect, ignoring cache, reclaimable slab
> > and buffers?
> > 
> > Cache includes tmpfs which is file backed not memory
> 
> No, this is not quite right. tmpfs/shmem is a general concept for
> shared memory with swap as a backing storage. That can be MAP_SHARED | MAP_ANON
> mapping, tmpfs mounted filesystem, memfd, System V shm  and many others.
> It is accounted as Cached because internally it is accounted as a Page
> cache, mostly for historical reasons. Something many of us in the MM
> area consider a mistake but something we cannot really change because
> that would be suprising for users who already know that is the case.
> There have been several attempts to change.
> 
> > Reclaimable slab, well it's reclaimable so it's not used.
> > Buffers, I'm not 100% sure why this is removed now but it made sense later.
> > 
> > If you follow the mail archive at
> > https://www.freelists.org/post/procps/OmegaPhilxxxxxxxxxxxxx-Bug799716-free-considers-cached-to-include-SUnreclaim
> > you can see how we got to these figures.
> > 
> > If you really want to go down the Memory Used rabbit hole, try
> > https://github.com/brndnmtthws/conky/issues/130
> 
> Well, the problem is that there are users relying on free(1) and they
> complain that some monitoring alerts are invalid because output of free
> shows the system in a healthy state while it is not because MemAvailable
> or other more precise metrics show a serious memory overutilization. We
> can sure go and tell those users that Used: of the free(1) output is
> incorrect but I believe it would be better to fix this in free instead.
> One potential way to go ahead would be Total - MemAvailable. I would
> still argue that Total - Free would be likely the only one which is
> always correct but using Memavailable can give at least arguably
> coherent view.
> -- 
> Michal Hocko
> SUSE Labs

-- 
Michal Hocko
SUSE Labs