Re: free: regression due to a different calculation of Used memory

tl;dr:
Logically and semantically, I understand "free" as anything that is not
"used" anyhow (even cache, reclaimable etc.). So No. 4 is the most correct
from my POV.
HOWEVER, it is ok to consider "free" memory as anything we can get/reclaim,
so if No. 3 (Used = Total - Available) is the general candidate, I'd vote
for it as "better than the Original".


The test does not need to be tricky.
The other day, I used 'mktemp' to create a file, wrote 50 megs every 2
seconds with 'dd' into it and then used 'watch free' to see what actually
rises rapidly.
Immediately after seeing that Cached just munches memory and 'Used' still
looks like "nothing happening here", I imagined an orchestra of (diskless)
VM boxes
in a rack which have journald's storage set to default in /run rather than
/var. It is not rare to see a journal many GiBs huge. The machines start
OOMing,
the admin runs 'free' and sees that "Used is not that high, so what's
happening?" Hence, I'd definitely choose anything else than the Original
calculation.

Anyway, thank you and everyone involved for looking into that!
Jan

On Fri, Jun 17, 2022 at 11:15 AM Craig Small <[email protected]> wrote:

> OK to see what's going on I've done some testing.
>
> The first thing I'd like to say is that testing this is tricky, the first
> set gave some terrible results for the proposed change because the free and
> cached value changed dramatically (~ 40M) during the test.
> It seemed that cached was low and increased once you start doing things
> and remains high.
>
> On Wed, 1 Jun 2022 at 19:50, Jan Rybar <[email protected]> wrote:
>
>> > You won't see that in meminfo.c or vmstat.c, but Shmem page belong to
>> the page cache and as such are accounted in cached.
>> > If you write a big file in tmpfs, you'll see that both shmem and cache
>> entry increase
>>
> I've done that very test.
> I added a 5MB file to a tmpfs directory twice, taking memory samples each
> time.
>
> Free /proc/meminfo
> State Total Used Free Shared Buff/Cache Available Free Cached SReclaimable
> Buffers Used: Original Used: No cache Used: Availabe Used: Simple
> No file 976.1 83.5 744.8 4.5 147.9 753.7 744.6 114 19.8 14.1 83.6 197.6
> 222.4 231.5
> First 5M file 976.1 83.4 739.9 9.5 152.9 748.8 739.9 119 19.8 14.1 83.3
> 202.3 227.3 236.2
> Delta from init 0 -0.1 -4.9 5 5 -4.9 -4.7 5 0 0 -0.3 4.7 4.9 4.7
> Second 5M file 976.1 83.5 734.7 14.5 157.9 743.7 734.5 124 19.8 14.2 83.6
> 207.6 232.4 241.6
> Delta from init 0 0 -10.1 10 10 -10 -10.1 10 0 0.1 0 10 10 10.1
> Remove files 976.1 83.3 744.8 4.5 148 753.8 744.8 114 19.8 14.2 83.3 197.3
> 222.3 231.3
> Delta from init 0 -0.2 0 0 0.1 0.1 0.2 0 0 0.1 -0.3 -0.3 -0.1 -0.2
>
> Adding the first file, we can use the current method for Used shows a
> *reduction* of 300k for used and no change for the second. Removing
> Cached or a simple Total-Free shows a reduction of 4.7 MB. Total-Available
> is 4.9
> With the second 5MB file, the current method shows no change but the other
> three methods we can see Used increasing by 10 or 10.1 MB.
>
> I think we can safely say putting Cached into the Used calculation is
> wrong. There is definitely 5 or 10MB of memory that is being used and the
> current free won't show this. It's probably a failure of Cached giving the
> "right" answer rather than free's fault, but we are beholden to what the
> kernel gives us. Cached used to mean cache for files paged/swapped to disk
> but it doesn't anymore.
>
> I also explored doing some delta of (Cached-Shmem) but for some systems
> they both increased by 5MB which was good, but for others Cached increased
> by 6MB and Shmem by 5MB.
>
> We have several options here:
>
>    1. Original: Used = Total - Free - Cached - SReclaimable - Buffers
>    2. No Cache: Used = Total - Free - SReclaimable - Buffers
>    3. Available: Used = Total - Available
>    4. Simple: Used = Total - Free
>
> I think we can all agree that at least for files going into a tmpfs the
> original formula is wrong. We definitely are using an extra 5 or 10MB here
> but free won't show it.
> Methods 2 and 4 can be accused of missing something or having the wrong
> definition of what Used should be ("you forgot to include X value")
>
> My preferred option is to say that Used memory is unavailable memory, in
> other words option 3 "Used =Total - Available".  The kernel team is
> responsible for the Available value, we are just reporting a simple
> difference between the two.  Remember that the used calculation predates
> the creation of the Available value which is why we have had various
> calculations for what Used is. Option 3 takes this away from procps which
> is my preferred option.
>
> If we can get a consensus we can put this to bed, so comments are welcome.
>
>  - Craig
>
>