Re: lvm2 deadlock

Hi,

On 2024/06/04 12:48, Roger Heflin wrote:

> Use the *_bytes values.  If they are non-zero then they are used and
> that allows setting even below 1% (quite large on anything with a lot
> of ram).
>
> I have been using this for quite a while:
> vm.dirty_background_bytes = 3000000
> vm.dirty_bytes = 5000000


crowsnest [13:32:48] ~ # sysctl vm.dirty_background_bytes=3000000
vm.dirty_background_bytes = 3000000
crowsnest [13:32:59] ~ # sysctl vm.dirty_bytes=500000000
vm.dirty_bytes = 500000000

And persisted via /etc/sysctl.conf

Thank you.  Must be noted this host doesn't do much else other than disk 
IO, so I'm hoping the 500MB value will be OK, this is just so IO won't 
block CPU heavy-at-the-time tasks.

The purpose of 256GB RAM was so that we could have ~250GB worth of disk 
cache (obviously we don't want all of that to be dirty, OS and "used" 
used to be below 4GB, now generally around 8-12GB, currently it's in 
"quiet" time, so a bit lower, just busy running some background 
compression).  As per iostat:

avg-cpu:  %user   %nice %system %iowait %steal   %idle
            7.73   18.43   18.96   37.86    0.00   17.01

Device             tps    MB_read/s    MB_wrtn/s    MB_dscd/s MB_read    
MB_wrtn    MB_dscd
md2             392.13        10.00         5.11         0.00 4244888    
2167644          0
md3            2270.12        43.88        56.82         0.00 18626309   
24120982          0
md4            1406.06        30.47        16.83         0.00 
12934654    7143330          0

That's total 35805851 MB (34.1B) read and 33431956 MB (31.9TB) written 
in just under 5 days.

What I am noticing immediately is that the "free" value as per "free -m" 
is definitely much higher, which to me is indicative that we're not 
caching as aggressively as can be done.  Will monitor this for the time 
being:

crowsnest [13:50:09] ~ # free -m
                total        used        free      shared buff/cache   
available
Mem:          257661        6911      105313           7 145436      248246
Swap:              0           0           0

The Total DISK WRITE and Current DISK Write values in in iotop seems to 
have a tighter correlation now (no longer seeing constant Total DISK 
WRITE with spikes in current, seems to be more even now).

Kind regards,
Jaco