Re: lvm2 deadlock

Hi,

Please refer below.

On 2024/06/03 21:25, Zdenek Kabelac wrote:
> Dne 03. 06. 24 v 14:56 Jaco Kroon napsal(a):
>> Hi,
>>
>> Thanks for the insight.  Please refer below.
>>
>> On 2024/05/31 14:34, Zdenek Kabelac wrote:
>>> Dne 30. 05. 24 v 12:21 Jaco Kroon napsal(a):
>>>> Hi,
>>>>
>>> I'm kind of missing here to see your 'deadlock' scenario from this 
>>> description.
>> Well, stuff blocks, until the cookie is released by using the dmset 
>> udevcomplete command, so wrong wording perhaps?
>>>
>>> Lvm2 takes the VG lock - creates LV - waits for udev till it's 
>>> finished with its job and confirms all the udev work with dmsetup 
>>> udevcomplete.
>>
>> So what I understand from this is that udevcomplete ends up never 
>> executing?  Is there some way of confirming this?
>
> udevcomplete needs  someone to create 'semaphore' for completion in 
> the first place.


I'm not familiar with the LVM internals and the flows of different 
processes, even though you can probably safely consider me a "semi power 
user".  I do have compsci background, so do understand most of the 
principles of locking etc, but how they are applied in the LVM 
environment I'm clueless.

>
>
>>>
>>> It's also unclear which OS are you using - Debian, Fedora, ???
>>
>> Gentoo.
>>
>>> Version of your packages ?
>>
>> I thought I did provide this:
>>
>> Kernel version was 6.4.12 when this hapened, is now 6.9.3.
>>
>> crowsnest [12:19:47] /run/lvm # udevadm --version
>> 254
>>
>> aka systemd-utils-254.10
>>
>> lvm2-2.03.22
>
> Since this is most likely your personal build - please provide full 
> output of
> 'lvm version'  command.


crowsnest [09:46:04] ~ # lvm version
   LVM version:     2.03.22(2) (2023-08-02)
   Library version: 1.02.196 (2023-08-02)
   Driver version:  4.48.0
   Configuration:   ./configure --prefix=/usr 
--build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu 
--mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share 
--sysconfdir=/etc --localstatedir=/var/lib --datarootdir=/usr/share 
--disable-dependency-tracking --disable-silent-rules 
--docdir=/usr/share/doc/lvm2-2.03.22-r5 
--htmldir=/usr/share/doc/lvm2-2.03.22-r5/html --enable-dmfilemapd 
--enable-dmeventd --enable-cmdlib --enable-fsadm --enable-lvmpolld 
--with-mirrors=internal --with-snapshots=internal --with-thin=internal 
--with-cache=internal --with-thin-check=/usr/sbin/thin_check 
--with-cache-check=/usr/sbin/cache_check 
--with-thin-dump=/usr/sbin/thin_dump 
--with-cache-dump=/usr/sbin/cache_dump 
--with-thin-repair=/usr/sbin/thin_repair 
--with-cache-repair=/usr/sbin/cache_repair 
--with-thin-restore=/usr/sbin/thin_restore 
--with-cache-restore=/usr/sbin/cache_restore --with-symvers=gnu 
--enable-readline --disable-selinux --enable-pkgconfig 
--with-confdir=/etc --exec-prefix= --sbindir=/sbin 
--with-staticdir=/sbin --libdir=/lib64 --with-usrlibdir=/usr/lib64 
--with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm 
--with-default-locking-dir=/run/lock/lvm --with-default-pid-dir=/run 
--enable-udev_rules --enable-udev_sync --with-udevdir=/lib/udev/rules.d 
--disable-lvmlockd-sanlock --disable-notify-dbus --disable-app-machineid 
--disable-systemd-journal --without-systemd-run --disable-valgrind-pool 
--with-systemdsystemunitdir=/lib/systemd/system CLDFLAGS=-Wl,-O1 
-Wl,--as-needed


>
> For the 'udev' synchronization, there needs to be '--enable-udev_sync' 
> configure option. So let's check which configure/build option were 
> used here.
> And also preferably upstream udev rules.


--enable-udev_sync all there.

To the best of my knowledge the udev rules are stock, certainly neither 
myself nor any of my colleagues modified them.  They would generally 
defer to me, and I won't touch that unless I understand the 
implications, which in this case I just don't.

>
>>
>> Thanks for the feedback, what you say makes perfect sense, and the 
>> implication is that there are only a few options:
>>
>> 1.  Something is resulting in the udev trigger to take longer than 
>> three minutes, and the dmsetup udevcomplete never being executed.
>
> systemd simply kills udev worker if takes too long.
>
> However on properly running system, it would be very very unusual to 
> hit these timeouts  - you would need to work with thousands of 
> devices....


32 physical NL-SAS drives, combined into 3 RAID6 arrays using mdadm.

These three md devices serve as PVs for LVM, single VG.

73 LVs, just over half of which are mounted.  Most of those are thin 
volumes inside:

crowsnest [09:54:04] ~ # lvdisplay /dev/lvm/thin_pool
   --- Logical volume ---
   LV Name                thin_pool
   VG Name                lvm
   LV UUID                twLSE1-3ckG-WRSO-5eHc-G3fY-YS2v-as4ABC
   LV Write Access        read/write (activated read only)
   LV Creation host, time crowsnest, 2020-02-19 12:26:00 +0200
   LV Pool metadata       thin_pool_tmeta
   LV Pool data           thin_pool_tdata
   LV Status              available
   # open                 0
   LV Size                125.00 TiB
   Allocated pool data    73.57%
   Allocated metadata     9.05%
   Current LE             32768000
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     1024
   Block device           253:11

The rest are snapshots of the LVs that are mounted so that we have a 
roll-back destination in case of filesystem corruption (these snaps are 
made in multiple steps, first a snap of the origin is made, this is then 
fsck'ed, if that's successful it's fstrim'ed before being renamed into 
the final "save" location - any previously saved copy is first lvremove'd).

I'd describe that as a few tens of devices, not a few hundred and 
certainly not thousands of devices.


>
>>
>> This could potentially be due to extremely heavy disk IO, or LVM 
>> itself freezing IO.
>
> well reducing the percentage of '/proc/sys/vm/dirty_ration' may 
> possibly help
> when your disk system is too slow and you create a very lengthy 'sync' 
> io queues...


crowsnest [09:52:21] /proc/sys/vm # cat dirty_ratio
20

Happy to lower that even more if it would help.

Internet (Redhat) states:

Starts active writeback of dirty data at this percentage of total memory 
for the generator of dirty data, via pdflush. The default value is |40|.

I'm assuming the default is 20 though, not 40, since I can't find that 
I've reconfigured this value.

Should probably remain higher than dirty_background_ratio (which is 
currently 10), dirty_background_bytes is 0.


>
>> I don't see the default value for udev_log from the config. 
>> Explicitly set to debug now, but still not seeing anything logged to 
>> syslog. Running with udevd --debug, which logs to a ramdisk on /run.  
>> Hopefully (if/when this happens again) that may shed some light.  
>> There is 256GB of RAM available, so as long as the log doesn't grow 
>> too quickly should be fine.
>
> A lot of RAM may possibly create a huge amount of dirty pages...


May I safely interpret this as "lower the dirty_ratio even further"?

Given a value of 10 and 20 I'm assuming that pdflush will start flushing 
out in the background when >~26GB of in-memory data is dirty, or if the 
data has been dirty for more than 5 seconds (dirty_writeback_centisecs = 
500).

Don't mind lowering the dirty_background ratio as low as 1 even? But 
won't the primary dirty_ratio start blocking processes from writing if 
 >40% of the caches/buffers are considered dirty?

Kind regards,
Jaco