Re: Help diagnosing lvm ioctl log spam

Erwin van Londen <[email protected]> Thu, 10 Oct 2024 00:21:44 +0000

Newsgroups	gmane.linux.lvm.general
Message-ID	<[email protected]>

Could it not be that, given the fact this is a volume carved out of a 
thin pool, the underlying free space threshold is near or at capacity 
and the the device mapper is more busy doing garbage collection than 
actual application IO's. I've seen many nightmares with thin provisioned 
volumes if this is not managed properly.

Have you used the "fstrim" command or is the filesystem mounted with the 
"-o discard" option?

Check some setting with, for example, "lvs -o 
lv_full_name,lv_health_status,lv_when_full". If it is a space issue make 
sure that you have enough free space in your volume groups and configure 
the "thin_pool_auto_extend_threshold" and 
"thin_pool_auto_extend_percentage". Also turn on "monitoring" if that is 
not enabled by default.

Another thing that could be an issue is that the filesystem has the "-o 
discard" mount option set, device mapper wants to propagate this to the 
underlying hardware (or in your case a hypervisor which may also send 
this through to the underlying storage array), and that is not supported 
on that hardware. The "Inappropriate ioctl for device" message hints in 
that direction. Has there been movements of volumes to other provisioned 
disks or changes on the hypervisor?

Cheers
Erwin

On 10/10/24 00:52, David Teigland wrote:
> On Wed, Oct 09, 2024 at 06:28:26AM -0300, Fabricio Winter wrote:
>> Hello people, we have been experiencing an issue with lvm2-thin on
>> _some_ of our production servers where out of nowhere
>> lvm2/device-mapper starts spamming error logs and I can't really seem
>> to trace down the root cause.
>>
>> This is what the logs look like;
>> Oct  9 06:25:02 U5bW8JT7 lvm[8020]: device-mapper: waitevent ioctl on
>> LVM-CP5Gw8QrWLqwhBcJL87R1mc9Q9KTBtQQmOowipTAFuM7hqzHz6pRVvUaNO9FGzeq-tpool
>> failed: Inappropriate ioctl for device
>> Oct  9 06:25:02 U5bW8JT7 lvm[8020]: waitevent: dm_task_run failed:
>> Inappropriate ioctl for device
> It appears related to dmeventd monitoring the thin pools, and the kernel
> returning ENOTTY when dmeventd does the DM_DEV_WAIT ioctl.  Maybe there's
> a fast retry loop in dmeventd on that error case rather than quitting.
> I wonder if there's a way you could kill dmeventd when this happens.
>
> Dave
>
>

OpenPGP_signature.asc (application/pgp-signature, 677 B)

erwin.vcf (text/vcard, 172 B)

BEGIN:VCARD
VERSION:4.0
N:van Londen;Erwin;;;
FN:Erwin van Londen
EMAIL;PREF=1;TYPE=work:[email protected]
URL;TYPE=work:https://erwinvanlonden.net
END:VCARD

OpenPGP_0x985B90929D90E282.asc (application/pgp-keys, 2.5 KB)