Re: Verify consistently fails after rebooting secondary node

Some updates on this issue=0A=
=0A=
To recap 2 nodes, primary and secondary, after an initial sync the secondar=
y is rebooted=0A=
and a verify always detects out of sync sectors. =0A=
=0A=
It seems to occur on 9.2.4 version of the driver as well as well as all of =
the kernel =0A=
versions we have been using, so appears to be unrelated to any changes we h=
ave =0A=
made in system startup. =0A=
=0A=
We are working around this problem by invalidating the disk and doing a ful=
l =0A=
resync after a reboot, this is fairly onerous for large disks. =0A=
=0A=
We have not been able to verify corruption when no connection is made back =
to =0A=
another node after the reboot, but this is harder to validate as system may=
 boot with corruption=0A=
=0A=
What expectations should we have for integrity on a shutdown? Reboot? Power=
 loss? =0A=
=0A=
Where could we look closer at trying to understand this issue? =0A=
=0A=
=0A=
=0A=
________________________________________=0A=
From:=A0Tim Westbrook <[email protected]>=0A=
Sent:=A0Tuesday, December 24, 2024 11:01 AM=0A=
To:[email protected] <[email protected]>=0A=
Subject:=A0Verify consistently fails after rebooting secondary node=0A=
=A0=0A=
Hello=0A=
=0A=
We are observing the following issue with resync after reboot.=0A=
=0A=
After rebooting a secondary node (in a 2 or 3 node cluster), the=0A=
secondary successfully connects to primary and reports UpToDate, but=0A=
when a verify is launched on the secondary node that was rebooted, it repor=
ts=0A=
out of sync blocks.=0A=
=0A=
If an "invalidate --reset-bitmap=3Dno" is issued on the resource on the sec=
ondary=0A=
node, the invalidate sync happens quickly and the next verify succeeds with=
=0A=
no out of sync blocks.=0A=
=0A=
This was initially detected when we promoted a backup node and it came up w=
ith=0A=
disk corruption. We traced this to the reboot occurring before the promotio=
n.=0A=
=0A=
=A0Versions=0A=
=0A=
The logs attached are using the 9.2.12 version of the driver on the 5.15.17=
3 kernel,=0A=
but we have also observed this issue on the 9.2.4 driver with the 5.15.166 =
kernel=0A=
=0A=
We have not seen the problem on 5.15.151 and version 9.2.4 of the driver.=
=0A=
=0A=
=0A=
=A0Attachments=0A=
=0A=
initsyncandverify_noreboot.txt - drbd logs from system prior to reboot , in=
cludes=0A=
verify before reboot=0A=
=0A=
verify_after_invalidate_no_reset.txt - drbd logs after reboot show initial =
failed=0A=
verify then, invalidate, then successful verify=0A=
=0A=
dynamic.res - drbd conf file - note use of separate metadata disk - we also=
=0A=
=0A=
=0A=
=A0Secondary Bring Up=0A=
=0A=
Secondary nodes enable drbd "persist" resource as follows=0A=
=A0=0A=
=A0"""=0A=
=A0 da up all || true=0A=
=A0 da secondary persist || true=0A=
=A0 da disconnect persist || true=0A=
=A0 da -- --discard-my-data connect persist || true=0A=
"""=