Re: Filesystem corruption with LVM's pvmove onto an encrypted volume with LUKS2 and a sector size of 4096

Hi Ingo,

On 2/21/19 1:03 PM, Ingo Franzki wrote:
> Hi,
> 
> we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume.
> After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted.
> 
> It seems to be related to a sector size of 4096 used with LUKS2. Once I use the default sector size (512) then the problem does not happen.
> It happens with LUKS2 and even plain mode, as soon as a sector size of 4096 is used. LUKS1 and the default sector size does not show the problem.
> 
> Not sure if this is a problem in dm-crypt or LVM, or a combination of both.

tldr; dm-crypt with encryption sector size set to 4096 may increase 
device limits to 4096 bytes (minimal i/o size and i/o alignment). It can 
corrupt filesystem originally created on top of 512B device and moved 
over to dm-crypt with increased encryption sector size (which you 
probably did with pvmove).

-----

You're right it's due to encryption sector size, but this is not a bug 
in dm-crypt nor in pvmove. Let me explain a bit:

dm-crypt advertise itself as a block device with physical sector size 
*at least* equal to encryption sector size. Traditionally it's been only 
512B. So classical dm-crypt mapped over device with phys. sector size = 
512B has no impact. If you mapped dm-crypt over block device with native 
physical sector size = 4096 you got dm-crypt exposing same limits as 
underlying block device. Again no problem. Just internally dm-crypt 
performed encryption on 512B blocks, but it had no impact on exposed limits.

But things get a bit different with encryption sector size > 512B.

If you map dm-crypt with encryption sector size set to 4096B over block 
device with physical sector size = 512B, dm-crypt must increase device 
limits to 4096. Because when it does encryption it must be aligned to 
4096 bytes (and same wrt minimal i/o size).

So to conclude it:

If you create dm-crypt with 4K enc. sector over *native* 4K block device 
it's ok and it must work everytime.

If you create dm-crypt with 4K enc. sector over 4K block device with 
512B emulation, you're ok to create new fs (only new!) on top of it.

Moving existing filesystem is always a bit tricky. You need to know with 
what limits the original fs was created. For example on those 4096/512e 
drives (emulating 512B), fs may have aligned its structures either to 
4096B or 512B during mkfs. For example ext4 does the decision based on 
device size (hope I recall it correctly).

Just a remark: with LVM2 you may get to similar problems using only 
lvextend command where you (even accidentally) mix PVs on 512B drives 
with 4K native drives (latest example: 
https://bugzilla.redhat.com/show_bug.cgi?id=1669751)

(We desperately need to start publishing blog posts related to these 
issues...sigh)

Kind regards
O.
_______________________________________________
dm-crypt mailing list
[email protected]
https://www.saout.de/mailman/listinfo/dm-crypt