Re: cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

On Wed, Nov 25, 2020 at 7:34 PM Milan Broz <[email protected]> wrote:

> On 25/11/2020 19:01, Michael Kjörling wrote:
> > On 25 Nov 2020 18:30 +0100, from [email protected] (Martin Jørgensen):
> >> I tried to look in version history but am not into the details of the
> >> difference in crypt-setup versions. Can anyone please tell why I got/get
> >> this error using 2.1.0 and not with 2.3.4?
> >>
> >> "mount: wrong fs type, bad option, bad superblock on /dev/mapper/....,
> >> missing codepage or helper program, or other error


Michael, please reply (reply all) to me also, it seems I only received your
comments/message via Milan's reply (otherwise I guess I would have seen it
in the digest, I've turned digest off for now to ensure I receive all
replies in the coming days)...

>
>
> _If_ you're able to unlock the container using both versions, _but_
> > with one you are able to mount the file system within the container
> > and with the other you are unable to do so, then I would suggest
>

Right, sorry. This is correctly understood and this is what happens. It's
the mount-command that causes problems, not actually cryptsetup itself,
good to have this clarified...


> > comparing the output of `file -s /dev/mapper/whatever` for the two
> > cases after unlocking. You could also try `cryptsetup luksDump
>

I think you meant "file -Ls".

For both machines I get: /dev/mapper/TESTLUKS: Linux rev 1.0 ext4
filesystem data, UUID=d808dde4-2e57-48ed-b9e7-5c21948fabe6 (needs journal
recovery) (extents) (64bit) (large files) (huge files)

But I don't understand "needs journal recovery", could this have anything
to do with it?


> > --dump-master-key` for each version of cryptsetup and comparing the
> > output, but be sure that you understand the consequences of the master
> > key being potentially compromised before you do so.
>
> Yes, but the key is validated through LUKS key digest, if the validation
> fails,
> it is never used for the mapping. I do not believe it is a wrong key
> problem.
>
> Right, the key is ok - I can unlock, but can't mount, thanks Milan.


> Also compare "dmsetup table <device>" on both systems for LUKS active
> device.
> It should produce exactly the same parameters (mainly offset, size, crypt
> options).
>

Looks exactly similar - in both cases: 0 6291423232 crypt aes-xts-plain64
:64:logon:cryptsetup:265a5d99-4563-49e6-a5a5-7d7cab4f137e-d0 0 7:1 32768

All this looks as a kernel issue unrelated to cryptsetup (note that you
> have different
> kernels if you have different distros and the data decryption and fs
> handling is the kernel).
>

Right, that makes sense, I agree with you, sounds very much like a kernel
thing with "mount"... Proxmox/debian runs 5.4.65-1. Arch Linux runs
5.9.9... Oh yeah, and by the way, I actually did upgrade the proxmox/debian
to another/new kernel just before copying the backup to another disk. I can
see in my /boot-dir, that I used to run 5.4.55-1 and later 5.4.60-1, just
before these issue appeared, maybe there's a problem with the 5.4.65-1
kernel, which wasn't there in 5.4.60-1...

There's actually some talk about "journal superblock" here
https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.4.61 (and in
changelog for 5.4.62, but I do not understand that) - and my error message
was something about "mount: wrong fs type, bad option, bad superblock on
/dev/mapper/...., missing codepage or helper program... hmm.


> > I'm sure someone will correct me if I'm wrong about this, but I am
> > _fairly_ certain that there is nothing in the luksOpen/luksClose code
> > that will change anything in the LUKS header, so something else must
> > be going on in your case.
>
> For LUKS1 is it always true, LUKS2 have two metadata areas, there is
> autocorrection
> in place if one area checksum is wrong (happens on the header load) but
> this is IMO
> unrelated to this case.
>

Damn, it's really creepy... I actually didn't check/test mounting on the
Debian/proxmox-machine since after I spend a lot of time seeing that it
worked on Arch Linux (and at that time not on the
Debian/proxmox-machine)... But now it works on both machines (arch+debian)?
Weird... It was 2 days ago I had these problems... Are you sure this
autocorrection is unrelated? Let me refresh what I did, because luckily I
still have the complete logs:

So, when I tried to repair the LUKS-encrypted container, google said to me
that I should try using another superblock. Here's what happened on the
debian/proxmox machine:

root@proxmox:/mnt# mke2fs -n /dev/mapper/hugeZFSdata
mke2fs 1.44.5 (15-Dec-2018)
Found a atari partition table in /dev/mapper/hugeZFSdata
Proceed anyway? (y,N) y
Creating filesystem with 786430976 4k blocks and 196608000 inodes
Filesystem UUID: 6af28573-a3b9-43a1-9562-f9c990a87598
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544

And this didn't seem to work at all (I tried MANY TIMES + rebooted each
time):
-----
# e2fsck  -b 32768 -v  /dev/mapper/hugeZFSdata
e2fsck 1.44.5 (15-Dec-2018)
e2fsck: Bad magic number in super-block while trying to open
/dev/mapper/hugeZFSdata
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

Found a atari partition table in /dev/mapper/hugeZFSdata
----

I remember that although I tried ALL the blocks I always got the same error
message about "Found a atari partition table in /dev/mapper/hugeZFSdata",
which made me think that the command failed. One post found via google said
I should try and ignore the message, just run it and reboot (to flush
something, I guess). I did reboot after around 1/2 of the times, but there
are 21 blocks to test and I didn't have patience to reboot after all 21
attempts... Maybe I was just lucky?

But I would *never* have tried the "work-around" I eventually - as a final
step - came up with, which is to SMB/CIFS/Samba-mount the encrypted
file/container to the Arch machine, where I mounted the folder and ran
"cryptsetup luksOpen ..." + mounted the dev/mapper/....-device
successfully...

Hmm... I'm still confused - I hope we find the explanation, so I know what
to do if it happens again. If it happens again, however, I'll run the
commands above and compare them again + hopefully write here, in case
there's a bug somewhere. But I think you're right, that it likely is some
kernel stuff with the mount-command. The "autocorrection" for the two
LUKS2-metadata areas, does that happen after a successful "mount"-command
or as part of the "cryptsetup luksOpen ..."-command?

It's weird that it seems to be working now, I think there's some
autocorrection going on (I did remember updating a few packages on the
debian-machine, but not much, chances are those upgrades, not even kernel
updates, didn't change this)... hmm.


> (If I understand the mail correctly, the LUKS device can be still
> activated but
> the filesystem is corrupted now? If not, we need --debug output from
> luksOpen command
> to check what is wrong.)
>
>  I'm very interested in hearing what you make out of the "The superblock
could not be read or does not describe a valid ext2/ext3/ext4 filesystem"
and the "Found a atari partition table in /dev/mapper/hugeZFSdata"-error
message I posted above, while I had the problem... I'm very afraid if the
problem happens again, damn, it was/is scary, got any ideas about what
could've happened? I'm really confused now :-)

Br,
M.

_______________________________________________
dm-crypt mailing list
[email protected]
https://www.saout.de/mailman/listinfo/dm-crypt