mkfs on luks USB flash drive blocks forever with "reset SuperSpeed USB device" (kernel 6.7.9)

Daniel Pouzzner <[email protected]> Fri, 26 Apr 2024 14:35:15 -0500
Newsgroups dev.linux.lists.cryptsetup
Message-ID <[email protected]>
I originally opened a ticket for this on the Gentoo bug board
(https://bugs.gentoo.org/930464) but that isn't really the right venue for it.  

I'm finding that on kernel 6.7.9, with a freshly created and opened luks
partition, mkfs goes into D wait, with "reset SuperSpeed USB device number #
using xhci_hcd" repeating every 30 seconds on each device.

The behavior occurs with either mkfs.btrfs (with or without btrfs RAID1) or
mkfs.ext4 (no RAID), and with either luks1 or luks2.

This is new behavior.  I've used the same procedure, same hardware, and
same/similar media, dozens of times before (most recently on kernel 6.4.3)
without seeing this.

The hardware itself is a pair of Kingston FCR-HS4 flash readers, and the
behavior occurs on both.  The media are 512GB microSDXC.  I've tried it with
fresh new Lexar and Sandisk media, both of them SKUs that have worked before. 
Same bad result.

If I run mkfs.btrfs directly on the GPT partition, that succeeds.

If I luksOpen and mount previously formatted and mkfs.btrfs'd media, and access
it extensively for an incremental backup run, that succeeds.

So what is specifically failing is mkfs on a luks partition.

The kernel log provides no real information, just entries like this:

[Mon Apr 22 18:08:59 2024] usb 2-2.3: reset SuperSpeed USB device number 12
using xhci_hcd
[Mon Apr 22 18:09:04 2024] usb 4-3: reset SuperSpeed USB device number 10 using
xhci_hcd

When the problem first occured, I discovered I was able to free up the stuck
mkfs process by pulling the cords on the Kingston readers, and then free up the
luks contexts with "cryptsetup close".

Using an older system running kernel 5.4, but with the same flash readers and
media, mkfs.btrfs succeeded immediately.  When I returned the readers to the
original system, the new filesystem mounted without delay or errors.

A quick look through the applicable Gentoo kernel patchsets didn't turn up
anything obvious, and the problem wasn't there in January on kernel 6.4.3. 
Also, a diff of my configs for the 6.4.3 and 6.7.9 kernels didn't turn up
anything obviously related.

Another data point: I ran a btrfs send/receive pipeline, writing to the
filesystem that I mkfs'd from the old kernel 5.4 installation.  It completed
successfully, 300GB transferred, with no kernel messages, and certainly no
resets.  A post-transfer scrub found and repaired a few sector errors on one of
the two devices, which is just par for the course on microSDXC.

So the problem seems to be very specific to mkfs over dm-crypt on this kernel. 
I'm not currently able to bisect the problem, so I don't yet know if it's still
present in kernels 6.8 and 6.9-RC.