Re: Schrödinger's hash

On 6/22/26 09:54, Charles Curley wrote:
> On Mon, 22 Jun 2026 00:32:19 -0700 David Christensen wrote:
>> Please run and post the following commands with both the previous
>> kernel and the backports kernel:
>>
>> $ cat /etc/debian_version
>>
>> $ uname -a
> 
> Backport
> 
> root@hawk:~# cat /etc/debian_version
> 13.5
> root@hawk:~# cat /etc/os-release
> PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
> NAME="Debian GNU/Linux"
> VERSION_ID="13"
> VERSION="13 (trixie)"
> VERSION_CODENAME=trixie
> DEBIAN_VERSION_FULL=13.5
> ID=debian
> HOME_URL="https://www.debian.org/"
> SUPPORT_URL="https://www.debian.org/support"
> BUG_REPORT_URL="https://bugs.debian.org/"
> root@hawk:~# uname -s
> Linux
> root@hawk:~# uname -a
> Linux hawk 7.0.10+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 7.0.10-1~bpo13+1 (2026-05-28) x86_64 GNU/Linux
> root@hawk:~#
> 
> Prior kernel is the same except the kernel is
> 
> linux-image-6.12.90+deb13.1-amd64	6.12.90-2		amd64


Good information.


> I think that this problem first showed up late April or early May. If
> that is correct, the following extracts from my apt logs might help.
> These were copied and pasted unwrapped, so unless they were mangled in
> transit, long lines should be preserved.
> 
> ...
> 
> Apparently, twice in April I had tried the backports kernel(s), and
> found them unsatisfactory. So possibly the fix came in between
> linux-image-6.19.11+deb13-amd64 and the current backports kernel,
> linux-image-7.0.10+deb13-amd64. 


AIUI if someone can come up with a shell script or program whose exit 
value reliably indicates the presence or absence of a bug, Git can do a 
binary search over a range of commits and locate the commit where the 
bug originated.


> Quite possibly a major version number
> might even inadvertently fix something as subtle as this.


Agreed.


>> I will assume your script spawns a separate, isolated process for
>> each checksum file.
> 
> Correct. It does a find on *.sha256sums, *.sha512sums, and several other
> suffixes. It then sliced and dices to figure out the appropriate
> program to call, sha256sum and sha512sum, respectively.
> 
> The files are created using paths relative to the current directory, so
> when the script runs, it will pushd to that directory
> 
> The key line is
> 
>      nice "${prog}" "${opts}" -c "${file}" &
> 
> Where $prog is the result of the slicing and dicing, opts='--quiet',
> and $file the file to be scanned.


I use a similar workflow for image files and also wrote a script to 
generate and verify checksum files.


> I recently added the option to limit the number of background tasks to
> the number of processors (nproc --all). That reduces but does not
> eliminate the number of errors.


That is another clue that there is a race condition related to parallel I/O.


>> If you have ruled out the power supply, memory, and disks, another
>> possibility could be a race condition in the kernel and/or I/O stack
>> that is triggered when multiple processes access storage in parallel.
> 
> I wouldn't say those are all ruled out. But the fact that the
> backport kernel is a major version number change, and that it appears
> to have solved the problem is highly suggestive. With that caveat, I
> concur. That is definitely something for the kernel folks to look at.


Agreed.


>> Are the checksum errors repeatable on another computer with a similar
>> storage architecture and the previous kernel?  If so, do they
>> disappear with the backports kernel?
> 
> I have a much more recent and much faster computer, peregrine, with nvme
> storage and 12 cores. Hawk has eight cores, and spinning rust. Hawk is
> where the problem has shown up. Peregrine does not show the problem.
> 
> For 18 gig of data, hawk: 2m18.318s, peregrine 0m24.233s.


That clue makes me think the race condition is the SCSI stack.


It looks like there is a newer kernel for Trixie.  It is best to file 
bug reports against current packages.  Can you test it?

https://packages.debian.org/stable/kernel/linux-image-6.12.94+deb13-amd64


David