Re: Re: Can I generate a checksum for a directory?

Frank Steinmetzger <[email protected]> Mon, 9 Mar 2026 22:25:47 +0100
Newsgroups	gmane.linux.gentoo.user
Message-ID	<aa862zu0WeGvhxOd@q>
Am Mon, Mar 02, 2026 at 09:52:45PM +0000 schrieb Nate Eldredge:
> > On Mar 2, 2026, at 14:32, Dale <[email protected]> wrote:
> > 
> > On 3/2/26 8:17 AM, Grant Edwards wrote:
> >>> On 2026-03-02, Dale <[email protected]> wrote:
> >>> 
> >>> I was hoping I could do a whole directory not each individual file.
> >> Do you only want to know if the _directory_ has changed and not if the
> >> contents of any file has changed?  That's possible, but it might
> >> require a small C program: I can't think of existing utility that does
> >> that.
> > 
> > What I was hoping, I could tell it to generate a checksum for a entire directory, the way it does for a file.  Then I do the same thing on my main version and see if my main version matches the backup version.  I think that even if this could be done, it would take days to generate it.  I was hoping for something that takes less than a hour at least, maybe even 15 or 20 minutes.  I'm getting the idea that what I'm wanting could take days or longer to perform.  42TBs of data is a lot to check.  Some 64,000 files.
> 
> If your CPU is reasonable, this will likely be disk bound. Are your disks fast enough to read 42 TB in 15 minutes?

Recently I did some repartitioning on my PC’s main SSD, which changed all my 
files’ inode numbers. So the next time I ran unison to sync my machines, it 
had to read everything again. Same for borg when I did my next incremental 
backup; the content has not changed (so the backup did not grow in size), 
but because the inode was different, the file did not match its entry in the 
Borg cache, so it had to be re-read.

Why am I telling this? Borg on my machine parsed around 500 MB/s on a Ryzen 
8700G clocked at 4 GHz¹. My md5 script manages 2.1 GB/s at the same clock 
speed on a file in tmpfs. So unless you use NVMe SSDs or a fast RAID, you 
are definitely bound by I/O. Moreso if there is no RAID involved.


PS.: Just checked: my puny Raspberry Pi 3B can md5-hash at around 180 MB/s. 
I guess no x86 CPU of the last 10..15 years will break a sweat here.

PPS.: My md5 script is at least smart enough that when you added new files 
and need to re-calculate the directory’s md5 file, it will only hash files 
that aren’t listed in the md5 file (or files whose timestamp is newer than 
the md5 file’s, which can happen for example if you amend MP3 metadata). 
This gives a huge speed-up if you keep adding files to a growing directory.


¹ The 8700G reaches up to 5.1 GHz, but this is an ITX machine with a smallish 
cooler and I was encoding video at the time. So I limited the clock to keep 
the heat down.

That’s what I really love about linux; I was encoding 8 videos in parallel 
(each with one thread), reading input from an external USB hard disk. At the 
same time I ran Borg to update my backup on another USB disk, causing 
500 MB/s permanent read. And I did not feel any slowdown of the desktop.

-- 
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

One kilo of dark matter weighs ten tons.
signature.asc (application/pgp-signature, 833 B)