Re: Re: Can I generate a checksum for a directory?

On 3/9/26 5:13 PM, Frank Steinmetzger wrote:
> Right on the first post of this thread I thought “weren’t you a big fan of 
> my md5 directory hasher script”? :D
> 
> 
> Am Mon, Mar 02, 2026 at 03:30:54PM -0600 schrieb Dale:
> 
>> What I was hoping, I could tell it to generate a checksum for a entire
>> directory, the way it does for a file.  Then I do the same thing on my main
>> version and see if my main version matches the backup version.  I think that
>> even if this could be done, it would take days to generate it.
> 
> Definitely. But if you generate the checksum files once, then you have them 
> available and use them to detect bitrot. And reference them when copying 
> only individual directories.
> 
>>   I was hoping
>> for something that takes less than a hour at least, maybe even 15 or 20
>> minutes.  I'm getting the idea that what I'm wanting could take days or
>> longer to perform.  42TBs of data is a lot to check.  Some 64,000 files.
> 
> Well as others mentioned, there’s no other way than reading them all in.


You can still try to avoid fork+exec, which is particularly slow if
there's an extremely large number of *small* files.


$ cd sys-apps/portage; find -print0 | bsdtar -cnf - --format=mtree
--options='!all,indent,use-set,type,uid,gid,mode,sha256,link' --null
--files-from -


#mtree
/set type=file uid=1000 gid=1000 mode=644
.               mode=755 type=dir
./Manifest
sha256digest=e8d539b1720acf5a75ba1ab9ea9a10f16ec480e2b2180ffdcc249430131d4acf
./metadata.xml
sha256digest=0bdd464be5414059a65ce4d24ededb045fc82db3adef9ca02466ec00aa110b65
./portage-3.0.72-r1.ebuild \

sha256digest=7f8eceaf4580339c9e432df705b9bc5f09354ec96af903c005269d75489e7701
./portage-3.0.77-r1.ebuild \

sha256digest=30baee2cf0c2005572a2c03e44af2dc6af45aa4ae051a597ae5dab8d2a237683
./portage-3.0.77-r3.ebuild \

sha256digest=3b7dc601cc4a4bcea91eb1e7a5db11a6891083fb15d8c9759cb073c696dafea0
./portage-3.0.77.ebuild \

sha256digest=67bfd907dc2339205fd77c79b3a549e1c7c71aaefdbf48a4f678961edeee3bd0
./portage-9999.ebuild \

sha256digest=59b4fb52c84f358594c8aa8b73d372d67fcbe64cbc5fa4b0807cc2b6a1d862a3
./files         mode=755 type=dir
./files/0001-bin-emerge-ebuild-don-t-define-signal-handlers-in-gl.patch \

sha256digest=7c382290a6091c61e6078744352d01e29e516bf6aed37ddc6b04a0a21bc6e24b
./files/0001-estrip-silence-noise-from-objcopy-if-build-ID-sectio.patch \

sha256digest=fc8c6c6da5a3bf95ff3498fee0f8329c9b6eb0959fe10e1f46cfa01d78f34b83
./files/README.RESCUE \

sha256digest=a3f7a467a73c97d42c7c034d87b9480640677ab6dc1a318ec7bdd0dbdd67fab9
./files/portage-ccache.conf \

sha256digest=5175a18d3b81bd0aa499e0da6a2f44b920cf26f650e8dadea8dc2dc6551f8a39
./files/portage-tmpdir.conf \

sha256digest=f20b0c86299ad7f94c1516f925af0321ade5c32faae7ca998dc46ec295f5846a




See also https://man.freebsd.org/cgi/man.cgi?query=mtree






-- 
Eli Schwartz

-----BEGIN PGP SIGNATURE-----

wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCaa88EwUDAAAAAAAKCRCEp9ErcA0vV6YB
AP9pjMYsPfEkLvPuOrsfJsyDLE8YlMsjuloYHHjMfS5LYQD/d9ua/56fkov2ZHLTC35KV77KHSp3
Tfh/StR+13ZkBQ8=
=m2fy
-----END PGP SIGNATURE-----