Re: gzimg hashes
"David H. Gutteridge" <[email protected]> Mon, 13 Jul 2026 23:20:13 -0400
| Newsgroups | gmane.os.netbsd.current |
|---|---|
| Message-ID | <[email protected]> |
On Tue, 2026-07-07 at 10:52 -0600, Brook Milligan wrote: >=20 > > On Jul 6, 2026, at 10:49, Brook Milligan <[email protected]> wrote= : > >=20 > >=20 > > > On Jul 5, 2026, at 20:12, David H. Gutteridge <[email protected]> w= rote: > > >=20 > > > On Sun, 5 Jul 2026 at 17:42:17 -0600, Brook Milligan wrote: > > > > I notice that the hashes generated for release files in the gzimg > > > > directory cover the uncompressed files, not the compressed files > > > > actually distributed.=C2=A0 This seems to be a mistake. > > > >=20 > > > > The fix is clearly to reorder the hash generation relative to the > > > > compression, but I=E2=80=99m checking here to make sure I=E2=80=99m= not missing > > > > something. > > > >=20 > > > > Cheers, > > > > Brook > > >=20 > > > Yes, that's one of the issues raised in PR 59195 (and its duplicate, > > > PR 60302), e.g., > > > https://nam10.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fm= ail-index.netbsd.org%2Fnetbsd-bugs%2F2025%2F04%2F03%2Fmsg088095.html&data= =3D05%7C02%7Cbrook%40biology.nmsu.edu%7Cce313e1520f249b728b708dedb04114e%7C= a3ec87a89fb84158ba8ff11bace1ebaa%7C1%7C0%7C639189007694159583%7CUnknown%7CT= WFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFO= IjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C60000%7C%7C%7C&sdata=3D%2Bv5UsD4R9GqmbVitcRx= AG%2F81hAsh4KAIIb0ILBfSbhc%3D&reserved=3D0 > >=20 > > OK, thanks.=C2=A0 I have a patch, see below, that solves most of this.= =C2=A0=20 >=20 > Here is (I believe) a better patch + the referenced mkhashes script.=C2= =A0 It does the following: >=20 > - reverts the commit to src/distrib/utils/embedded/mkimage (rev 1.91) so = mkimage is no longer responsible for hashes >=20 > - creates the script src/distrib/utils/embedded/mkimage, which makes a st= andard set of hashes.=C2=A0 The boilerplate is based upon mkimage, so I hop= e that is OK. >=20 > - uses mkhashes to generate hashes for all (that I know of) users, i.e., = in src/etc/etc.*/Makefile.inc >=20 > Note that only a single hash for any file will be retained and multiple h= ashes will accumulate in the files. >=20 > I think this solves the issues mentioned in PR 59195 and PR 60302. >=20 > Comments are welcome. >=20 > Thanks a lot. >=20 > Cheers, > Brook Hello, As is, it doesn't work correctly for me with builds where there's more than one resulting image, e.g.: [disciple@arcusxx:gzimg]$ cat MD5 = = =20 MD5 (arm64mbr.img) =3D 8db3453cda03be499e0c663ccb608cf1 MD5 (arm64.img.gz) =3D 688ce2c196c9aa86ddb5ca4577f0942f [disciple@arcusxx:gzimg]$ cksum -cw MD5 (MD5) arm64mbr.img: FAILED [disciple@arcusxx:gzimg]$ md5 arm64.img.gz = = =20 MD5 (arm64.img.gz) =3D 688ce2c196c9aa86ddb5ca4577f0942f [disciple@arcusxx:gzimg]$ md5 arm64mbr.img.gz = = =20 MD5 (arm64mbr.img.gz) =3D 4d3ae51f9a103ab6e08a1f4b81b77f19 [disciple@arcusxx:gzimg]$ cat SHA512 = = =20 SHA512 (arm64mbr.img) =3D d788ac9d38ff026f97d2504f57f19a719d966c6abdf3cd8ae= c70eea69a2546ff3b758ac4596ada8a58eeeabb112a825e8e8c1dc5946cfeb450d2fa91398b= c2b4 SHA512 (arm64.img.gz) =3D bc0c519eaf4f207bcff34842a67bab67b44bc2d387234c07e= f79d2f4562bd4f7702eedabf92308aafd347063ebb9a4ec13daed4c020847d494460595c71a= 322a [disciple@arcusxx:gzimg]$ cksum -cw SHA512 = = =20 (SHA512) arm64mbr.img: FAILED [disciple@arcusxx:gzimg]$ sha512 arm64.img.gz = = =20 SHA512 (arm64.img.gz) =3D bc0c519eaf4f207bcff34842a67bab67b44bc2d387234c07e= f79d2f4562bd4f7702eedabf92308aafd347063ebb9a4ec13daed4c020847d494460595c71a= 322a [disciple@arcusxx:gzimg]$ sha512 arm64mbr.img.gz = = =20 SHA512 (arm64mbr.img.gz) =3D 7543ee99b9e8af9e6d86fc1beb39bbbbaaf62a7352dd6b= b70d9d50849c92d0ba4cd21d29b393f2854b7b3fb872dbeaf536fcf6f9170dd112ea1046daa= 01fdfc3 This is where I've built the "release" target from a clean state, then done an update build afterward (once further changes to the tree occurred). (I had a different approach I imagined for this, but I'm not sure it would work as I expect (I haven't tried), and would also be ugly in that I think it would require a cleanup make target in a different file than where the rest of the action occurs.) A couple of other comments follow in-line below. >=20 > Index: distrib/utils/embedded/mkimage > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > RCS file: /cvsroot/src/distrib/utils/embedded/mkimage,v > retrieving revision 1.97 > diff -u -r1.97 mkimage > --- distrib/utils/embedded/mkimage 22 May 2026 14:44:16 -0000 1.97 > +++ distrib/utils/embedded/mkimage 7 Jul 2026 16:36:35 -0000 > @@ -425,8 +425,4 @@ > =C2=A0 image=3D"${image}.gz" > =C2=A0fi > =C2=A0 > -cd "${IMAGEDIR}" > -${CKSUM} -a MD5 "$(basename "${image}")" > MD5 > -${CKSUM} -a SHA512 "$(basename "${image}")" > SHA512 > - > =C2=A0echo ${bar} Image is ${image} ${bar} > Index: etc/etc.evbarm/Makefile.inc > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > RCS file: /cvsroot/src/etc/etc.evbarm/Makefile.inc,v > retrieving revision 1.138 > diff -u -r1.138 Makefile.inc > --- etc/etc.evbarm/Makefile.inc 20 Feb 2026 07:20:28 -0000 1.138 > +++ etc/etc.evbarm/Makefile.inc 7 Jul 2026 16:36:35 -0000 > @@ -3,6 +3,7 @@ > =C2=A0# etc.evbarm/Makefile.inc -- evbarm-specific etc Makefile targets > =C2=A0# > =C2=A0 > +MKHASHES=3D${NETBSDSRCDIR}/distrib/utils/embedded/mkhashes > =C2=A0MKIMAGE=3D ${NETBSDSRCDIR}/distrib/utils/embedded/mkimage > =C2=A0 > =C2=A0# If you change the list of distributed kernels, don't forget > @@ -96,8 +97,6 @@ > =C2=A0 MACHINE=3D${MACHINE} \ > =C2=A0 MKDTB=3D${MKDTB} \ > =C2=A0 MKPOSTFIX=3D${MKPOSTFIX} \ > - CKSUM=3D${CKSUM} \ > - IMAGEDIR=3D${IMAGE.dir} \ > =C2=A0 ${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \ > =C2=A0 =C2=A0=C2=A0=C2=A0 -S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS= .${.TARGET}} \ > =C2=A0 =C2=A0=C2=A0=C2=A0 ${IMAGE.dir}/${.TARGET:S/smp_//}.img > @@ -107,10 +106,12 @@ > =C2=A0 @${TOOL_INSTALLBOOT} -m ${MACHINE} -o board=3D${f} \ > =C2=A0 ${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img \ > =C2=A0 && ${TOOL_GZIP} -f ${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img \ > + && ${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img.g= z \ > =C2=A0 || rm ${IMAGE.dir}/${.TARGET:S/smp_//}-${f}.img > =C2=A0.endfor > =C2=A0 @echo "=3D=3D=3D> Compressing image ${IMAGE.dir}/${.TARGET:S/smp_/= /}.img" > =C2=A0 ${TOOL_GZIP} -f ${GZIP_N_FLAG} -9 ${IMAGE.dir}/${.TARGET:S/smp_//}= .img > + ${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz=20 > =C2=A0 > =C2=A0ARCHES.earmv4=3D armv4 > =C2=A0ARCHES.earmv4eb=3D armv4 > Index: etc/etc.evbmips/Makefile.inc > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > RCS file: /cvsroot/src/etc/etc.evbmips/Makefile.inc,v > retrieving revision 1.32 > diff -u -r1.32 Makefile.inc > --- etc/etc.evbmips/Makefile.inc 22 Mar 2025 09:25:14 -0000 1.32 > +++ etc/etc.evbmips/Makefile.inc 7 Jul 2026 16:36:35 -0000 > @@ -52,6 +52,9 @@ > =C2=A0IMAGEENDIAN=3D le > =C2=A0.endif > =C2=A0 > +# Support for mkhashes > +MKHASHES=3D=C2=A0 ${NETBSDSRCDIR}/distrib/utils/embedded/mkhashes > + > =C2=A0__mkimage: .USE > =C2=A0 TOOL_MAKE=3D${MAKE} \ > =C2=A0 TOOL_MAKEFS=3D${TOOL_MAKEFS} \ > @@ -70,11 +73,10 @@ > =C2=A0 MACHINE=3D${MACHINE} \ > =C2=A0 MKDTB=3D${MKDTB} \ > =C2=A0 MKPOSTFIX=3D${MKPOSTFIX} \ > - CKSUM=3D${CKSUM} \ > - IMAGEDIR=3D${IMAGE.dir} \ > =C2=A0 ${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = -S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz > + ${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz > =C2=A0 > =C2=A0.if !empty(MACHINE_ARCH:Mmips*64eb) > =C2=A0smp_octeon: __mkimage > Index: etc/etc.evbppc/Makefile.inc > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > RCS file: /cvsroot/src/etc/etc.evbppc/Makefile.inc,v > retrieving revision 1.22 > diff -u -r1.22 Makefile.inc > --- etc/etc.evbppc/Makefile.inc 10 Jan 2026 06:56:58 -0000 1.22 > +++ etc/etc.evbppc/Makefile.inc 7 Jul 2026 16:36:35 -0000 > @@ -39,6 +39,9 @@ > =C2=A0IMAGE.kern=3D=C2=A0=C2=A0=C2=A0=C2=A0 ${IMAGE.rel}/binary/kernel > =C2=A0IMAGEENDIAN=3D be > =C2=A0 > +# Support for mkhashes > +MKHASHES=3D=C2=A0 ${NETBSDSRCDIR}/distrib/utils/embedded/mkhashes > + > =C2=A0.if ${MACHINE_ARCH} =3D=3D "powerpc" > =C2=A0__mkimage: .USE > =C2=A0 TOOL_MAKE=3D${MAKE} \ > @@ -58,11 +61,10 @@ > =C2=A0 MACHINE=3D${MACHINE} \ > =C2=A0 MKDTB=3D${MKDTB} \ > =C2=A0 MKPOSTFIX=3D${MKPOSTFIX} \ > - CKSUM=3D${CKSUM} \ > - IMAGEDIR=3D${IMAGE.dir} \ > =C2=A0 ${HOST_SH} -x ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = -S ${NETBSDSRCDIR} -B ${IMAGEENDIAN} ${MKI_OPTS.${.TARGET}} \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz > + ${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz > =C2=A0 > =C2=A0smp_nintendo: __mkimage > =C2=A0 cd ${KERNSRCDIR}/arch/evbppc/stand/wii && ${MAKE} release > Index: etc/etc.riscv/Makefile.inc > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > RCS file: /cvsroot/src/etc/etc.riscv/Makefile.inc,v > retrieving revision 1.7 > diff -u -r1.7 Makefile.inc > --- etc/etc.riscv/Makefile.inc 22 Mar 2025 09:25:14 -0000 1.7 > +++ etc/etc.riscv/Makefile.inc 7 Jul 2026 16:36:35 -0000 > @@ -15,6 +15,9 @@ > =C2=A0IMAGE.dir=3D=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ${IMAGE.rel}/binary/gzim= g > =C2=A0IMAGE.kern=3D=C2=A0=C2=A0=C2=A0=C2=A0 ${IMAGE.rel}/binary/kernel > =C2=A0 > +# Support for mkhashes > +MKHASHES=3D=C2=A0 ${NETBSDSRCDIR}/distrib/utils/embedded/mkhashes > + > =C2=A0# TOOL_MKUBOOTIMAGE=3D${TOOL_MKUBOOTIMAGE} \ > =C2=A0 > =C2=A0__mkimage: .USE > @@ -34,12 +37,10 @@ > =C2=A0 MACHINE=3D${MACHINE} \ > =C2=A0 MKDTB=3D${MKDTB} \ > =C2=A0 MKPOSTFIX=3D${MKPOSTFIX} \ > - CKSUM=3D${CKSUM} \ > - IMAGEDIR=3D${IMAGE.dir} \ > =C2=A0 ${HOST_SH} ${MKIMAGE} -h ${.TARGET:S/smp_//} -D ${DESTDIR} \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = -S ${NETBSDSRCDIR} ${MKI_OPTS.${.TARGET}} \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz > - > + ${MKHASHES} -c ${CKSUM} ${IMAGE.dir}/${.TARGET:S/smp_//}.img.gz > =C2=A0 > =C2=A0.if ${MACHINE_ARCH} =3D=3D "riscv64" > =C2=A0.if ${HAVE_UEFI} =3D=3D "yes" >=20 >=20 > #!/bin/sh > # $NetBSD$ > # > # Copyright (c) 2026 The NetBSD Foundation, Inc. > # All rights reserved. > # > # This code is derived from software contributed to The NetBSD Foundation > # by Brook Milligan. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions > # are met: > # 1. Redistributions of source code must retain the above copyright > #=C2=A0=C2=A0=C2=A0 notice, this list of conditions and the following dis= claimer. > # 2. Redistributions in binary form must reproduce the above copyright > #=C2=A0=C2=A0=C2=A0 notice, this list of conditions and the following dis= claimer in the > #=C2=A0=C2=A0=C2=A0 documentation and/or other materials provided with th= e distribution. > # 3. Neither the name of The NetBSD Foundation nor the names of its > #=C2=A0=C2=A0=C2=A0 contributors may be used to endorse or promote produc= ts derived > #=C2=A0=C2=A0=C2=A0 from this software without specific prior written per= mission. > # > # THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTO= RS > # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIM= ITED > # TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICU= LAR > # PURPOSE ARE DISCLAIMED.=C2=A0 IN NO EVENT SHALL THE FOUNDATION OR CONTR= IBUTORS > # BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF > # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINES= S > # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN > # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) > # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF = THE > # POSSIBILITY OF SUCH DAMAGE. > # The licensing terms standard for NetBSD is the two clause version, rather than three: https://www.netbsd.org/about/redistribution.html (I see mkimage has the older version; no one has changed it.) >=20 > # > # Makes a set of hashes for a file. > # >=20 > set -e >=20 > PROG=3D"$(basename "$0")" >=20 > cksum=3D${CKSUM:-cksum} > hashes=3D${HASHES:-"MD5 SHA512"} > imagedir=3D${IMAGEDIR} >=20 > bar=3D"=3D=3D=3D" >=20 > tmp=3D"$(mktemp -d "${TMPDIR:-/tmp}/$PROG.XXXXXX")" mktemp(1) is not part of POSIX (AFAIK) and is not guaranteed to be present on a given build host (even though that should be highly likely in our context), so this should be using the host tools framework, not a bare "mktemp" reference. (As is already done with cksum(1) here.) (The same is probably true for grep(1), since it's also part of the host tools.) >=20 > trap "cleanup" 0 1 2 3 15 >=20 > cleanup() { > case "$tmp" in > "${TMPDIR:-/tmp}/$PROG."*) rm -fr "$tmp";; > esac > } >=20 > usage() { > cat << EOF 1>&2 > Usage: $PROG [-H <hashes>] [-c <cksum>] [-d <imagedir>] <image> >=20 > -H Hash algorithms [${hashes}] > -c Checksum program [${cksum}] > -d Image directory [dirname \${image}] > EOF > exit 1 > } >=20 > OPTS=3D"H:c:d:" > while getopts "$OPTS" f > do > case $f in > H) hashes=3D"$OPTARG";; > c) cksum=3D"$OPTARG";; > d) imagedir=3D"$OPTARG";; > *) usage;; > esac > done >=20 > shift $(( $OPTIND - 1 )) > if [ -n "$1" ]; then > # take the next argument as being the image name > image=3D"$1" > shift > fi >=20 > if [ -z "${image}" ]; then > echo "${PROG}: image undefined" > usage > fi >=20 > if [ -z "${imagedir}" ]; then > imagedir=3D"$(dirname "${image}")" > fi >=20 > image=3D"$(basename "${image}")" >=20 > cd "${imagedir}" >=20 > if [ ! -f ${image} ]; then > echo "${PROG}: ${imagedir}/${image} does not exist" > usage > fi >=20 > for h in ${hashes}; do > echo "${bar} ${h} hash: ${imagedir}/${image} ${bar}" > if [ -f ${h} ]; then > grep -v "${image}" ${h} > ${tmp}/${h} || true > else > cp /dev/null ${tmp}/${h} > fi > ${cksum} -a ${h} "${image}" >> ${tmp}/${h} > mv ${tmp}/${h} ${h} > done Sorry for the delayed response, my sometimes meagre time for NetBSD was mostly taken up by a recent hardware failure. I could run more tests and look at what's happening in more detail soon. Thanks, Dave