Re: portscout fails with 905 Github ports

Moin Rahman <[email protected]> Sat, 2 Aug 2025 17:43:33 +0200
Newsgroups	gmane.os.freebsd.devel.ports
Message-ID	<[email protected]>

> On Aug 2, 2025, at 17:17, Sergei Vyshenski <[email protected]> wrote:
> 
> Hi,
> 
> As a rule, Github offers dynamic tarball generation. Dynamic tarballs do not exist at Gighub, they are generated on user's request from the sources, after user clicks on a special link. Which link points to virtual (non-existent) directory on Github. This technology is supported by FreeBSD's USE_GITHUB pragma.
> 
> But sometime project owner chooses to generate static tarballs by hand, and places them into a special and real directory at Github. This approach is used to escape cyclic dependencies or abnormally long compilations.
> 
> Cf. discussion here:
> https://lists.freebsd.org/pipermail/freebsd-ports/2020-July/119000.html
> 
> The technology of static tarballs is NOT supported by FreeBSD's USE_GITHUB pragma. To comply with it, FreeBSD's porters have to use something like:
> 
> MASTER_SITES= https://github.com/uriparser/uriparser/releases/download/uriparser-${DISTVERSION}/
> 
> Note that as it happens MASTER_SITES here obligatory involve a directory name with explicit version number.
> 
> It is exactly the point where Portscout fails. Because it keeps looking for new tarballs inside the directory of the old tarball. I have explicitly checked it by running portscout from my local host and asking to check a couple of ports of this kind.
> 
> The total number of problematic ports is 905:
> grep -RIl -E -e 'MASTER_SITES=.*//github.com/.*VERSION' --include Makefile --exclude-dir Mk /usr/ports > gh-static-names.txt
> 
> Full list gh-static-names.txt of these problematic ports (which use static tarballs from Github) is here:
> https://drive.google.com/file/d/13ExEXbtF5e1Rpgtu-YrsaO7-3FWVJ-7w/view?usp=sharing
> 
> Once portscout fails to process them, a workaround to trace new versions is: maintainer can subscribe to appropriate notifications from Github.
> 
> Do you know a way how developers or maintainers of portscout could be kindly asked to elaborate portscout to cope with this kind of ports?
> What PR (and where) can be created to address this issue?
> 
> Regards, Sergei

Hi,

I forgot when but sometimes ago there was a discussion in GH that from a certain
time forward they will not guaranty the checksum/hashes for the dynamically
generated tarballs. That is actually against how our current ports framework
works. In case the upstream tarballs hash changes the port will fail during
fetch. So from a certain time forward MASTER_SITES is/will be the preferred or
recommended method for upstream dists. So we cannot avoid that.

Now from a maintainer's point of view of portscout I am not exactly a maintainer
by choice but by circumstances. The original development of portscout has seized
I do not know since when. But I am often responsible to maintain the portscout
instance that is running in our cluster that is https://portscout.freebsd.org <https://portscout.freebsd.org/>.
So I just maintain in a sense like whatever is needed to do to support newer
perl versions and regular host updates. If someone has a patch which works I
will be happy to test and add those changes to the tree.

Kind regards,
Moin
signature.asc (application/pgp-signature, 963 B)