Re: Clones with fetch.bundleURI slower than standard, full clone?

On 2026-07-28 at 23:42:03, Knop, Ryszard wrote:
> Hey all,
> 
> I'm working on a Linux kernel-related CI system where we need to
> perform full clones of the kernel repo in most jobs. Because of some
> jobs in the pipeline, it usually cannot be a shallow clone :( Since the
> kernel repo is large and slow to clone, and I don't want to put undue
> stress on the remote host, I used git bundles, where CI clones the repo
> over a weekend, packages that as a bundle, then in jobs it gets used
> like this (weird, but works for <REF> being a branch, tag or a specific
> commit hash):
> 
> git init
> git remote add origin <REPO-URL>
> git config set fetch.bundleURI <BUNDLE-URL>
> git fetch origin <REF>
> git reset --hard FETCH_HEAD
> 
> Cloning a repo this way takes 4-5mins. Unpacking a bundle appears to be
> super slow. Not even faster than just running a full, normal clone from
> the remote server, actually (~3-4mins for a single branch).

You're comparing apples to oranges here.  A clone of a single branch
includes only that line of history and only those objects, but when you
use a bundle with multiple refs, Git has to handle all of the objects in
the bundle's entire pack, not just the ref you've specified.  In order
to compare adequately, you'd have to compare a bundle containing only
that one ref with the single-branch clone or a regular clone of the full
repository with your full bundles.

> On one of the build VMs, with Git 2.53 (stock Ubuntu 26.04), GIT_TRACE
> suggests most of the time is spent in some variation of `/usr/lib/git-
> core/git index-pack --stdin -v --fix-thin '--keep=fetch-pack 39430 on
> build-server' --check-self-contained-and-connected`, and indeed that
> process burns 100% of its single thread for most of that time.

As far as I can tell, the unbundling code just calls index-pack, so it
should honour pack.threads.  Setting that value to 0 causes this code to
be executed:

		/*
		 * Experiments show that going above 20 threads doesn't help,
		 * no matter how many cores you have. Below that, we tend to
		 * max at half the number of online_cpus(), presumably because
		 * half of those are hyperthreads rather than full cores. We'll
		 * never reduce the level below "3", though, to match a
		 * historical value that nobody complained about.
		 */
		if (nr_threads < 4)
			; /* too few cores to consider capping */
		else if (nr_threads < 6)
			nr_threads = 3; /* historic cap */
		else if (nr_threads < 40)
			nr_threads /= 2;
		else
			nr_threads = 20; /* hard cap */

So I would expect this to not be single threaded unless Git was compiled
without pthreads, run on a machine with few cores, or configured to use
only a single thread.  If you can get threading to work here, I expect
it will perform better, although I don't have any experience with bundle
URIs so I can't really say for certain.

> Is it expected that doing it this way is so slow? The alternative is to
> just package and work with the whole bare repo, but bundles appear to
> be an elegant way of dealing with exactly this scenario.

To be clear, it is insecure to do anything with an untrusted repo except
clone or fetch from it, so you will almost certainly not want to
distribute bare repos, since that will encourage people to use them
as-is (which is insecure).  Even if these are internal users who can
trust you, it encourages an anti-pattern which has security problems in
the general case.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

-----BEGIN PGP SIGNATURE-----

wr0EABYKAG8FgmppRfkJEHwMSWKIh6KBRxQAAAAAAB4AIHNhbHRAbm90YXRpb25z
LnNlcXVvaWEtcGdwLm9yZ0zXM82Pexkkg9aR2JSCOJDStKau6BqcfxIei7rOya9Z
FiEECCzmip28ZfuD0cORfAxJYoiHooEAAPf1AP95/BM8TKVi2AwkdPIDi9jBvyVY
N5NTcZwEUYDdW7u0YQEAlSWifugbBmhK5u/NNx3AtWrjLJKmj6/SJJPhhTeJ6gU=
=28D9
-----END PGP SIGNATURE-----