Re: [PATCH RFC v3 2/2] Move libgit.a sources into separate "lib/" directory

On Thu, Aug 13, 2026 at 7:27 AM Junio C Hamano <[email protected]> wrote:
>
> Johannes Schindelin <[email protected]> writes:
>
> > Of course, it would be even nicer if `lib/` was split up further, but
> > then:
> >
> > 1) You've got to start _somewhere_. As we saw with so many things on this
> >    list, they never materialized because reviewers asked for too much and
> >    weren't happy to get incremental improvements first.
>
> That is why moving everything to 'lib/' and thinking about the rest
> later will not work.  Instead, moving a specific component to a
> specific subdirectory (not 'lib/') would be a reasonably
> self-contained first step.  Consider 'builtin/' as an example:  it
> is focused, and anyone can easily tell what the criterion is.  If
> it is the top-level cmd_foo() implementation, it goes there;
> otherwise, it does not.  Then, you can proceed to the second step,
> and then the third.  Iterate enough times, and the top level will
> become thin enough that you can either make your final step a
> no-op and leave the remaining files there, or create one last
> group to house the hodgepodge of leftover bits and move them there.
>
> > 2) Naming is hard. As we saw with _many_ refactorings (I am thinking about
> >    the low-level merge stuff as well as the ODB stuff, for example), it is
> >    unlikely to get the origanization right the first time. So I'd think
> >    that first moving the bulk of the library code to lib/ is a good start,
> >    and worth merging, leaving later contributions to chop off further
> >    parts into subdirectories of lib/.
>
> Again, this is because you are trying to do everything at once.
> Instead, come up with one clear concept, name it well, move the
> related files there, and then iterate.
>

Reading the discussion initially, I was inclined to agree with the
"move to lib/" idea. As a newer contributor, I do think the number of
files at the top level are a lot to navigate. Taking a step back to
roughly frame things, points of tension I see are:

  - the level of granularity for a conceptual grouping.
  - the level of conceptual cohesiveness before a grouping should be
    created, including what "cohesiveness" even means.
  - the need to create conceptual groups in a timely, deliberate
    manner without being too quick to create half-baked ones.

After considering these points, I ended up being in favor of coming up
with more granular, clear conceptual groupings before creating a
coarser, general grouping like "lib/". However, I do think this
grouping process can happen in a more timely and deliberate way. The
main appeal of moving stuff to "lib/" is to make progress on an effort
that otherwise has no concrete roadmap and allows us to "not let
perfect be the enemy of good."

There are already conceptual grouping efforts, both ongoing and past,
that might provide useful signals for how to make further progress.
A good set of signals would:

  - correctly reflect the conceptual groups that were already created
    successfully.
  - indicate candidate group areas as well as areas not "settled."
  - be generated by an ongoing, mechanical process that encodes
    the project's conceptual grouping standards.

I'm not necessarily sure if generating such a set is feasible or a
well-formed plan, but I wanted to provide a motivating example to
criticize.

One mechanical signal is how tightly a set of files shares the same
internal headers: if two .c files include many of the same project
headers (dropping the plumbing everything includes), they lean on the
same machinery. Averaged over every pair in the group, that gives one
cohesion number.

The already-carved directories have the following scores: refs/ scores
3.1, trace2/ 1.2, odb/ 1.0, while the whole root is 0.3. The scores
also seem predictive: run at the end of 2020, before odb/ existed, the
object-database files then at the root already scored 1.2, as cohesive
as trace2/ or negotiator/ were and 6x the root. odb/ is being carved
out of those exact files now.

Pointing forward, merging today's root .c files by that measure until
cohesion drops below the carved band proposes these groups, with no
hand-picked list [1]:

  6.9  connect.c fetch-pack.c remote-curl.c send-pack.c transport.c
       upload-pack.c
  6.8  diff-lib.c entry.c read-cache.c unpack-trees.c
  6.6  commit.c log-tree.c ref-filter.c revision.c sequencer.c
  6.3  diff.c diffcore-rename.c merge-ort.c
  6.0  delta-islands.c pack-bitmap-write.c pack-bitmap.c
carved directories, for reference:
  3.1  refs/
  1.0  odb/
  1.0  trace2/

Every proposed group is more cohesive than any directory already
carved: the same measure that would have flagged odb/ in 2020 points
now at a wire, index, revision, diff, and pack group.

I'm not sure these results necessarily provide the best signal, but
I believe it would be possible to mine a signal from Git's history /
existing project structure that would provide a principled roadmap
for deliberately grouping concepts.

[1] the (Claude generated) script:

import subprocess, re
from collections import defaultdict
from itertools import combinations

# shared distinctive #includes for every .c file; "distinctive" = not
# plumbing, i.e. included by <=12% of the root .c files
pat = r'^[[:space:]]*#[[:space:]]*include[[:space:]]+"[^"]+"'
out = subprocess.check_output(
    ["git", "grep", "-E", pat, "--", "*.c"], text=True)
q = re.compile(r'"([^"]+)"')
inc = defaultdict(set)
for line in out.splitlines():
    path, rest = line.split(":", 1)
    m = q.search(rest)
    if m:
        h = m.group(1).split("/")[-1]
        inc[path].add(h[:-2] if h.endswith(".h") else h)
root = [f for f in inc if "/" not in f]
docs = defaultdict(int)
for f in root:
    for h in inc[f]: docs[h] += 1
dist = {h for h, d in docs.items() if d <= len(root) * 0.12}
for f in inc: inc[f] &= dist
files = sorted(f for f in root if inc[f])

def sim(a, b): return len(inc[a] & inc[b])
def coh(m):
    ps = list(combinations(m, 2))
    return sum(sim(a, b) for a, b in ps) / len(ps) if ps else 0.0

# average-linkage agglomeration of the root files until the best merge's
# linkage drops below T; link[i][j] holds the summed pair similarity
T = 5.0
mem = {i: [f] for i, f in enumerate(files)}
size = {i: 1 for i in mem}
link = defaultdict(dict)
for a, b in combinations(range(len(files)), 2):
    s = sim(files[a], files[b])
    if s: link[a][b] = s; link[b][a] = s
act = set(mem); nid = len(files)
while True:
    best, bv = None, T
    for i in act:
        for j, ss in link[i].items():
            if i < j and ss / (size[i] * size[j]) >= bv:
                bv, best = ss / (size[i] * size[j]), (i, j)
    if not best: break
    i, j = best; c = nid; nid += 1
    mem[c] = mem[i] + mem[j]; size[c] = size[i] + size[j]
    for x in (set(link[i]) | set(link[j])) - {i, j}:
        s = link[i].get(x, 0) + link[j].get(x, 0)
        link[c][x] = s; link[x][c] = s
    for x in list(link[i]): link[x].pop(i, None)
    for x in list(link[j]): link[x].pop(j, None)
    link.pop(i, None); link.pop(j, None)
    act.discard(i); act.discard(j); act.add(c)

groups = sorted((mem[c] for c in act if size[c] >= 3),
                key=lambda m: -coh(m))
print("candidate subsystems (ranked by cohesion):")
for m in groups:
    print(f"  {coh(m):.1f}  {' '.join(sorted(m))}")
tree = subprocess.check_output(
    ["git", "ls-files", "--", "*.c"], text=True).split()
print("carved directories, for reference:")
for d in ["refs", "reftable", "odb", "trace2"]:
    c = coh([f for f in tree if f.startswith(d + "/")])
    if c: print(f"  {c:.1f}  {d}/")