Re: Hardlink support feasible?
Alan Savage <[email protected]> Sat, 17 Jan 2026 06:23:50 -0800
| Newsgroups | gmane.network.unison.general |
|---|---|
| Message-ID | <CAPPk+=MVG7tQ78XJRPb0YJLi1RoaoU2xW=O35gJorsNMc+xkdg@mail.gmail.com> |
--000000000000a13d0d0648963546
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Thanks for getting back to me quickly.
> it may be instructive to understand rsync's edge cases.
Agreed. `rsync -H` only preserves source hard links, not destination hard
links, so it is not symmetrical. If the destination does not support
hard-links the files are silently duplicated. If you then reverse source
and destination, the hard links are destroyed.
The `rsync -H` implementation builds an in-memory hard-link table keyed by
(device, inode). I think we'd want something similar, but that's getting
into implementation.
For a second pass, I'd try and preserve any asymmetry. E.g. if on Side A 2
files are hardlinked and on Side B they are not, then we keep them that
way. If one of the files on Side B is touched then the question would be
whether to
> if hardlinks are specified by POSIX and if real implementations follow
POSIX
POSIX specifies hardlink behavior for `linkat`, `unlinkat` and the
`fstatat` calls, so we can rely on the POSIX where it's available. I'm not
sure about Windows.
- https://urldefense.com/v3/__https://pubs.opengroup.org/onlinepubs/9799=
919799/functions/link.html__;!!IBzWLUs!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW=
0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1-5wAVdxZKN-3f1YkOQ$=20
- https://urldefense.com/v3/__https://pubs.opengroup.org/onlinepubs/9799=
919799/functions/unlink.html__;!!IBzWLUs!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyV=
nW0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1-5wAVdxZKN_OsrMeBQ$=20
- https://urldefense.com/v3/__https://pubs.opengroup.org/onlinepubs/9799=
919799/functions/fstatat.html__;!!IBzWLUs!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMy=
VnW0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1-5wAVdxZKN_L8icLZw$=20
Some of the relevant rsync code seems to rely on the POSIX API:
- generator.c atomic_create
<https://urldefense.com/v3/__https://github.com/RsyncProject/rsync/blob/=
2a97d81e99f61519acd4a683a9605216df1f91d4/generator.c*L1992__;Iw!!IBzWLUs!Vr=
ySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1-5wAV=
dxZKN-vKO3fIw$ >
->
- hlink.c hard_link_one
<https://urldefense.com/v3/__https://github.com/RsyncProject/rsync/bl=
ob/2a97d81e99f61519acd4a683a9605216df1f91d4/hlink.c*L454__;Iw!!IBzWLUs!VryS=
HPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1-5wAVdx=
ZKN-UPWcE7A$ >
->
- syscall.c do_link
<https://urldefense.com/v3/__https://github.com/RsyncProject/rsync=
/blob/2a97d81e99f61519acd4a683a9605216df1f91d4/syscall.c*L141__;Iw!!IBzWLUs=
!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1-5=
wAVdxZKN8RN53qMg$ >
- generator.c recv_generator
<https://urldefense.com/v3/__https://github.com/RsyncProject/rsync/blob/=
2a97d81e99f61519acd4a683a9605216df1f91d4/generator.c*L1207__;Iw!!IBzWLUs!Vr=
ySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1-5wAV=
dxZKN8NYCVBpg$ >
is
the big 1000-line top-level function that handles creating individual
file/dir with its parents. It'll take me some time to understand it and
document all the edge cases.
> what happens on filesystems without this facility
What happens right now? I would start with preserving that behavior or
leaving a big caveat that the behavior is undefined and logging a warning
with future updates having better conflict handling. What do you think?
In general, I would think we can treat POSIX errors the same way we
currently treat file creation/editing/deletion errors, but I'd have to dig
in to see if that's really sufficient.
> The second hard part is extending the sync protocol to represent
> hardlinks and decomposing the rules for what happens into rules for the
> two sides.
How can we proceed with this? Is there a good example to follow?
On Fri, Jan 16, 2026 at 4:20=E2=80=AFPM Greg Troxel <[email protected]> wrote:
> Alan Savage <[email protected]> writes:
>
> > Is it feasible to add hardlink support to Unison?
> >
> > My use case is an unstructured downloads folder used to seed torrents
> and a
> > structured folder containing hardlinks back to the downloads folder for
> use
> > with Jellyfin.
> >
> > E.g.
> > /data/downloads/SomeMovie-1999.mkv
> > /data/media/Movies/Some Movie (1999)/SomeMovie-1999.mkv # hardlinked
> with
> > downloads
> >
> > The /data folder would be synced with Unison between two computers over
> a
> > fairly slow WAN.
> >
> > I can potentially contribute code if the developers support the feature
> in
> > theory.
>
> I'm not opposed in concept, but I don't understand exactly what you
> mean.
>
> I don't deal with movies or torrents, so it would be good to frame the
> discussion purely in terms of hardlinked files :-)
>
>
>
> It seems clear that if you have within the replica two directory entries
> that are hard links to the same file (traditional unix hard link, not
> trying to be complicated here), and on the other replica neither exist,
> then propagation could create hardlinks to one file.
>
> It would be good to analzye how rsync, syncthing and other programs do
> or don't address this now. Just running rsync -av, source NetBSD zfs,
> dest NetBSD tmpfs, converted two names for one inode
>
> -rw-r--r-- 2 gdt wheel 1048576 Aug 2 2019 foo
> -rw-r--r-- 2 gdt wheel 1048576 Aug 2 2019 random.1M
>
> into two files
>
> -rw-r--r-- 1 gdt wheel 1048576 Aug 2 2019 foo
> -rw-r--r-- 1 gdt wheel 1048576 Aug 2 2019 random.1M
>
> adding H got me
>
> -rw-r--r-- 2 gdt wheel 1048576 Aug 2 2019 foo
> -rw-r--r-- 2 gdt wheel 1048576 Aug 2 2019 random.1M
>
> so it may be instructive to understand rsync's edge cases.
>
>
>
>
> I suspect that the first hard part is saying what should happen, in all
> the edge cases. Includes discussion of if hardlinks are specified by
> POSIX and if real implementations follow POSIX. Includes discussion of
> what happens on filesystems without this facility. And if A and B being
> hardlinks on one side is a conflict vs A and B being independent files
> on the other side (I vote yes, without thinking). Then there's
> hardlinks to out of the replica.
>
> The second hard part is extending the sync protocol to represent
> hardlinks and decomposing the rules for what happens into rules for the
> two sides.
>
> And finally the third part, to write code to do this.
>
> I'd say the first part belongs here and the second two on unison-hackers.
>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to [email protected].
--000000000000a13d0d0648963546
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>Thanks for getting back to me quickly.</div><div><br>=
</div><div>>=C2=A0it may be instructive to understand rsync's edge c=
ases.</div><div><br></div><div>Agreed. `rsync -H` only preserves source har=
d links, not destination hard links, so it is not symmetrical. If the desti=
nation does not support hard-links the files are silently duplicated. If yo=
u then reverse source and destination, the hard links are destroyed.</div><=
div><br></div><div>The `rsync -H` implementation builds an in-memory hard-l=
ink table keyed by (device, inode). I think we'd want something similar=
, but that's getting into implementation.</div><div><br></div><div>For =
a second pass, I'd try and preserve any asymmetry. E.g. if on Side A 2 =
files are hardlinked and on Side B they are not, then we keep them that way=
. If one of the files on Side B is touched then the question would be wheth=
er to=C2=A0</div><div><br></div><div>>=C2=A0if hardlinks are specified b=
y POSIX and if real implementations follow POSIX</div><div><br></div><div>P=
OSIX specifies hardlink behavior for `linkat`, `unlinkat` and the `fstatat`=
calls, so we can rely on the POSIX where it's available. I'm not s=
ure about Windows.</div><div><ul><li><a href=3D"https://urldefense.com/v3/_=
_https://pubs.opengroup.org/onlinepubs/9799919799/functions/link.html__;!!I=
BzWLUs!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RC=
Okf1-5wAVdxZKN-3f1YkOQ$">https://pubs.opengroup.org/onlinepubs/9799919799/f=
unctions/link.html</a></li><li><a href=3D"https://urldefense.com/v3/__https=
://pubs.opengroup.org/onlinepubs/9799919799/functions/unlink.html__;!!IBzWL=
Us!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1=
-5wAVdxZKN_OsrMeBQ$">https://pubs.opengroup.org/onlinepubs/9799919799/funct=
ions/unlink.html</a></li><li><a href=3D"https://urldefense.com/v3/__https:/=
/pubs.opengroup.org/onlinepubs/9799919799/functions/fstatat.html__;!!IBzWLU=
s!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1-=
5wAVdxZKN_L8icLZw$">https://pubs.opengroup.org/onlinepubs/9799919799/functi=
ons/fstatat.html</a></li></ul><div>Some of the relevant rsync code seems to=
rely on the POSIX API:<br><ul><li><a href=3D"https://urldefense.com/v3/__h=
ttps://github.com/RsyncProject/rsync/blob/2a97d81e99f61519acd4a683a9605216d=
f1f91d4/generator.c*L1992__;Iw!!IBzWLUs!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyVn=
W0vTgZC3dpfMeUkLr8pGUOs0t1_Gpn_RCOkf1-5wAVdxZKN-vKO3fIw$">generator.c atomi=
c_create</a> -></li><ul><li><a href=3D"https://urldefense.com/v3/__https=
://github.com/RsyncProject/rsync/blob/2a97d81e99f61519acd4a683a9605216df1f9=
1d4/hlink.c*L454__;Iw!!IBzWLUs!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3d=
pfMeUkLr8pGUOs0t1_Gpn_RCOkf1-5wAVdxZKN-UPWcE7A$">hlink.c hard_link_one</a>=
=C2=A0-></li><ul><li><a href=3D"https://urldefense.com/v3/__https://gith=
ub.com/RsyncProject/rsync/blob/2a97d81e99f61519acd4a683a9605216df1f91d4/sys=
call.c*L141__;Iw!!IBzWLUs!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3dpfMeU=
kLr8pGUOs0t1_Gpn_RCOkf1-5wAVdxZKN8RN53qMg$">syscall.c do_link</a></li></ul>=
</ul><li><a href=3D"https://urldefense.com/v3/__https://github.com/RsyncPro=
ject/rsync/blob/2a97d81e99f61519acd4a683a9605216df1f91d4/generator.c*L1207_=
_;Iw!!IBzWLUs!VrySHPw7QE0AC21f37BiZJex1Ebz-uEMyVnW0vTgZC3dpfMeUkLr8pGUOs0t1=
_Gpn_RCOkf1-5wAVdxZKN8NYCVBpg$">generator.c recv_generator</a>=C2=A0is the =
big 1000-line top-level function that handles creating individual file/dir =
with its parents. It'll take me some time to understand it and document=
all the edge cases.</li></ul></div><div>>=C2=A0what happens on filesyst=
ems without this facility</div><div><br></div><div>What happens right now? =
I would start with preserving that behavior or leaving a big caveat that th=
e behavior is undefined and logging a warning with future updates having be=
tter conflict handling. What do you think?</div><div><br></div><div>In gene=
ral, I would think we can treat POSIX errors the same way we currently trea=
t file creation/editing/deletion errors, but I'd have to dig in to see =
if that's really sufficient.</div></div><div><br></div><div><br>> Th=
e second hard part is extending the sync protocol to represent<br>> hard=
links and decomposing the rules for what happens into rules for the<br>>=
two sides.</div><div><br></div><div>How can we proceed with this? Is there=
a good example to follow?</div><br><div class=3D"gmail_quote gmail_quote_c=
ontainer"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, Jan 16, 2026 at 4:2=
0=E2=80=AFPM Greg Troxel <<a href=3D"mailto:[email protected]">[email protected]=
om</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margi=
n:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex=
">Alan Savage <<a href=3D"mailto:[email protected]" target=3D"_blank">=
[email protected]</a>> writes:<br>
<br>
> Is it feasible to add hardlink support to Unison?<br>
><br>
> My use case is an unstructured downloads folder used to seed torrents =
and a <br>
> structured folder containing hardlinks back to the downloads folder fo=
r use <br>
> with Jellyfin.<br>
><br>
> E.g.<br>
> /data/downloads/SomeMovie-1999.mkv<br>
> /data/media/Movies/Some Movie (1999)/SomeMovie-1999.mkv # hardlinked w=
ith <br>
> downloads<br>
><br>
> The /data folder would be synced with Unison between two computers ove=
r a <br>
> fairly slow WAN.<br>
><br>
> I can potentially contribute code if the developers support the featur=
e in <br>
> theory.<br>
<br>
I'm not opposed in concept, but I don't understand exactly what you=
<br>
mean.<br>
<br>
I don't deal with movies or torrents, so it would be good to frame the<=
br>
discussion purely in terms of hardlinked files :-)<br>
<br>
<br>
<br>
It seems clear that if you have within the replica two directory entries<br=
>
that are hard links to the same file (traditional unix hard link, not<br>
trying to be complicated here), and on the other replica neither exist,<br>
then propagation could create hardlinks to one file.<br>
<br>
It would be good to analzye how rsync, syncthing and other programs do<br>
or don't address this now.=C2=A0 Just running rsync -av, source NetBSD =
zfs,<br>
dest NetBSD tmpfs, converted two names for one inode<br>
<br>
=C2=A0 -rw-r--r--=C2=A0 2 gdt=C2=A0 wheel=C2=A0 1048576 Aug=C2=A0 2=C2=A0 2=
019 foo<br>
=C2=A0 -rw-r--r--=C2=A0 2 gdt=C2=A0 wheel=C2=A0 1048576 Aug=C2=A0 2=C2=A0 2=
019 random.1M<br>
<br>
into two files<br>
<br>
=C2=A0 -rw-r--r--=C2=A0 1 gdt=C2=A0 wheel=C2=A0 1048576 Aug=C2=A0 2=C2=A0 2=
019 foo<br>
=C2=A0 -rw-r--r--=C2=A0 1 gdt=C2=A0 wheel=C2=A0 1048576 Aug=C2=A0 2=C2=A0 2=
019 random.1M<br>
<br>
adding H got me<br>
<br>
=C2=A0 -rw-r--r--=C2=A0 2 gdt=C2=A0 wheel=C2=A0 1048576 Aug=C2=A0 2=C2=A0 2=
019 foo<br>
=C2=A0 -rw-r--r--=C2=A0 2 gdt=C2=A0 wheel=C2=A0 1048576 Aug=C2=A0 2=C2=A0 2=
019 random.1M<br>
<br>
so it may be instructive to understand rsync's edge cases.<br>
<br>
<br>
<br>
<br>
I suspect that the first hard part is saying what should happen, in all<br>
the edge cases.=C2=A0 Includes discussion of if hardlinks are specified by<=
br>
POSIX and if real implementations follow POSIX.=C2=A0 Includes discussion o=
f<br>
what happens on filesystems without this facility.=C2=A0 And if A and B bei=
ng<br>
hardlinks on one side is a conflict vs A and B being independent files<br>
on the other side (I vote yes, without thinking).=C2=A0 Then there's<br=
>
hardlinks to out of the replica.<br>
<br>
The second hard part is extending the sync protocol to represent<br>
hardlinks and decomposing the rules for what happens into rules for the<br>
two sides.<br>
<br>
And finally the third part, to write code to do this.<br>
<br>
I'd say the first part belongs here and the second two on unison-hacker=
s.<br>
</blockquote></div></div>
<p></p>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:[email protected]">unison-u=
[email protected]</a>.<br />
--000000000000a13d0d0648963546--