Re: What is needed for ZFS block cloning to work?
Rich <[email protected]> Sun, 12 Jul 2026 13:57:03 -0400
| Newsgroups | gmane.os.freebsd.devel.file-systems |
|---|---|
| Message-ID | <CAOeNLuqbnUhrHveLLFOrdiKBD5C72c5=4QtvB6thVSuTKPUuEw@mail.gmail.com> |
--0000000000003567df06566db4d8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Unless the design changed from my memory, if you try to clone a block with the dedup bit set, it just increments the dedup table refcount and points to that copy, so BRT's accounting is never gonna get involved. - Rich On Sun, Jul 12, 2026 at 11:08=E2=80=AFAM John F Carr <[email protected]> wrote: > > > > On Jul 11, 2026, at 10:13=E2=80=AFPM, Rob Norris <[email protected]>= wrote: > > > > Yeah, the decision making is tricky. It's implementation-defined on > purpose, because we sometimes change things (mostly trying to remove > restrictions). > > > > Almost always it's something to do with the block alignment, but I'm no= t > sure off top of my head from this one > > > > On Sun, 12 Jul 2026, at 7:34 AM, John F Carr wrote: > >> I enabled block cloning on 15.1-STABLE but copy_file_range > >> between two files in the same directory does not cause > >> bcloneused to change. I have vfs.zfs.bclone_enabled=3D1 > >> and feature@block_cloning=3Denabled. Is anything more > >> needed? > > > > Your settings from the output you showed suggests everything was enable= d. > > > > Incidentally, the feature goes from enabled->active when the first bloc= k > is actually cloned, and from active->enabled with the last cloned block o= n > the pool is destroyed (that's normal ZFS feature flag management). > > > >> I looked at zfs_clone_range and zfs_freebsd_copy_file_range > >> and saw nothing obvious that would make the copy fail. > > > > Confirming those are the correct functions to look at. However, some of > the fine details are tricky to see, like the storage tier migration stuff= , > or the block sizes. > > > >> $ grep copy_file_range /tmp/t > >> copy_file_range(0x3,0x0,0x4,0x0,0x7fffffffffffffff,0x0) =3D 33289458 > (0x1fbf4f2) > >> copy_file_range(0x3,0x0,0x4,0x0,0x7fffffffffffffff,0x0) =3D 0 (0x0) > > > > NULL as the offsets (args 2 and 4) mean to use the current file offset. > So unless it did a seek or something to start the source at a weird offse= t, > that should be about the best possible case (and it wouldn't make any sen= se > to do that, just saying). > > > > So without looking at the dataset config itself, and the nature of the > file question, its hard to tell why not. > > > > The usual way I try to see what's happening is to trace the function, o= r > the SET_ERROR calls, something like: > > > > # dtrace -n 'fbt::zfs_clone_range:return { print(arg0) }' > > # dtrace -n 'sdt:::set-error /arg0 !=3D 0/ { stack(1); print(arg0) = } > > > > I am bad at dtrace, so there's probably much better ways. The output > though can help you get back to which condition it hit. > > > > The other way that is sometimes helpful is to use zdb to dump the L0 > block structure on the original and the copy (the one that didn't clone): > > > > # zdb tank -vv -dddddd -bbbbbb -O 20240930_195916.jpg | grep ' L0 ' > > 0 L0 DVA[0]=3D<0:72200:20000> [L0 ZFS plain file] > fletcher4 uncompressed unencrypted LE contiguous unique single > size=3D20000L/20000P birth=3D58L/58P fill=3D1 > cksum=3D000034f7743c1b68:0d33d8a49e7c498f:bdabce29daf9c36e:5a94839df86ca5= b9 > > 20000 L0 DVA[0]=3D<0:12200:20000> [L0 ZFS plain file] > fletcher4 uncompressed unencrypted LE contiguous unique single > size=3D20000L/20000P birth=3D58L/58P fill=3D1 > cksum=3D00003d5c8b629f10:0f4220ab15d3d245:c7109b9f43780acb:a25a0e574b4c23= c4 > > 40000 L0 DVA[0]=3D<0:32200:20000> [L0 ZFS plain file] > fletcher4 uncompressed unencrypted LE contiguous unique single > size=3D20000L/20000P birth=3D58L/58P fill=3D1 > cksum=3D00003d82eb6af316:0f5669851b30428c:e031566c1c107d50:9855785ebcc8bc= 0e > > ... > > > > Sometimes comparing the block sizes and options can tell you what > happened. The clone setup failed because it wanted to produce something > like the destination but couldn't turn the source into that. > > > > Yeah, complicated. > > > > Rob. > > > According to dtrace it's working fine, but bcloneused never changes. > > zfs_clone_range is called with input and output offset 0 and returns 0. > > I probed functions called from zfs_clone_range. > > dmu_read_l0_bps is called once, returning 0 > sa_bulk_update is called 4 times, returning 0 > dmu_brt_clone is called once, returning 0 > dmu_tx_create is called 6 times, returning the same pointer each time > dmu_tx_commit is called 6 times > zfs_log_clone_range is called once > > Probes in zfs_verify_zp and zfs_znode_update_vfs do not trigger even > though the functions have to be called. > > The nonzero SET_ERROR calls are > > 0x02 zap_lookup_impl+0x219 > 0x1c dsl_dir_tempreserve_space+0x449 > 0x02 dmu_spill_hold_existing+0x145 > 0x02 sa_attr_op+0x586 > 0x02 dbuf_hold_impl+0xb80 > 0x1c dsl_dir_tempreserve_space+0x449 > 0x02 dbuf_hold_impl+0xb80 [repeats hundreds of times] > 0x1c dsl_dir_tempreserve_space+0x449 > 0x02 dmu_spill_hold_existing+0x145 > 0x02 sa_attr_op+0x586 > 0x1c dsl_dir_tempreserve_space+0x449 > 0x1c dsl_dir_tempreserve_space+0x449 > > The ENOSPC returns are ignored by dmu_tx_assign because flags =3D=3D > DMU_TX_WAIT. > > My pool has deduplicated data and cache, which may take it out of the > usual code path. > > NAME SIZE ALLOC FREE AG CAP DEDUP HEALTH > data 72.0T 25.9T 46.1T 3% 35% 1.19x ONLINE > raidz2-0 72.0T 25.9T 46.1T 3% 36.0% - ONLINE > ada0p3 18T - - - - - ONLINE > ada1p3 18T - - - - - ONLINE > ada2p3 18T - - - - - ONLINE > ada3p3 18T - - - - - ONLINE > cache - - - - - - - > ada4p5 150G 146G 3.99G 0% 97.3% - ONLINE > > (blank columns omitted for clarity) > > Block cloning works on another single-SSD pool on the same server. > > What I don't understand is, if the cloning code path is taken and thinks > it succeeded, how does the accounting end up thinking a copy was made? > > John > > > --0000000000003567df06566db4d8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Unless the design changed from my memory, if you try = to clone a block with the dedup bit set, it just increments the dedup table= refcount and points to that copy, so BRT's accounting is never gonna g= et involved.</div><div><br></div><div>- Rich</div></div><br><div class=3D"g= mail_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On = Sun, Jul 12, 2026 at 11:08=E2=80=AFAM John F Carr <<a href=3D"mailto:jfc= @mit.edu">[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_qu= ote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,20= 4);padding-left:1ex"><br> <br> > On Jul 11, 2026, at 10:13=E2=80=AFPM, Rob Norris <<a href=3D"mailto= :[email protected]" target=3D"_blank">[email protected]</a>> wrote= :<br> > <br> > Yeah, the decision making is tricky. It's implementation-defined o= n purpose, because we sometimes change things (mostly trying to remove rest= rictions).<br> > <br> > Almost always it's something to do with the block alignment, but I= 'm not sure off top of my head from this one<br> > <br> > On Sun, 12 Jul 2026, at 7:34 AM, John F Carr wrote:<br> >> I enabled block cloning on 15.1-STABLE but copy_file_range<br> >> between two files in the same directory does not cause<br> >> bcloneused to change.=C2=A0 I have vfs.zfs.bclone_enabled=3D1<br> >> and feature@block_cloning=3Denabled.=C2=A0 Is anything more<br> >> needed?<br> > <br> > Your settings from the output you showed suggests everything was enabl= ed.<br> > <br> > Incidentally, the feature goes from enabled->active when the first = block is actually cloned, and from active->enabled with the last cloned = block on the pool is destroyed (that's normal ZFS feature flag manageme= nt).<br> > <br> >> I looked at zfs_clone_range and zfs_freebsd_copy_file_range<br> >> and saw nothing obvious that would make the copy fail.<br> > <br> > Confirming those are the correct functions to look at. However, some o= f the fine details are tricky to see, like the storage tier migration stuff= , or the block sizes.<br> > <br> >> $ grep copy_file_range /tmp/t<br> >> copy_file_range(0x3,0x0,0x4,0x0,0x7fffffffffffffff,0x0) =3D 332894= 58 (0x1fbf4f2)<br> >> copy_file_range(0x3,0x0,0x4,0x0,0x7fffffffffffffff,0x0) =3D 0 (0x0= )<br> > <br> > NULL as the offsets (args 2 and 4) mean to use the current file offset= . So unless it did a seek or something to start the source at a weird offse= t, that should be about the best possible case (and it wouldn't make an= y sense to do that, just saying).<br> > <br> > So without looking at the dataset config itself, and the nature of the= file question, its hard to tell why not.<br> > <br> > The usual way I try to see what's happening is to trace the functi= on, or the SET_ERROR calls, something like:<br> > <br> >=C2=A0 =C2=A0 =C2=A0# dtrace -n 'fbt::zfs_clone_range:return { prin= t(arg0) }'<br> >=C2=A0 =C2=A0 =C2=A0# dtrace -n 'sdt:::set-error /arg0 !=3D 0/ { st= ack(1); print(arg0) }<br> > <br> > I am bad at dtrace, so there's probably much better ways. The outp= ut though can help you get back to which condition it hit.<br> > <br> > The other way that is sometimes helpful is to use zdb to dump the L0 b= lock structure on the original and the copy (the one that didn't clone)= :<br> > <br> >=C2=A0 =C2=A0 =C2=A0# zdb tank -vv -dddddd -bbbbbb -O 20240930_195916.j= pg | grep ' L0 '<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0= =C2=A0 L0 DVA[0]=3D<0:72200:20000> [L0 ZFS plain file] fletcher4 unco= mpressed unencrypted LE contiguous unique single size=3D20000L/20000P birth= =3D58L/58P fill=3D1 cksum=3D000034f7743c1b68:0d33d8a49e7c498f:bdabce29daf9c= 36e:5a94839df86ca5b9<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 20000=C2=A0 L0 = DVA[0]=3D<0:12200:20000> [L0 ZFS plain file] fletcher4 uncompressed u= nencrypted LE contiguous unique single size=3D20000L/20000P birth=3D58L/58P= fill=3D1 cksum=3D00003d5c8b629f10:0f4220ab15d3d245:c7109b9f43780acb:a25a0e= 574b4c23c4<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 40000=C2=A0 L0 = DVA[0]=3D<0:32200:20000> [L0 ZFS plain file] fletcher4 uncompressed u= nencrypted LE contiguous unique single size=3D20000L/20000P birth=3D58L/58P= fill=3D1 cksum=3D00003d82eb6af316:0f5669851b30428c:e031566c1c107d50:985578= 5ebcc8bc0e<br> >=C2=A0 =C2=A0 =C2=A0...<br> > <br> > Sometimes comparing the block sizes and options can tell you what happ= ened. The clone setup failed because it wanted to produce something like th= e destination but couldn't turn the source into that.<br> > <br> > Yeah, complicated.<br> > <br> > Rob.<br> <br> <br> According to dtrace it's working fine, but bcloneused never changes.<br= > <br> zfs_clone_range is called with input and output offset 0 and returns 0.<br> <br> I probed functions called from zfs_clone_range.<br> <br> dmu_read_l0_bps is called once, returning 0<br> sa_bulk_update is called 4 times, returning 0<br> dmu_brt_clone is called once, returning 0<br> dmu_tx_create is called 6 times, returning the same pointer each time<br> dmu_tx_commit is called 6 times<br> zfs_log_clone_range is called once<br> <br> Probes in zfs_verify_zp and zfs_znode_update_vfs do not trigger even<br> though the functions have to be called.<br> <br> The nonzero SET_ERROR calls are<br> <br> 0x02=C2=A0 =C2=A0 zap_lookup_impl+0x219<br> 0x1c=C2=A0 =C2=A0 dsl_dir_tempreserve_space+0x449<br> 0x02=C2=A0 =C2=A0 dmu_spill_hold_existing+0x145<br> 0x02=C2=A0 =C2=A0 sa_attr_op+0x586<br> 0x02=C2=A0 =C2=A0 dbuf_hold_impl+0xb80<br> 0x1c=C2=A0 =C2=A0 dsl_dir_tempreserve_space+0x449<br> 0x02=C2=A0 =C2=A0 dbuf_hold_impl+0xb80 [repeats hundreds of times]<br> 0x1c=C2=A0 =C2=A0 dsl_dir_tempreserve_space+0x449<br> 0x02=C2=A0 =C2=A0 dmu_spill_hold_existing+0x145<br> 0x02=C2=A0 =C2=A0 sa_attr_op+0x586<br> 0x1c=C2=A0 =C2=A0 dsl_dir_tempreserve_space+0x449<br> 0x1c=C2=A0 =C2=A0 dsl_dir_tempreserve_space+0x449<br> <br> The ENOSPC returns are ignored by dmu_tx_assign because flags =3D=3D DMU_TX= _WAIT.<br> <br> My pool has deduplicated data and cache, which may take it out of the usual= code path.<br> <br> NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SIZE=C2=A0 ALLOC=C2=A0 =C2=A0FREE=C2= =A0 AG=C2=A0 =C2=A0 CAP=C2=A0 DEDUP=C2=A0 =C2=A0 HEALTH<br> data=C2=A0 =C2=A0 =C2=A0 =C2=A0 72.0T=C2=A0 25.9T=C2=A0 46.1T=C2=A0 3%=C2= =A0 =C2=A0 35%=C2=A0 1.19x=C2=A0 =C2=A0 ONLINE<br> =C2=A0 raidz2-0=C2=A0 72.0T=C2=A0 25.9T=C2=A0 46.1T=C2=A0 3%=C2=A0 36.0%=C2= =A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE<br> =C2=A0 =C2=A0 ada0p3=C2=A0 =C2=A0 18T=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 = =C2=A0 -=C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 = =C2=A0 ONLINE<br> =C2=A0 =C2=A0 ada1p3=C2=A0 =C2=A0 18T=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 = =C2=A0 -=C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 = =C2=A0 ONLINE<br> =C2=A0 =C2=A0 ada2p3=C2=A0 =C2=A0 18T=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 = =C2=A0 -=C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 = =C2=A0 ONLINE<br> =C2=A0 =C2=A0 ada3p3=C2=A0 =C2=A0 18T=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 = =C2=A0 -=C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -=C2=A0 = =C2=A0 ONLINE<br> cache=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A0 -=C2=A0= =C2=A0 =C2=A0 -=C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 -= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-<br> =C2=A0 ada4p5=C2=A0 =C2=A0 =C2=A0150G=C2=A0 =C2=A0146G=C2=A0 3.99G=C2=A0 0%= =C2=A0 97.3%=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE<br> <br> (blank columns omitted for clarity)<br> <br> Block cloning works on another single-SSD pool on the same server.<br> <br> What I don't understand is, if the cloning code path is taken and think= s<br> it succeeded, how does the accounting end up thinking a copy was made?<br> <br> John<br> <br> <br> </blockquote></div> --0000000000003567df06566db4d8--