Re: Truncated archive/backup reparation

--===============4403370315859398490==
Content-Type: multipart/alternative; boundary="0000000000008369af0630181243"

--0000000000008369af0630181243
Content-Type: text/plain; charset="UTF-8"

Hi Dennis,

Thanks for the quick response, and for confirming that the behaviour I am
seeing is actually a bug and not user error on my part!

Regarding the new repaired backup that is created by -y, am I right in
understanding this would create a copy of all the slice data from the
truncated archive? I'm in the process of attempting to do a backup of
several hundred TBs, so ideally would like to be able to resume backing up
without duplicating all the existing backup if i need to repair a truncated
archive during the process. Is there an alternative mechanism by which the
repair process can read through a truncated archive to create an additional
file with the missing catalogue data at the end, from which a new
incremental backup can be run to continue the backup from where the
truncated one left off?

Best,

Scott


On Tue, 11 Mar 2025 at 18:27, Denis Corbin <[email protected]> wrote:

> On 11/03/2025 13:40, Scott Ritchie wrote:
> > Hi,
>
> Hi Scott,
>
> >
> > I'm trying to understand how to use the -y/--add-catalogue command to
> > repair a truncated archive, however, unfortunately have not been
> > successful in my own attempts after reading the various documentation/
> > tutorials/man page.
> >
> > First, I create a test backup where I simulate a power fault/computer
> > reboot part way through by sending the SIGKILL signal to the process:
> >
> > dar -c path/to/backup/backup_slice -s 1G -R folder/to/backup -g subfolder
> >
> > Let's say I am creating a backup of 10GB of data, and I send the SIGKILL
> > signal part way through the creation of slice 4 to simulate a power
> > failure / computer reboot.
> >
> > My understanding is the -y command should be able to recover (and
> > continue?) with this backup.
>
> The truncated archive lacks the catalogue at its end to be fully useful,
> the -y option *creates* a second backup while sequentially reading and
> gathering the metadata present along the source archive to create the
> table of content / catalog at the end of the new backup.
>
> Then you can use this repaired backup as reference in a differential
> backup process to continue the backup.
>
> And also that's up to you to decide whether to delete the truncated
> backup. Why not modifying the original backup? By precaution, for
> example what if, during the repairing process, an new power outage
> occurs... or you just hit a bug? And also some media may be read-only
> like CD/DVD-R/RAM...
>
> But if you really want at the end a single full backup you can convert a
> differential backup to a full backup, but this is yet a data copy...:
>
> http://dar.linux.free.fr/doc/FAQ.html#full-from-diff
>
>
> > To do so, I run:
> >
> > dar -y path/to/cont_backup/backup_slice -A path/to/backup/backup_slice -
> > s 1G -R folder/to/backup
> >
> > After reading through the incomplete archive, dar then tells me the next
> > backup slice (in this case backup_slice.5.dar) is required for further
> > operation, and asks me to provide this file. However, this file
> > obviously does not exist, since the initial backup was interrupted
> > during slice 4.
> >
> > When I hit ESC to tell dar I will not provide the file, dar outputs a
> > message that it failed reading data CRC for the last file in the
> > incomplete slice, and that the file may be damaged and will be marked as
> > dirty, before crashing with a segmentation fault.
>
> I could reproduce this bug/behavior, and will investigate this. The
> operations succeeds without slicing, but fails with slicing. For the
> rest, what you did are the expected actions.
>
> >
> > I'm clearly missing something here, but I'm not sure what. I'd be
> > grateful if someone on the mailing list could clarify, or provide a
> > working example?
>
> sure, first I'll fixe the bug you reported :)
>
> >
> > The version of dar I'm running is 2.7.16
> >
> > Kind Regards,
> >
> > Scott Ritchie
> >
>
> Cheers,
> Denis
>

--0000000000008369af0630181243
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Dennis,<br><br>Thanks for the quick response, and for c=
onfirming that the behaviour I am seeing is actually a bug and not user err=
or on my part!<div><br></div><div>Regarding the new repaired backup that is=
 created by -y, am I right in understanding this would create a copy of all=
 the slice data from the truncated archive? I&#39;m in the process of attem=
pting to do a backup of several hundred TBs, so ideally would like to be ab=
le to resume backing up without duplicating all the existing backup if i ne=
ed to repair a truncated archive during the process. Is there an alternativ=
e mechanism by which the repair process can read through a truncated archiv=
e to create an additional file with the missing catalogue data at the end, =
from which a new incremental backup can be run to continue the backup from =
where the truncated one left off?=C2=A0</div><div><br></div><div>Best,</div=
><div><br>Scott=C2=A0=C2=A0</div><div><br></div></div><br><div class=3D"gma=
il_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Tu=
e, 11 Mar 2025 at 18:27, Denis Corbin &lt;<a href=3D"mailto:dar.linux@free.=
fr">[email protected]</a>&gt; wrote:<br></div><blockquote class=3D"gmail_qu=
ote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,20=
4);padding-left:1ex">On 11/03/2025 13:40, Scott Ritchie wrote:<br>
&gt; Hi,<br>
<br>
Hi Scott,<br>
<br>
&gt; <br>
&gt; I&#39;m trying to understand how to use the=C2=A0-y/--add-catalogue co=
mmand to <br>
&gt; repair a truncated archive, however, unfortunately have not been <br>
&gt; successful in my own attempts after reading the various documentation/=
 <br>
&gt; tutorials/man page.<br>
&gt; <br>
&gt; First, I create a test backup where I simulate a power fault/computer =
<br>
&gt; reboot part way through=C2=A0by sending the SIGKILL signal to the proc=
ess:<br>
&gt; <br>
&gt; dar -c path/to/backup/backup_slice -s 1G -R folder/to/backup -g subfol=
der<br>
&gt; <br>
&gt; Let&#39;s say I am creating a backup of 10GB of data, and I send the S=
IGKILL <br>
&gt; signal part way through the creation of slice 4 to simulate a power <b=
r>
&gt; failure / computer reboot.<br>
&gt; <br>
&gt; My understanding is the -y command should be able to recover (and <br>
&gt; continue?) with this backup.<br>
<br>
The truncated archive lacks the catalogue at its end to be fully useful, <b=
r>
the -y option *creates* a second backup while sequentially reading and <br>
gathering the metadata present along the source archive to create the <br>
table of content / catalog at the end of the new backup.<br>
<br>
Then you can use this repaired backup as reference in a differential <br>
backup process to continue the backup.<br>
<br>
And also that&#39;s up to you to decide whether to delete the truncated <br=
>
backup. Why not modifying the original backup? By precaution, for <br>
example what if, during the repairing process, an new power outage <br>
occurs... or you just hit a bug? And also some media may be read-only <br>
like CD/DVD-R/RAM...<br>
<br>
But if you really want at the end a single full backup you can convert a <b=
r>
differential backup to a full backup, but this is yet a data copy...:<br>
<br>
<a href=3D"http://dar.linux.free.fr/doc/FAQ.html#full-from-diff" rel=3D"nor=
eferrer" target=3D"_blank">http://dar.linux.free.fr/doc/FAQ.html#full-from-=
diff</a><br>
<br>
<br>
&gt; To do so, I run:<br>
&gt; <br>
&gt; dar -y path/to/cont_backup/backup_slice -A path/to/backup/backup_slice=
 - <br>
&gt; s 1G -R folder/to/backup<br>
&gt; <br>
&gt; After reading through the incomplete archive, dar then tells me the ne=
xt <br>
&gt; backup slice (in this case backup_slice.5.dar) is required for further=
 <br>
&gt; operation, and asks me to provide this file. However, this file <br>
&gt; obviously does not exist, since the initial backup was interrupted <br=
>
&gt; during slice 4.<br>
&gt; <br>
&gt; When I hit ESC to tell dar I will not provide the file, dar outputs a =
<br>
&gt; message that it failed reading data CRC for the last file in the <br>
&gt; incomplete slice, and that the file may be damaged and will be marked =
as <br>
&gt; dirty, before crashing with a segmentation fault.<br>
<br>
I could reproduce this bug/behavior, and will investigate this. The <br>
operations succeeds without slicing, but fails with slicing. For the <br>
rest, what you did are the expected actions.<br>
<br>
&gt; <br>
&gt; I&#39;m clearly missing something here, but I&#39;m not sure what. I&#=
39;d be <br>
&gt; grateful if someone on the mailing list could clarify, or provide a <b=
r>
&gt; working example?<br>
<br>
sure, first I&#39;ll fixe the bug you reported :)<br>
<br>
&gt; <br>
&gt; The version of dar I&#39;m running is 2.7.16<br>
&gt; <br>
&gt; Kind Regards,<br>
&gt; <br>
&gt; Scott Ritchie<br>
&gt; <br>
<br>
Cheers,<br>
Denis<br>
</blockquote></div>

--0000000000008369af0630181243--


--===============4403370315859398490==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline


--===============4403370315859398490==--