Re: Truncated archive/backup reparation
Scott Ritchie <[email protected]> Fri, 14 Mar 2025 14:01:43 +0000
| Newsgroups | gmane.comp.sysutils.backup.dar.support |
|---|---|
| Message-ID | <CAO1VBV0cC_+cCBvwwqR=0Ge+81qr8b+u41Y6G3BffWnS61n7KA@mail.gmail.com> |
--===============6512515770673628571== Content-Type: multipart/alternative; boundary="0000000000008d87f706304de17f" --0000000000008d87f706304de17f Content-Type: text/plain; charset="UTF-8" Hi Denis, I suspect you are right, that using the -E and -F options would afford me the flexibility I'm looking for. In the end however, I have decided to go with a simpler but more manual approach of terminating dar manually every for TB while it is not in the paused state, effectively splitting the full backup into multiple incremental backups. For posterity, in case someone else encounters the issues, basically I've been tasked with creating backups of my department's datasets which are currently stored on a HPC system. The backup system is done via creating backup files on a "research cold store" ( https://docs.hpc.cam.ac.uk/storage/rcs/best-practice.html) which is then backed up to tape behind the scenes by the HPC admin team. The HPC team have asked me to bake in pauses into the dar backup creation process, as the rate of data transfer to the research cold store disk cache is exceeding the rate of data transfer from the disk cache to tape. The amount of data I'm backing up is also such that the overall transfer time is exceeding the interval between routine HPC node reboots. In the process of learning dar, I ran into two scenarios where dar does not cleanly exit (i.e. delayed termination does not take place): (1) when par2 is terminated, and (2) when dar is terminated while in the paused state waiting for user confirmation to continue. In both cases, dar gets stuck waiting for user input, both requiring an initial <ESC> input to confirm that either the par2 command should not be rerun, or in the pause state, that we do not want to continue, then a sequence of 2-3 enter keystrokes on subsequent prompts until dar cleanly goes through the process of delayed termination. My thought process in the previous email was that perhaps in the wrapper bash script I'd written I could intercept the shutdown signals sent system wide on node reboot, and then programmatically provide the sequence of desired keystrokes in case dar was currently in the waiting state between slices. I found the bash trap command for intercepting signals, and thought I might be able to programmatically write to the dar process stdin file connection (i.e. writing to /proc/<pid>/fd/0), however, after quite a bit of trial and error and googling, I learnt that this was not possible without root access, requiring the IOCTL facilities to push to the stdin buffer, rather than simply piping text to /proc/<pid>/fd/0. In case anyone else finds this thread at a later date and wants to go down this rabbit hole, see https://unix.stackexchange.com/questions/385771/writing-to-stdin-of-a-process This brings me back to my opening paragraph - I suspect the correct solution to my problem would be using the -E option to write some more flexible code between slices that does what I wanted, but instead I opted for the simpler approach of just running the backup and manually terminating the dar process on a regular basis in such a way that it cleanly exits with delayed termination. Partly because this would also have the added advantage of checkpointing the backup process so that only a small part of the backup would need to be redone if there was a power failure or other problem leading to a truncated archive, which also alleviates the need to wait for the stable release of version 2.8.0. Best, Scott On Fri, 14 Mar 2025 at 05:00, Denis Corbin <[email protected]> wrote: > On 13/03/2025 14:36, Scott Ritchie wrote: > > Hi Denis, > > Hi Scott, > > thanks for reporting this error, I will investigate this ASAP. > > About the signal impact on dar: when dar is waiting for an answer from > the user, libdar is suspending within a blocking read() system call > while signals are blocked. I must see the impact of not blocking INT and > TERM signals in that context... as this has probably been done that way > for some reasons, some decades ago... > > In the meanwhile, while I have not all the constraints regarding the > backup objective you have, so far when dar/libdar users had to automate > a backup process, the use scripts with -E, -F, -~ options used in > conjunction with -Q was enough to address all needs, eventually wrapping > also dar in a customized script (Some user shared their experiences > which I have kept with their agreement in the doc/samples directory of > the source package, for illustration). > > Why, in your context, do you need to have dar pausing between slices (-p > option)? > > Regards, > Denis > > > > > Thanks so much for all the advice and for implementing/patching so > > quickly. I appreciate that it will take some time for these to be rolled > > out into the stable releases. > > > > RE: the -Q option, after seeing that -Q is not compatible with -p, I've > > been playing around with the possibility of trapping the SIGTERM signal > > in my the bash script I've written to run dar with a view towards > > programmatically providing the sequence of ESC/ENTER commands required > > to respond to the user input prompts to lead to a clean delayed > > termination. > > > > In doing so I managed to trigger the following exception which the dar > > command has asked me to report: > > > > Aborting program. An error occurred while calling libdar: Cannot write > > down the archive: user_interaction::inherited_pause should not throw an > > exception toward libdar > > ############################################### > > # NOT CAUGHT EXCEPTION, # > > # E X I T I N G ! # > > # # > > ############################################### > > THANKS TO REPORT THE PREVIOUS OUTPUT TO MAINTAINER > > GIVING A DESCRIPTION OF THE CIRCUMSTANCES. > > IF POSSIBLE TRY TO PRODUCE THIS ERROR, A > > SCENARIO THAT CAN REPRODUCE IT WOULD HELP MUCH > > IN SOLVING THIS PROBLEM. THANKS > > > > I think this uncaught exception can be reliably reproduced by calling > > dar asynchronously from a bash script and then killing the script: > > > > ``` > > #/bin/bash > > > > # Catch interrupt signals and cleanup > > function cleanup { > > # do stuff > > exit 1 > > } > > trap cleanup SIGHUP SIGINT SIGQUIT SIGTERM SIGUSR1 > > > > # Run dar asynchronously and wait for the response, so we can use trap > > dar -c path/to/backup -s 10GB & > > wait > > ``` > > > > Execute above script, then kill -15 the the process running the bash > > script without killing the running dar process. > > > > Hopefully this provides helpful insight into one of the possible ways > > this scenario can be triggered. > > > > Best, > > > > Scott > > > > On Wed, 12 Mar 2025 at 23:18, Denis Corbin <[email protected] > > <mailto:[email protected]>> wrote: > > > > On 12/03/2025 16:32, Scott Ritchie wrote: > > > Thanks Dennis, > > > > Hi Scott > > > > As said, you have a fix in 2.7.17.RC2 for the repairing of a sliced > > backup in git with tag v2.7.17.RC2 and now in addition a package > here: > > > > https://dar.edrusb.org/dar.linux.free.fr/Interim_releases <https:// > > dar.edrusb.org/dar.linux.free.fr/Interim_releases> > > > > I currently run the non-regression tests on that code, which takes > > several days, so be patient before the release 2.7.17. > > > > I could add the possibility to create an isolated a catalog from a > > truncated backup, but this requires the -affs feature that is only > > available in the 2.8.0 dev branch (= git master branch), so I have > > added > > this new feature there it will be released with 2.8.0, in a few > months. > > Also added a FAQ on the possible way to manage a truncated backup, > --- > > OK this is not a Frequently Asked question as nobody ever asked on > that > > feature :) ---- but that was the easiest place to add this short > > documentation. > > > > > > > > RE: archive truncation, another thing that I've come across is the > > > behaviour of dar when sent the SIGTERM (or similar) signals. > > Normally > > > this leads to delayed termination including appending the catalog > > to the > > > end of the archive; however, this is not the case if dar is sent > the > > > SIGTERM signal while paused between slices dar continues to wait > for > > > user input. Would it be possible to override this so that if dar > > is sent > > > a SIGTERM signal while waiting for user input it instead > > continues with > > > the intended delayed termination so that the archive doesn't end > > up in a > > > truncated state (e.g. if the process is killed due to non- > > response to > > > the SIGTERM signal)? Also worth noting that the same behaviour > > occurs > > > when dar is interrupted while waiting for par2 to finish running > > between > > > slices. > > > > AFAIR, the process is suspending on a system call when it waits for > > user > > to answer... To address this type of use case, where you want to > > automate a process, there is the -Q option that has been added. Maybe > > this will address you need better than having the dar questions to > > answer manually. > > > > > > > > The backup I am trying to create (100s of Tb) is unfortunately > > going to > > > take quite a long time to run, longer than the routinely scheduled > > > reboot periods of the cluster I am working on, so am trying to > code > > > around the scenario where a reboot signal is sent while par2 is > > running > > > on a slice or during a period when dar2 is otherwise paused > (which I > > > have been asked to schedule in every few TBs). > > > > You could also split the data to backup in several smaller backups > (one > > per top directory or something in that taste that makes sense) and > > gather then into a single dar_manager database. You can then choose > > different rotation frequency and retention period between these > smaller > > backups and have a single point of access through dar_manager. > > > > > > > > Best, > > > > > > Scott > > > > Cheers, > > Denis > > > > > > --0000000000008d87f706304de17f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi Denis,<br><br>I suspect you are right, that using the -= E and -F options would afford me the flexibility I'm looking for. In th= e end however, I have decided to go with a simpler but more manual approach= of terminating dar manually every for TB while it is not in the paused sta= te, effectively splitting the full backup into multiple incremental backups= .<div><br></div><div>For posterity, in case someone else encounters the iss= ues, basically I've been tasked with creating backups of my=C2=A0depart= ment's datasets which are currently stored on a HPC system. The backup = system is done via creating backup files on a "research cold store&quo= t; (<a href=3D"https://docs.hpc.cam.ac.uk/storage/rcs/best-practice.html">h= ttps://docs.hpc.cam.ac.uk/storage/rcs/best-practice.html</a>) which is then= backed up to tape behind the scenes by the HPC admin team. The HPC team ha= ve asked me to bake in pauses into the dar backup creation process, as the = rate of data transfer to the research cold store disk cache is exceeding th= e rate of data transfer from the disk cache to tape. The amount of data I&#= 39;m backing up is also such that the overall transfer time is exceeding th= e interval between routine HPC node reboots.</div><div><br></div><div>In th= e process of learning dar, I ran into two scenarios where dar does not clea= nly exit (i.e. delayed termination does not take place): (1) when par2 is t= erminated, and (2) when dar is terminated while in the paused state waiting= for user confirmation to continue. In both cases, dar gets stuck waiting f= or user input, both requiring an initial <ESC> input to confirm that = either the par2 command should not be rerun, or in the pause state, that we= do not want to continue, then a sequence of 2-3 enter keystrokes on subseq= uent prompts until dar cleanly goes through the process of delayed terminat= ion.=C2=A0<br><br>My thought process in the previous email was that perhaps= in the wrapper bash script I'd written I could intercept=C2=A0 the shu= tdown signals sent system wide on node reboot, and then programmatically pr= ovide the sequence of desired keystrokes in case dar was currently in the w= aiting state between slices. I found the bash trap command for intercepting= signals, and thought I might be able to programmatically write to the dar = process stdin file connection (i.e. writing to /proc/<pid>/fd/0), how= ever, after quite a bit of trial and error and googling, I learnt that this= was not possible without root access, requiring the IOCTL facilities to pu= sh to the stdin buffer, rather than simply piping text to /proc/<pid>= /fd/0. In case anyone else finds this thread at a later date and wants to g= o down this rabbit hole, see=C2=A0<a href=3D"https://unix.stackexchange.com= /questions/385771/writing-to-stdin-of-a-process">https://unix.stackexchange= .com/questions/385771/writing-to-stdin-of-a-process</a>=C2=A0</div><div><br= ></div><div>This brings me back to my opening paragraph - I suspect the cor= rect solution to my problem would be using the -E option to write some more= flexible code between slices that does what I wanted, but instead I opted = for the simpler approach of just running the backup and manually terminatin= g the dar process on a regular basis in such a way that it cleanly exits wi= th delayed termination. Partly because this would also have the added advan= tage of checkpointing the backup process so that only a small part of the b= ackup would need to be redone if there was a power failure or other problem= leading to a truncated archive, which also alleviates the need to wait for= the stable release of version 2.8.0.</div><div><br></div><div>Best,</div><= div><br></div><div>Scott</div></div><br><div class=3D"gmail_quote gmail_quo= te_container"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, 14 Mar 2025 at = 05:00, Denis Corbin <<a href=3D"mailto:[email protected]">dar.linux@free= .fr</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg= in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e= x">On 13/03/2025 14:36, Scott Ritchie wrote:<br> > Hi Denis,<br> <br> Hi Scott,<br> <br> thanks for reporting this error, I will investigate this ASAP.<br> <br> About the signal impact on dar: when dar is waiting for an answer from <br> the user, libdar is suspending within a blocking read() system call <br> while signals are blocked. I must see the impact of not blocking INT and <b= r> TERM signals in that context... as this has probably been done that way <br= > for some reasons, some decades ago...<br> <br> In the meanwhile, while I have not all the constraints regarding the <br> backup objective you have, so far when dar/libdar users had to automate <br= > a backup process, the use scripts with -E, -F, -~ options used in <br> conjunction with -Q was enough to address all needs, eventually wrapping <b= r> also dar in a customized script (Some user shared their experiences <br> which I have kept with their agreement in the doc/samples directory of <br> the source package, for illustration).<br> <br> Why, in your context, do you need to have dar pausing between slices (-p <b= r> option)?<br> <br> Regards,<br> Denis<br> <br> > <br> > Thanks so much for all the advice and for implementing/patching so <br= > > quickly. I appreciate that it will take some time for these to be roll= ed <br> > out into the stable releases.<br> > <br> > RE: the -Q option, after seeing that -Q is not compatible with -p, I&#= 39;ve <br> > been playing around with the possibility of trapping the SIGTERM signa= l <br> > in my the bash script I've written to run dar with a view towards = <br> > programmatically providing the sequence of ESC/ENTER commands required= <br> > to respond to the user input prompts to lead to a clean delayed <br> > termination.<br> > <br> > In doing so I managed to trigger the following exception which the dar= <br> > command has asked me to report:<br> > <br> > Aborting program. An error occurred while calling libdar: Cannot write= <br> > down the archive: user_interaction::inherited_pause should not throw a= n <br> > exception toward libdar<br> > ###############################################<br> > # =C2=A0 NOT CAUGHT EXCEPTION, =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 #<br> > # =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 E X I T I N G ! =C2=A0 =C2=A0 #<br> > # =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 #<br> > ###############################################<br> >=C2=A0 =C2=A0THANKS TO REPORT THE PREVIOUS OUTPUT TO MAINTAINER<br> >=C2=A0 =C2=A0GIVING A DESCRIPTION OF THE CIRCUMSTANCES.<br> >=C2=A0 =C2=A0IF POSSIBLE TRY TO PRODUCE THIS ERROR, A<br> >=C2=A0 =C2=A0SCENARIO THAT CAN REPRODUCE IT WOULD HELP MUCH<br> >=C2=A0 =C2=A0IN SOLVING THIS PROBLEM. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0THANKS<br> > <br> > I think this uncaught exception can be reliably reproduced by calling = <br> > dar asynchronously from a bash script and then killing the script:<br> > <br> > ```<br> > #/bin/bash<br> > <br> > # Catch interrupt signals and cleanup<br> > function cleanup {<br> >=C2=A0 =C2=A0 =C2=A0# do stuff<br> >=C2=A0 =C2=A0 =C2=A0exit 1<br> > }<br> > trap cleanup SIGHUP SIGINT SIGQUIT SIGTERM SIGUSR1<br> > <br> > # Run dar asynchronously and wait for the response, so we can use trap= <br> > dar -c path/to/backup -s 10GB &<br> > wait<br> > ```<br> > <br> > Execute above script, then kill -15 the the process running the bash <= br> > script without killing the running dar process.<br> > <br> > Hopefully this provides helpful insight into one of the possible ways = <br> > this scenario can be triggered.<br> > <br> > Best,<br> > <br> > Scott<br> > <br> > On Wed, 12 Mar 2025 at 23:18, Denis Corbin <<a href=3D"mailto:dar.l= [email protected]" target=3D"_blank">[email protected]</a> <br> > <mailto:<a href=3D"mailto:[email protected]" target=3D"_blank">dar.= [email protected]</a>>> wrote:<br> > <br> >=C2=A0 =C2=A0 =C2=A0On 12/03/2025 16:32, Scott Ritchie wrote:<br> >=C2=A0 =C2=A0 =C2=A0 > Thanks Dennis,<br> > <br> >=C2=A0 =C2=A0 =C2=A0Hi Scott<br> > <br> >=C2=A0 =C2=A0 =C2=A0As said, you have a fix in 2.7.17.RC2 for the repai= ring of a sliced<br> >=C2=A0 =C2=A0 =C2=A0backup in git with tag v2.7.17.RC2 and now in addit= ion a package here:<br> > <br> >=C2=A0 =C2=A0 =C2=A0<a href=3D"https://dar.edrusb.org/dar.linux.free.fr= /Interim_releases" rel=3D"noreferrer" target=3D"_blank">https://dar.edrusb.= org/dar.linux.free.fr/Interim_releases</a> <https://<br> >=C2=A0 =C2=A0 =C2=A0<a href=3D"http://dar.edrusb.org/dar.linux.free.fr/= Interim_releases" rel=3D"noreferrer" target=3D"_blank">dar.edrusb.org/dar.l= inux.free.fr/Interim_releases</a>><br> > <br> >=C2=A0 =C2=A0 =C2=A0I currently run the non-regression tests on that co= de, which takes<br> >=C2=A0 =C2=A0 =C2=A0several days, so be patient before the release 2.7.= 17.<br> > <br> >=C2=A0 =C2=A0 =C2=A0I could add the possibility to create an isolated a= catalog from a<br> >=C2=A0 =C2=A0 =C2=A0truncated backup, but this requires the -affs featu= re that is only<br> >=C2=A0 =C2=A0 =C2=A0available in the 2.8.0 dev branch (=3D git master b= ranch), so I have<br> >=C2=A0 =C2=A0 =C2=A0added<br> >=C2=A0 =C2=A0 =C2=A0this new feature there it will be released with 2.8= .0, in a few months.<br> >=C2=A0 =C2=A0 =C2=A0Also added a FAQ on the possible way to manage a tr= uncated backup, ---<br> >=C2=A0 =C2=A0 =C2=A0OK this is not a Frequently Asked question as nobod= y ever asked on that<br> >=C2=A0 =C2=A0 =C2=A0feature :) ---- but that was the easiest place to a= dd this short<br> >=C2=A0 =C2=A0 =C2=A0documentation.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 ><br> >=C2=A0 =C2=A0 =C2=A0 > RE: archive truncation, another=C2=A0thing th= at I've come across is the<br> >=C2=A0 =C2=A0 =C2=A0 > behaviour of dar when sent the SIGTERM (or si= milar) signals.<br> >=C2=A0 =C2=A0 =C2=A0Normally<br> >=C2=A0 =C2=A0 =C2=A0 > this leads to delayed termination including a= ppending the catalog<br> >=C2=A0 =C2=A0 =C2=A0to the<br> >=C2=A0 =C2=A0 =C2=A0 > end of the archive; however, this is not the = case if dar is sent the<br> >=C2=A0 =C2=A0 =C2=A0 > SIGTERM signal while paused between slices=C2= =A0dar continues to wait for<br> >=C2=A0 =C2=A0 =C2=A0 > user input. Would it be possible to override = this so that if dar<br> >=C2=A0 =C2=A0 =C2=A0is sent<br> >=C2=A0 =C2=A0 =C2=A0 > a SIGTERM signal while waiting for user input= it instead<br> >=C2=A0 =C2=A0 =C2=A0continues with<br> >=C2=A0 =C2=A0 =C2=A0 > the intended delayed termination so that the = archive doesn't end<br> >=C2=A0 =C2=A0 =C2=A0up in a<br> >=C2=A0 =C2=A0 =C2=A0 > truncated state (e.g. if the process is kille= d due to non-<br> >=C2=A0 =C2=A0 =C2=A0response to<br> >=C2=A0 =C2=A0 =C2=A0 > the SIGTERM signal)? Also worth noting that t= he same behaviour<br> >=C2=A0 =C2=A0 =C2=A0occurs<br> >=C2=A0 =C2=A0 =C2=A0 > when dar is interrupted while waiting for par= 2 to finish running<br> >=C2=A0 =C2=A0 =C2=A0between<br> >=C2=A0 =C2=A0 =C2=A0 > slices.<br> > <br> >=C2=A0 =C2=A0 =C2=A0AFAIR, the process is suspending on a system call w= hen it waits for<br> >=C2=A0 =C2=A0 =C2=A0user<br> >=C2=A0 =C2=A0 =C2=A0to answer... To address this type of use case, wher= e you want to<br> >=C2=A0 =C2=A0 =C2=A0automate a process, there is the -Q option that has= been added. Maybe<br> >=C2=A0 =C2=A0 =C2=A0this will address you need better than having the d= ar questions to<br> >=C2=A0 =C2=A0 =C2=A0answer manually.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 ><br> >=C2=A0 =C2=A0 =C2=A0 > The backup I am trying to create (100s of Tb)= is unfortunately<br> >=C2=A0 =C2=A0 =C2=A0going to<br> >=C2=A0 =C2=A0 =C2=A0 > take quite a long time to run, longer than th= e routinely scheduled<br> >=C2=A0 =C2=A0 =C2=A0 > reboot periods of the cluster I am working on= , so am trying to code<br> >=C2=A0 =C2=A0 =C2=A0 > around the scenario where a reboot signal is = sent while par2 is<br> >=C2=A0 =C2=A0 =C2=A0running<br> >=C2=A0 =C2=A0 =C2=A0 > on a slice or during a period when dar2 is ot= herwise paused (which I<br> >=C2=A0 =C2=A0 =C2=A0 > have been asked to schedule in every few TBs)= .<br> > <br> >=C2=A0 =C2=A0 =C2=A0You could also split the data to backup in several = smaller backups (one<br> >=C2=A0 =C2=A0 =C2=A0per top directory or something in that taste that m= akes sense) and<br> >=C2=A0 =C2=A0 =C2=A0gather then into a single dar_manager database. You= can then choose<br> >=C2=A0 =C2=A0 =C2=A0different rotation frequency and retention period b= etween these smaller<br> >=C2=A0 =C2=A0 =C2=A0backups and have a single point of access through d= ar_manager.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 ><br> >=C2=A0 =C2=A0 =C2=A0 > Best,<br> >=C2=A0 =C2=A0 =C2=A0 ><br> >=C2=A0 =C2=A0 =C2=A0 > Scott<br> > <br> >=C2=A0 =C2=A0 =C2=A0Cheers,<br> >=C2=A0 =C2=A0 =C2=A0Denis<br> > <br> > <br> <br> </blockquote></div> --0000000000008d87f706304de17f-- --===============6512515770673628571== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============6512515770673628571==--