Re: code 11: files changed during backup

Lasse Kliemann wrote:
> Hello,
> 
> * Denis Corbin writes:
> 
>>Lasse Kliemann wrote:
> 
>  
> 
>>>I have got two suggestions how to deal with the situation when a file changes 
>>>while being backed up with dar -- I mean the cases where code 11 is returned. 
>>>There are two approaches that could both be implemented at the same time:
>>>
>>>1. If dar encounters a changing file during backup, dar could re-try to 
>>>backup this file several times, until it succeeds in getting a clean copy of 
>>>it. The number of re-tries (before giving up on that file) could be set using 
>>>a command line option.
>>
>>The problem with this first point is that when creating an archive dar
>>cannot skip back to overwrite data (dar must transparently handle the
>>case where the archive is output to standard output or named pipe, two
>>devices that do not support seeking). Thus this first point is not
>>possible to implement.
> 
> 
> I had not thought about that. However, there is a variation of my idea that 
> can still be realized. As suggested below, a file that could not be copied 
> into the archive cleanly shall be marked as 'unclean'. As you explained, we 
> cannot remove an unclean file from the archive. Still, we could try to add a 
> clean copy of the file _in_ _addition_ to an unclean one. One might end up 
> with several unclean copies of a file in the archive and, if one is lucky, a 
> clean one also. This will make the archive larger, but in many cases this is 
> acceptable. 

I agree, this is acceptable that dar appends several copies of a
'unclean' files up to the time it can:
- either properly save the file (which is no more recorded as 'unclean')
- reach the maximum retry count specified by the user.

This feature should not be activated by default (a command line option
is required for that), file are still marked as 'unclean' but no retry
is performed by default.

At restoration, unclean files are restored with a warning (for backward
compatibility this is better than not restoring them at all).
If a file has been saved several times (may the last copy be also an
unclean or a clean version of this file) only the last occurence is
"visible" and accessible to the user. The other copies (thus except the
last one) of a given file (clean or not) are just garbage in the archive.

For testing or diffing, only the last copy of an unclean file is
considered (the diffing will most probably fail on such unclean files as
they were changing at the time of the backup).

With theses restrictions, it seems possible to have this feature in dar.

> 
> I use dar to backup my system automatically in the background every 3 hours. 
> The archive is compressed and pushed over the network to a machine with 
> several hundreds of gigabytes of space. It really does not matter if the 
> archive grows by a few megabytes or even more.

I can understand this. :-)

> 
> It should, of course, be user-configurable, whether retries should take place 
> and how many of them at most per file. Maybe one could also allow the user to 
> make the number of retries dependent on the size of the file.

maybe it would be better to have a user definable amount of byte that
can be afforded to be wasted by file or by archive. If this amount is
not exhausted, a new try could take place. In addition the retry per
file could also be used.

> So, for 
> example, the user could say that for files above a certain size, no retries 
> should be made. One could also allow the user to limit the total size that is 
> spent on retries.

Note that a file can be detected as 'unclean' only once it has been
saved completely.

> There are a lot of possibilities. For my purposes, it would 
> be enough if I could say: 'give every file at most n retries and do not retry 
> on files larger than x bytes.'
>

Of course, it would not be much more complicated to have user definable
"and" and "or" operation between different type of criterium, like what
is done for EA or file selection. So far here I see several atomic tests:
- max number of retry per file
- max amount of already wasted bytes by unclean files in the archive
- max amount of byte to waste by unclean data for each file

Theses could be "ANDed" or "ORed" to make something more complex like:
"(retry file at most n times) and ((do not waste more than 20MB for
unclean files) or (do not waste more than 500KB per file))"

thus an unclean file of 1GB could be tried at most n times if there is
not yet more than 20MB of unclean data in the archive so far. While
smaller files than 500KB could be tried at least n times even if already
more than 20MB of unclean data is present in the archive so far.

> 
> 
> 
> 
>>>2. If dar fails to integrate a file cleanly into the archive (maybe even 
>>>though dar tried several times to do so, but the file keeps changing 
>>>rapidly), this file could be marked as 'unclean' (or such) in the archive. On 
>>>extraction, the user could decide whether such files should be extracted or 
>>>not.
>>
>>This second point instead is possible to implement and is interesting.
>>This let the user keep trace of probably not well saved files. For
>>backward compatibility, the 'unclean' marking can be systematic at
>>archive creation time, but 'unclean' file should be be filtered out
>>unless the user say so using a given option. Or maybe better the
>>'unclean' files could be filtered out by default (during file
>>extraction) but a warning/aknowledgment would be issued first before
>>starting file extraction if such file has to be filtered out.
> 
> 
> I suggest implementing two switches, say A and B (you will know better what 
> letters to use)

:-) I miss letters in the alphabet! :-)

. If none of them is given, the user will be prompted when the
> extraction process encounters an unclean file. 

I have commented this point in this reply some lines above. I wish to
keep the default behavior that let extracting (with a warning) unclean
files unless the user asks not to extract them.

> If A is given, all unclean
> files are skipped automatically (maybe a warning is displayed).

not warning is necessary, as this is a user explicit choice ('A'
switch). If verbosity is asked (-v), an informational message could be
displayed however.

> If B is 
> given, all files (also the unclean ones, without prompting) are extracted 
> (maybe with a warning). If -Q is given, I suggest that dar should behave as
> if switch A was given.
>
> 
> 
> 
> 
> All the best, Lasse
> 
> 
>

Regards,
Denis.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
Dar-discussions mailing list
Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/dar-discussions

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFF+GJEpC5CI8gYGlIRApPlAJ0YblbXiMX/PpRIuU95KtnBhF9+tQCeLZES
DZVPqlxYurPRK1xUISE9Gqo=
=3aBq
-----END PGP SIGNATURE-----