Re: RSync like diff for incremental stuff

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

Cyril Russo wrote:
> Sterling Windmill a écrit :
>> Have you tried rdiff-backup ?
> Yes
> And, even better, duplicity.
> 
> However, all thoses software still lack a major technical feature.
> rdiff-backup doesn't compress (unless you have gzip-rsyncable) nor 
> encrypt, so it's useless if you don't master/trust the remote host location.
> duplicity compress and encrypt and use rsync, but it uses TAR 
> internally, so browsing a backup is a real pain (as there isn't any 
> catalog in the archive itself it's very very slow).
> Duplicity comes with a real giant list of backend for accessing remote 
> storage (from sshfs, imap, webdav, ftp, S3, tahoe, local, etc...).
> 
> I though about an ultimate backup tool that would use the smartness of 
> DAR (catalog / compression / encryption / incremental), but with a 
> optimal disk consumption (thanks to rsync).
> That way, browsing the backup set would still be very fast.
> 
> I think the unix philosophy of "One tools does one thing well", I would 
> let DAR create a local backup archive on a remote mounted filesystem 
> (like NFS, SSHFS, WebdavFS etc...).
> 
> Anyway, how complex would it require to implement in DAR ?

First, to be able to make a binary diff you must have the original
binary beside the current one to compare them (actually dar relies on
inode data changes to decide whether to save the whole file again or to
not save it at all).

Consequences is that this is not compatible with differential backup
based on catalogue, the feature must thus be an option.

Worse it is will be difficult to base such a backup on a differential
backup. Suppose the file is already binary diffed ... how to know if the
part of the file that is not present in the file has changed? Yes, using
 signatures (or ~ CRC) but still the risk that the file changed and
signature stays the same thus changed would not be saved.

The second point is that at restoration time you must not blindly patch
an existing binary file with the portion of data that have been saved.
The use of signature (CRC for example) done on the original file is
necessary, but will not warranty that the original file is the really
the same as the one over which the binary diff has been made (there is
still a little chance that two different files get the same signature)
and I don't see any mean to be sure that the live file to restore over a
binary diff is exactly the same suite of byte as the one used at backup
time, except than comparing byte by byte, thus requiring any previous
diff backup up to the first where the whole data is present. But now, if
the user is warned about this risk and wants to take it ... I see no
problem.

This was for the functional level.

Seen now the implementation, this should be feasable, but at the cost of
some major change in the way dar stores files. We must now not only
store a suite of bytes to be copied to a brand new file, but a list of
blocks with offset, length and data (an probably checksum). Well, there
is a feature in the pipe that will bring some changes at this level
which is the possibility to properly store sparse files [
https://sourceforge.net/tracker/?func=detail&aid=1457710&group_id=65612&atid=511615
]. I could extend a little bit this feature to let room for CRC fields
and the like (re-using the TLV class).

Will lack the algorithm to inspect binary files and decide which way to
define blocks of changed data. The problem here is that it will cost
much more disk I/O to compare two binaries: It must first be necessary
to inspect the whole files (old backup and live filesystem) to find out
which amount of data has changed and which pattern do changes follow. It
may well be more costly to systematically do a diff in place of a full
backup if a byte every N byte has changed for example. Then, we could
proceed either to full backup or binary diff backup.


Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKNqiJpC5CI8gYGlIRAgPmAJ0SCvF25/l5QWQudF1PpU7kzWxnegCeLywp
JxJqhIVf26F8xf/ss0c5s58=
=8p/S
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing 
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Dar-discussions mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-discussions