Delta patch not working on incremental restores

Minh Van Le <mvanle-m/[email protected]> Sun, 06 Aug 2023 00:40:03 +1000
Newsgroups gmane.comp.sysutils.backup.dar.support
Message-ID <[email protected]> 
Hello dar-support, 

Delta patches do not appear to restore for incremental backups.

I have created 3 different files with the same filename ("abc") and
performed a full, differential and incremental backup of each file
(each time using the previous backup as reference):

        dar.exe -N -va --delta sig -c dar1 -g abc
        dar.exe -N -va --delta sig -c dar2 -g abc -A dar1
        dar.exe -N -va --delta sig -c dar3 -g abc -A dar2

When restoring the backups with:

        dar.exe -N -va -x dar1 -w
        dar.exe -N -va -x dar2 -w
        dar.exe -N -va -x dar3 -w

dar1 and dar2 restore as expected. However dar3 gives "Error while
restoring : File the patch is about to be applied to is not the
expected one, aborting the patch operation".

The dar man page says:

        Dar adds a second level of checksum, to detect data corruption
inside the archive 
        and to check that the file the delta patch is about to be
applied is the expected 
        base file, this reduces the risk of "collision" but does not
remove it completely.

I am wondering whether this collision detection is interfering with
restores ?

Please see the attached test script and logfile ("bash -x" output) for
reference.
testDeltaSigOnIncremental.sh (application/x-shellscript, 2.1 KB) 
#!env bash

#/********************************************************************************
# * Constants
# ********************************************************************************/

dar=/c/home/programs/dar64-2.7.10-win64/dar.exe

testFile=abc
basename=dar

#/********************************************************************************
# * Versions
# ********************************************************************************/

bash --version

$dar -V

#/********************************************************************************
# * Pre-execution
# ********************************************************************************/

rm -fv $testFile $testFile? $basename?.*

#/********************************************************************************
# * Backup
# ********************************************************************************/

backup() {
    referenceArchive=$1

	local basename=$basename$((++i))

    dd if=/dev/urandom of=$testFile bs=1M count=1
    $dar -N -va --delta sig -c $basename -g $testFile $referenceArchive
	$dar -l $basename
    mv -fv $testFile $testFile$((i))
}

#/* Create new testFile and do full backup */
backup

#/* Create new testFile and do differential backup */
backup "-A $basename$i"

#/* Create new testFile and do incremental backup */
backup "-A $basename$i"

#/********************************************************************************
# * Restore
# ********************************************************************************/

i=0

restore() {
    $dar -N -va -x $basename$((++i)) -w
    sha256sum $testFile*
}

#/* testFile should match testFile1 from full backup */
restore

#/* testFile should match testFile2 from differential backup */
restore

#/* testFile should match testFile3 from incremental backup */
restore


#// Expected:
#// 
#//     Restore from incremental backup should match original testFile3.
#// 
#// Actual:
#// 
#//     Get error:
#// 
#//         Error while restoring /cygdrive/c/tmp/2/abc : File the patch is about to be applied to is not the expected one, aborting the patch operation
testDeltaSigOnIncremental.sh.log (text/x-log, 17.6 KB) 
+ dar=/c/home/programs/dar64-2.7.10-win64/dar.exe
+ testFile=abc
+ basename=dar
+ bash --version
GNU bash, version 4.4.19(2)-release (x86_64-pc-msys)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -V
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.

 dar version 2.7.10, Copyright (C) 2002-2023 Denis Corbin
   Long options support         : YES

 Using libdar 6.6.0 built with compilation time options:
   gzip compression (libz)      : YES
   bzip2 compression (libbzip2) : YES
   lzo compression (liblzo2)    : YES
   xz compression (liblzma)     : YES
   zstd compression (libzstd)   : YES
   lz4 compression (liblz4)     : YES
   Strong encryption (libgcrypt): YES
   Public key ciphers (gpgme)   : NO
   Extended Attributes support  : YES
   Large files support (> 2GB)  : YES
   ext2fs NODUMP flag support   : NO
   Integer size used            : 64 bits
   Thread safe support          : YES
   Furtive read mode support    : YES
   Linux ext2/3/4 FSA support   : NO
   Mac OS X HFS+ FSA support    : YES
   Linux statx() support        : NO
   Detected system/CPU endian   : little
   Posix fadvise support        : YES
   Large dir. speed optimi.     : YES
   Timestamp read accuracy      : 1 nanosecond
   Timestamp write accuracy     : 1 nanosecond
   Restores dates of symlinks   : YES
   Multiple threads (libthreads): YES (1.4.0 - barrier using pthread_barrier_t)
   Delta compression (librsync) : YES
   Remote repository (libcurl)  : YES (libcurl/8.0.1 OpenSSL/1.1.1t zlib/1.2.13 brotli/1.0.9 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.10.0 nghttp2/1.53.0 libgsasl/2.2.0)
   argon2 hashing (libargon2)   : YES

 compiled the Jun 25 2023 with GNUC version 11.3.0
 dar is part of the Disk ARchive suite (Release 2.7.10)
 dar comes with ABSOLUTELY NO WARRANTY; for details
 type `dar -W'. This is free software, and you are welcome
 to redistribute it under certain conditions; type `dar -L | more'
 for details.

+ rm -fv abc 'abc?' 'dar?.*'
+ backup
+ referenceArchive=
+ local basename=dar1
+ dd if=/dev/urandom of=abc bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151142 s, 69.4 MB/s
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -N -va --delta sig -c dar1 -g abc
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
Creating low layer: Writing archive into a plain file object...
Adding a new layer on top: Caching layer for better performances...
Writing down the archive header...
Adding a new layer on top: Escape layer to allow sequential reading...
Adding a streamed compression layer
All layers have been created successfully
Building the catalog object...
Processing files for backup...
Furtive read mode requires either root permission and FOWNER capability, falling back to normal filesystem read
Adding file to archive: /cygdrive/c/tmp/2/abc
building delta signature with block size of 2048 bytes
Dumping delta signature structure for saved file: /cygdrive/c/tmp/2/abc
Saving Filesystem Specific Attributes for /cygdrive/c/tmp/2/abc
Skipping file: /cygdrive/c/tmp/2/dar1.1.dar
Skipping file: /cygdrive/c/tmp/2/testDeltaSigOnIncremental.sh
Skipping file: /cygdrive/c/tmp/2/testDeltaSigOnIncremental.sh.log
Writing down archive contents...
Closing the escape layer...
Writing down the first archive terminator...
Writing down archive trailer...
Writing down the second archive terminator...
Closing archive low layer...
Archive is closed.


 --------------------------------------------
 1 inode(s) saved
   including 0 hard link(s) treated
 0 inode(s) changed at the moment of the backup and could not be saved properly
 0 byte(s) have been wasted in the archive to resave changing files
 0 inode(s) with only metadata changed
 0 inode(s) not saved (no inode/file change)
 0 inode(s) failed to be saved (filesystem error)
 3 inode(s) ignored (excluded by filters)
 0 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode(s) considered: 4
 --------------------------------------------
 EA saved for 0 inode(s)
 FSA saved for 1 inode(s)
 --------------------------------------------
Making room in memory (releasing memory used by archive of reference)...
Final memory cleanup...
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -l dar1
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[Saved][D]       [H--][     ][ ]  -rwxrwx---   197608	197121	1 Mio	Sun Aug  6 00:16:13 2023	abc
+ mv -fv abc abc1
'abc' -> 'abc1'
+ backup '-A dar1'
+ referenceArchive='-A dar1'
+ local basename=dar2
+ dd if=/dev/urandom of=abc bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169332 s, 61.9 MB/s
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -N -va --delta sig -c dar2 -g abc -A dar1
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
No user target found on command line
Opening archive dar1 ...
Opening the archive using the multi-slice abstraction layer...
Reading the archive trailer...
Opening construction layer...
Considering cyphering layer...
No cyphering layer opened
Opening escape sequence abstraction layer...
Opening the compression abstraction layer (compression algorithm used is none)...
streamed compression layer open (single threaded)
All layers have been created successfully
Loading catalogue into memory...
Locating archive contents...
Reading archive contents...
Creating low layer: Writing archive into a plain file object...
Adding a new layer on top: Caching layer for better performances...
Writing down the archive header...
Adding a new layer on top: Escape layer to allow sequential reading...
Adding a streamed compression layer
All layers have been created successfully
Building the catalog object...
Processing files for backup...
Furtive read mode requires either root permission and FOWNER capability, falling back to normal filesystem read
Delta saving file to archive: /cygdrive/c/tmp/2/abc
building delta signature with block size of 2048 bytes
Dumping delta signature structure for saved file: /cygdrive/c/tmp/2/abc
Saving Filesystem Specific Attributes for /cygdrive/c/tmp/2/abc
Skipping file: /cygdrive/c/tmp/2/abc1
Skipping file: /cygdrive/c/tmp/2/dar1.1.dar
Skipping file: /cygdrive/c/tmp/2/dar2.1.dar
Skipping file: /cygdrive/c/tmp/2/testDeltaSigOnIncremental.sh
Skipping file: /cygdrive/c/tmp/2/testDeltaSigOnIncremental.sh.log
Adding reference to files that have been destroyed since reference backup...
Writing down archive contents...
Closing the escape layer...
Writing down the first archive terminator...
Writing down archive trailer...
Writing down the second archive terminator...
Closing archive low layer...
Archive is closed.


 --------------------------------------------
 1 inode(s) saved
   including 0 hard link(s) treated
 0 inode(s) changed at the moment of the backup and could not be saved properly
 0 byte(s) have been wasted in the archive to resave changing files
 0 inode(s) with only metadata changed
 0 inode(s) not saved (no inode/file change)
 0 inode(s) failed to be saved (filesystem error)
 5 inode(s) ignored (excluded by filters)
 0 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode(s) considered: 6
 --------------------------------------------
 EA saved for 0 inode(s)
 FSA saved for 1 inode(s)
 --------------------------------------------
Making room in memory (releasing memory used by archive of reference)...
Final memory cleanup...
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -l dar2
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[Delta][D]       [H--][Worse][ ]  -rwxrwx---   197608	197121	1 Mio	Sun Aug  6 00:16:14 2023	abc
+ mv -fv abc abc2
'abc' -> 'abc2'
+ backup '-A dar2'
+ referenceArchive='-A dar2'
+ local basename=dar3
+ dd if=/dev/urandom of=abc bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153115 s, 68.5 MB/s
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -N -va --delta sig -c dar3 -g abc -A dar2
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
No user target found on command line
Opening archive dar2 ...
Opening the archive using the multi-slice abstraction layer...
Reading the archive trailer...
Opening construction layer...
Considering cyphering layer...
No cyphering layer opened
Opening escape sequence abstraction layer...
Opening the compression abstraction layer (compression algorithm used is none)...
streamed compression layer open (single threaded)
All layers have been created successfully
Loading catalogue into memory...
Locating archive contents...
Reading archive contents...
Creating low layer: Writing archive into a plain file object...
Adding a new layer on top: Caching layer for better performances...
Writing down the archive header...
Adding a new layer on top: Escape layer to allow sequential reading...
Adding a streamed compression layer
All layers have been created successfully
Building the catalog object...
Processing files for backup...
Furtive read mode requires either root permission and FOWNER capability, falling back to normal filesystem read
Delta saving file to archive: /cygdrive/c/tmp/2/abc
building delta signature with block size of 2048 bytes
Dumping delta signature structure for saved file: /cygdrive/c/tmp/2/abc
Saving Filesystem Specific Attributes for /cygdrive/c/tmp/2/abc
Skipping file: /cygdrive/c/tmp/2/abc1
Skipping file: /cygdrive/c/tmp/2/abc2
Skipping file: /cygdrive/c/tmp/2/dar1.1.dar
Skipping file: /cygdrive/c/tmp/2/dar2.1.dar
Skipping file: /cygdrive/c/tmp/2/dar3.1.dar
Skipping file: /cygdrive/c/tmp/2/testDeltaSigOnIncremental.sh
Skipping file: /cygdrive/c/tmp/2/testDeltaSigOnIncremental.sh.log
Adding reference to files that have been destroyed since reference backup...
Writing down archive contents...
Closing the escape layer...
Writing down the first archive terminator...
Writing down archive trailer...
Writing down the second archive terminator...
Closing archive low layer...
Archive is closed.


 --------------------------------------------
 1 inode(s) saved
   including 0 hard link(s) treated
 0 inode(s) changed at the moment of the backup and could not be saved properly
 0 byte(s) have been wasted in the archive to resave changing files
 0 inode(s) with only metadata changed
 0 inode(s) not saved (no inode/file change)
 0 inode(s) failed to be saved (filesystem error)
 7 inode(s) ignored (excluded by filters)
 0 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inode(s) considered: 8
 --------------------------------------------
 EA saved for 0 inode(s)
 FSA saved for 1 inode(s)
 --------------------------------------------
Making room in memory (releasing memory used by archive of reference)...
Final memory cleanup...
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -l dar3
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[Delta][D]       [H--][Worse][ ]  -rwxrwx---   197608	197121	1 Mio	Sun Aug  6 00:16:14 2023	abc
+ mv -fv abc abc3
'abc' -> 'abc3'
+ i=0
+ restore
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -N -va -x dar1 -w
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
Opening archive dar1 ...
Opening the archive using the multi-slice abstraction layer...
Reading the archive trailer...
Opening construction layer...
Considering cyphering layer...
No cyphering layer opened
Opening escape sequence abstraction layer...
Opening the compression abstraction layer (compression algorithm used is none)...
streamed compression layer open (single threaded)
All layers have been created successfully
Loading catalogue into memory...
Locating archive contents...
Reading archive contents...
Restoring file's data: /cygdrive/c/tmp/2/abc
Restoring file's FSA: /cygdrive/c/tmp/2/abc


 --------------------------------------------
 1 inode(s) restored
    including 0 hard link(s)
 0 inode(s) not restored (not saved in archive)
 0 inode(s) not restored (overwriting policy decision)
 0 inode(s) ignored (excluded by filters)
 0 inode(s) failed to restore (filesystem error)
 0 inode(s) deleted
 --------------------------------------------
 Total number of inode(s) considered: 1
 --------------------------------------------
 EA restored for 0 inode(s)
 FSA restored for 0 inode(s)
 --------------------------------------------
Final memory cleanup...
+ sha256sum abc abc1 abc2 abc3
519976867d35609781b090c386d5f60264e988dea5c4292ce2c7ad61dc7c1f1e *abc
519976867d35609781b090c386d5f60264e988dea5c4292ce2c7ad61dc7c1f1e *abc1
e903386c8765c65f2073dc925adac22f4548fe1811b5094ec3af9d9ce5ebeeb9 *abc2
9c850304908dc7be9c5bcc82775f99781716d77227a09b4f037153cf9ad9774c *abc3
+ restore
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -N -va -x dar2 -w
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
Opening archive dar2 ...
Opening the archive using the multi-slice abstraction layer...
Reading the archive trailer...
Opening construction layer...
Considering cyphering layer...
No cyphering layer opened
Opening escape sequence abstraction layer...
Opening the compression abstraction layer (compression algorithm used is none)...
streamed compression layer open (single threaded)
All layers have been created successfully
Loading catalogue into memory...
Locating archive contents...
Reading archive contents...
Restoring file's data using a delta patching: /cygdrive/c/tmp/2/abc


 --------------------------------------------
 1 inode(s) restored
    including 0 hard link(s)
 0 inode(s) not restored (not saved in archive)
 0 inode(s) not restored (overwriting policy decision)
 0 inode(s) ignored (excluded by filters)
 0 inode(s) failed to restore (filesystem error)
 0 inode(s) deleted
 --------------------------------------------
 Total number of inode(s) considered: 1
 --------------------------------------------
 EA restored for 0 inode(s)
 FSA restored for 1 inode(s)
 --------------------------------------------
Final memory cleanup...
+ sha256sum abc abc1 abc2 abc3
e903386c8765c65f2073dc925adac22f4548fe1811b5094ec3af9d9ce5ebeeb9 *abc
519976867d35609781b090c386d5f60264e988dea5c4292ce2c7ad61dc7c1f1e *abc1
e903386c8765c65f2073dc925adac22f4548fe1811b5094ec3af9d9ce5ebeeb9 *abc2
9c850304908dc7be9c5bcc82775f99781716d77227a09b4f037153cf9ad9774c *abc3
+ restore
+ /c/home/programs/dar64-2.7.10-win64/dar.exe -N -va -x dar3 -w
No terminal found for user interaction. All questions will be assumed a negative answer (less destructive choice), which most of the time will abort the program.
Opening archive dar3 ...
Opening the archive using the multi-slice abstraction layer...
Reading the archive trailer...
Opening construction layer...
Considering cyphering layer...
No cyphering layer opened
Opening escape sequence abstraction layer...
Opening the compression abstraction layer (compression algorithm used is none)...
streamed compression layer open (single threaded)
All layers have been created successfully
Loading catalogue into memory...
Locating archive contents...
Reading archive contents...
Restoring file's data using a delta patching: /cygdrive/c/tmp/2/abc
Error while restoring /cygdrive/c/tmp/2/abc : File the patch is about to be applied to is not the expected one, aborting the patch operation


 --------------------------------------------
 0 inode(s) restored
    including 0 hard link(s)
 0 inode(s) not restored (not saved in archive)
 0 inode(s) not restored (overwriting policy decision)
 0 inode(s) ignored (excluded by filters)
 1 inode(s) failed to restore (filesystem error)
 0 inode(s) deleted
 --------------------------------------------
 Total number of inode(s) considered: 1
 --------------------------------------------
 EA restored for 0 inode(s)
 FSA restored for 0 inode(s)
 --------------------------------------------
Final memory cleanup...
All files asked could not be restored
+ sha256sum abc abc1 abc2 abc3
e903386c8765c65f2073dc925adac22f4548fe1811b5094ec3af9d9ce5ebeeb9 *abc
519976867d35609781b090c386d5f60264e988dea5c4292ce2c7ad61dc7c1f1e *abc1
e903386c8765c65f2073dc925adac22f4548fe1811b5094ec3af9d9ce5ebeeb9 *abc2
9c850304908dc7be9c5bcc82775f99781716d77227a09b4f037153cf9ad9774c *abc3