Re: dar_manager - incremental backups and removed file

Denis Corbin <[email protected]> Sun, 14 Apr 2024 21:01:16 +0200
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>
I review code and did a test to double check, but could not find any 
problem:

dar_manager -B base.dmd -f bash
Warning: using insecure memory!
         1       Sun Apr 14 20:21:42 2024  saved           absent
         2       Sun Apr 14 20:21:42 2024  present         absent
         3       Sun Apr 14 20:21:42 2024  present         absent
         4       Sun Apr 14 20:23:37 2024  removed         absent

No warning showed. I just realize that dates are evaluated a the 
nanosecond accuracy, but the display has only the second as precision, 
thus this may be the reason of the warning while you see no difference.
Unfortunately there is no way to get this information except restoring 
files.

For the archive #1 #2 and #3 they should have the exact same date, else 
dar would not have resaved the files as the modification date (which is 
shown here) is taken into account, amount other things, to decide to 
resave the file's data or not (here "present" means mtime did not changed).

The last archive is the removal of the file, in your case this is quite 
weird to have the exact same date (compare with the above list). Here 
why I find this weird:

The removal date is taken from the last modification date of the parent 
directory, which should be updated when the directory content is 
modified (new entry or removed entry as here). In summary the following 
events took place:
  - the parent directory was created at time T0
  - the file was created at time T1,
  - you complete the three backups (#1, #2 and #3) and completed backup 
#3 at time T2. Making three backups in a row should take more than one 
second even in the worse case where you create them right after the 
previous one (which should be quite strange idea, by the way).
  - the file is removed at T3
  - you create the last backup at T4 which records the entry as deleted
    and use as date the mtime of the parent directory which should be at 
least T3.

we have TO < T1 < T2 < T3 and even T1 + 1 < T2

the removal date for #4 (T3) should thus be more recent than for #1, #2 
and #3 (T1). But as this is not the case, I *suspect* that directory 
mtime to not be modified after directory creation, which date (T0) is of 
course older than the date of #1, #2 and #3 (T1).  In this scenario, if 
for example the data was restored, the parent directory and its content 
could be created in very short time by dar, having an equivalent visible 
data (less than a second of difference: T1 - T0 < 1) but still different 
(T0 < T1), where from the error reported by dar_manager as this would 
result in this in the dar_manager database for that file:
#1 T1
#2 T1
#3 T1
#4 T0

what filesystem is used? Which option have been used to mount it?

Cheers,
Denis


On 14/04/2024 19:44, Denis Corbin wrote:
> On 14/04/2024 18:32, Joost Roeleveld via Dar-support wrote:
>> Hi all,
> 
> Hi Joost,
> 
>>
>> I am running incremental backups and also keeping dar_manager 
>> databases up-to-date by automatically removing/adding entries.
>>
>> Recently, I got an issue where, when files are deleted between 2 
>> backups, it complains about the order not being correct. (I name the 
>> archives based on the timestamp of when they are created, so am 100% 
>> certain the order should be correct.
>>
>> The messages I see:
>>
>>
>> $ dar_manager -B 
>> /store/backups/san1/cat/zdata_os_services_binhost_root.dmd -c
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/OTF/.uuid
>> Dates are not increasing for all files when database's archive number 
>> grows, working with this database may lead to improper file's restored 
>> version. Please reorder the archive within the database in the way 
>> that the older is the first archive and so on up to the most recent 
>> archive being the last of the database
>> Do you want to ignore the same type of error for other files? [return 
>> = YES | Esc = NO]
>> Escaping...
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/hack/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/urw-fonts/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/misc/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/encodings/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/encodings/large/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/dejavu/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/Type1/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/cyrillic/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/liberation-fonts/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/libertine/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/kanjistrokeorders/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/75dpi/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/noto/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/util/.uuid
>> Dates of file's data are not increasing when database's archive number 
>> grows. Concerned file is: usr/share/fonts/100dpi/.uuid
>> Error met while processing operation: Some files do not follow 
>> chronological order when archive index increases withing the database, 
>> this can lead dar_manager to restored a wrong version of these files
>> Some files do not follow chronological order when archive index 
>> increases withing the database, this can lead dar_manager to restored 
>> a wrong version of these files
>>
>> admin@mngrnl ~ $ dar_manager -B zdata_os_services_binhost_root.dmd -f 
>> usr/share/fonts/liberation-fonts/.uuid
>>          1       Thu Jun  3 13:32:10 2021  saved            absent
>>          2       Thu Jun  3 13:32:10 2021  present            absent
>>          3       Thu Jun  3 13:32:10 2021  present            absent
>>          4       Thu Jun  3 13:32:10 2021  removed            absent
>>
>> [[[[ This result is the same for any of the other .uuid files ]]]]
> 
> we should check all the dates, not only mtime, but also used ctime for 
> EA and FSA, which does not show here, do you confirm that there is just 
> one column of date displayed for the previous dar_manager output?
> 
>>
>> $ dar_manager -B 
>> /store/backups/san1/cat/zdata_os_services_binhost_root.dmd -l
>>
>> dar path         :
>> dar options      :
>> database version : 6
>> compression used : gzip
>> compression level: 9
>>
>> archive #   |    path      |    basename
>> ------------+--------------+---------------
>>          1       . 
>> MNGR_20240301T000001_san1__zdata_os_services_binhost_root
>>          2       . 
>> MNGR_20240401T000002_san1__zdata_os_services_binhost_root
>>          3       . 
>> MNGR_20240407T000001_san1__zdata_os_services_binhost_root
>>          4       . 
>> MNGR_20240414T000001_san1__zdata_os_services_binhost_root
>>
>>
>> Any suggestions on how to resolve this?
> 
> this is weird, I will check that, but if you have provided the backup in 
> order, then you can safely ignore these warnings (see -ai option for 
> dar_manager to not show them at all).
> 
>>
>> Many thanks,
>>
>> Joost
>>
>>
> 
> Cheers,
> Denis
>
OpenPGP_signature.asc (application/pgp-signature, 840 B)