Re: dar_manager - incremental backups and removed file

Denis Corbin <[email protected]> Sun, 21 Apr 2024 21:28:48 +0200
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>
On 20/04/2024 15:23, Joost Roeleveld via Dar-support wrote:
> 
> 
> ----------------------- Original message -----------------------
> From: Denis Corbin <[email protected]>
> To: [email protected]
> Date: Sat, 20 Apr 2024 13:25:59 +0200
> ----------------------------------------------------------------
> 
>> Hi Joost,
>>
>> could you provide the output of 'dar -V'?
> 
> # dar -V
> 
>   dar version 2.7.13, Copyright (C) 2002-2023 Denis Corbin
>   Long options support : YES
> 

[...]

>   Timestamp read accuracy : 1 nanosecond
>   Timestamp write accuracy : 1 nanosecond

this is above the info I wanted to check and this is not what I 
expected, but OK. The problem seen is not caused by the way dar has been 
compiled and seems to read dates from the filesystem the same manner
over the time, I mean it has always read date from the fields of the 
system "struct stat" returned by lstat() system call:
- stat::st_atim.tv_sec
- stat::st_mtim.tv_sec
- stat::st_ctim.tv_sec

as implemented in src/libdar/filesystem_hard_link_read.cpp

[...]
> 
>> Have you recently recompiled or upgraded the dar binary used here to 
>> make backups on that host?

Here is why I ask, I'll take just the usr/share/fonts/hack/.uuid file to 
illustrate:

[Sorry for the ugly formated output... too wide to fit in email]

# dar -l MNGR_20240301T000001_san1__zdata_os_services_binhost_root -g
usr/share/fonts/hack -afdd
Warning: using insecure memory!
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[InRef][-]       [-L-][  70%][ ]  drwxr-xr-x   root     root    5 Gio
Mon Jul  1 11:34:26 2019 + 370501096 ns usr
[InRef][-]       [-L-][  65%][ ]  drwxr-xr-x   root     root    872 Mio
Mon Feb 19 17:56:59 2024 + 297538137 ns usr/share
[InRef][-]       [-L-][  48%][ ]  drwxr-xr-x   root     root    182 Mio
Thu Jun  3 15:32:10 2021 + 211378 us    usr/share/fonts
[InRef][-]       [-L-][   0%][ ]  drwxr-xr-x   root     root    36 o
Thu Jun  3 15:32:10 2021 + 301378 us    usr/share/fonts/hack
[InRef][ ]       [-L-][-----][ ]  -rw-r--r--   root     root    36 o
Thu Jun  3 15:32:10 2021 + 301378785 ns usr/share/fonts/hack/.uuid

 From the above output of the oldest archive, all dates have nanosecond 
precision [the -afdd option has been added to display *F*ully *D*etailed 
*D*ates. This option will be available with release 2.8.0 for dar and 
dar_manager]


# dar -l MNGR_20240401T000002_san1__zdata_os_services_binhost_root -g
usr/share/fonts/hack -afdd
Warning: using insecure memory!
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[InRef][-]       [-L-][  71%][ ]  drwxr-xr-x   root     root    4 Gio
Thu Mar 28 10:57:11 2024 + 379415752 ns usr
[InRef][-]       [-L-][  66%][ ]  drwxr-xr-x   root     root    832 Mio
Thu Mar 28 07:53:02 2024 + 534986022 ns usr/share
[     ][-]       [-l-][  48%][ ]  drwxr-xr-x   root     root    182 Mio
Thu Jun  3 15:32:10 2021 + 211378 us    usr/share/fonts
[     ][-]       [-l-][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021 + 301378 us    usr/share/fonts/hack
[     ][ ]       [-l-][-----][ ]  -rw-r--r--   root     root    36 o
Thu Jun  3 15:32:10 2021 + 301378785 ns usr/share/fonts/hack/.uuid

But for the second backup above, the precision changed to microsecond 
(displayed "us" above) for usr/share/fonts and usr/share/fonts/hack but 
is kept as nonaseconds for usr/share/fonts/.hack and usr.

Reading code on how differential backup behave about dates, 
(src/libdar/filtre.cpp line 790), a cat_inode object is created reading 
information from the filesystem, which mtime is compared with the one of 
the cat_inode taken from the archive of reference. The comparison takes 
into account the possible date precision difference (datetime::loose_diff).
If the date is equal inode created from the filesystem just drops 
attributes to its data but keeps the date and its accuracy from the 
filesystem.

Why microsecnfs for usr/share/fonts  but not for 
usr/share/fonts/hack/.uuid ? For libdar there is not code difference for 
directory and other inodes concerning date handling (same cat_inode is 
the ancestor class that handles all that is common to inodes, in 
particular dates).

Same for the third backup below:

# dar -l MNGR_20240407T000001_san1__zdata_os_services_binhost_root -g
usr/share/fonts/hack -afdd
Warning: using insecure memory!
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
Thu Mar 28 10:57:11 2024 + 379415752 ns usr
[     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
Thu Mar 28 07:53:02 2024 + 534986022 ns usr/share
[     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021 + 211378 us    usr/share/fonts
[     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021 + 301378 us    usr/share/fonts/hack
[     ][ ]       [---][-----][ ]  -rw-r--r--   root     root    36 o
Thu Jun  3 15:32:10 2021 + 301378785 ns usr/share/fonts/hack/.uuid

Finally for the fourth backup, the precision has even dropped one step 
further, to second for usr/share, usr/share/fonts and the deleted entry 
holding place of usr/share/fonts/hack/.uuid which date is read at the 
time of the backup from the mtime value of the parent directory:

# dar -l MNGR_20240414T000001_san1__zdata_os_services_binhost_root -g
usr/share/fonts/hack -afdd
Warning: using insecure memory!
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[     ][-]       [---][  74%][ ]  drwxr-xr-x   root     root    450 Mio
Thu Mar 28 10:57:11 2024 + 379415752 ns usr
[InRef][-]       [-L-][  91%][ ]  drwxr-xr-x   root     root    233 Mio
Mon Apr  8 13:19:02 2024 + 244801698 ns usr/share
[     ][-]       [-L-][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021        usr/share/fonts
[     ][-]       [-L-][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021        usr/share/fonts/hack
[--- REMOVED ENTRY ----] (Thu Jun  3 15:32:10 2021) [-]
usr/share/fonts/hack/.uuid

I tried reproducing this with backup, differential backup, backup 
isolation on an filesystem with mounted with relatime option without 
success... all works as expected on my side.

> 
> It was recompiled recently (on the 10th), with me noticing the issue on 
> the 13th or 14th.
> But the other hosts are backed up using the exact same Dar-version. (Dar 
> runs on the storage host which contains all the VMs. Some VMs are 
> practically identical)
> 
> A backup is basically (in reality more complex due to automation) done 
> as follows:
> - filesystem snapshot is created
> - snapshot is mounted
> - snapshot is backed up using dar
> - snapshot is umounted + removed

So my question is now about the snapshot process and the way you expose 
it to dar.

Could you issue the following command for example in such snapshot:

stat /usr/share/fonts/hack

you should see something like this:

# stat /usr/share/fonts
   File: /usr/share/fonts
   Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 802h/2050d      Inode: 193473      Links: 11
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-04-21 21:08:01.923873113 +0200
Modify: 2020-06-12 20:15:09.546370763 +0200
Change: 2020-06-12 20:15:09.546370763 +0200
  Birth: 2019-04-23 20:32:20.976000000 +0200

In conclusion so far, the explanation of the warning issued by 
dar_manager is caused by a loss of time precision and is not a bug in 
dar_manager, maybe still possible a bug in dar but that is not the most 
probable explanation IMHO.

[...]

> 
> -- 
> Joost
> 
> 

Cheers,
Denis
OpenPGP_signature.asc (application/pgp-signature, 840 B)