Re: Dar iterating between "random" and last slice in diff mode

Denis Corbin <[email protected]> Fri, 2 Feb 2024 22:35:38 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>
On 01/02/2024 20:04, Mihai Moldovan wrote:
> Hi

Hi

> 
> 
> Backup setup: ~11 TiB of data, 99.5 GiB per slice, 72 slices per disk, no
> encryption, no compression, split onto two 8 TB disks
> 
> dar 2.6.13, static + infinitint version, based on Debian Unstable (sorry, I
> forgot to update dar before starting the backup, so I'm now using an old version
> from three years ago :/)
> 
[...]

> 
> Luckily, I keep a log of all of dar's output, so I checked the creation process
> logs and it looks like the files it's currently processing have been added at
> the end of the creation process. Unfortunately, it also looks like I understand
> what's going on:

Correct! I was perplex reading the symptoms you described, but your 
analysis perfectly makes sense.

> 
> [...]
> Recording hard link into the archive:
> /mnt/rootfs/root/.debug/lib64/modules/5.5.0-OSS4.2/extra/mISDN_core.ko.xz/a3c824ac20ad8df2a801a945675fbbc612cfb1f2/elf
> [...]
> Recording hard link into the archive:
> /mnt/rootfs/root/.debug/lib64/modules/5.5.0-OSS4.2/extra/mISDN_dsp_oslec.ko.xz/c84fcb4e2d1dfbe3c49ac0beeeb2007127196210/elf
> 
> 
> The affected files are hard links to other places! This means that dar's
> behavior is actually intended and not a bug - good.
> 
> The man page says:
> 
> "Doing a difference in sequential read mode is possible but hard linked inodes
> can only be compared to the filesystem the first time they are met, next hard
> links to this same inode cannot obtain the corresponding data because skipping
> backward in sequential read mode is forbidden. In that situation, the hard links
> are reported as skipped, meaning that data comparison could not be performed."
> 
> 
> This makes sense, in general.
> 
> I could work around my issue by executing the comparison (-d) mode in
> --sequential-read mode, but that would be less efficient and is unnecessary,
> because dar is free to actually switch between slices as long as it's only using
> data of one -p segment (i.e., in my case slices {1..72} and {73..117}).
> 
> However, wouldn't it be useful to also provide the hard link skipping feature as
> an option even in non-sequential read mode? 

Well it could be added but only as an option:

- archive testing objective is to validate the archive structure, data 
and internal coherence. When it comes to read a hard-linked inode (which 
is stored only once), the data's CRC is checked and for any further 
reference to this data (corresponding to hard links on the filesystem) 
there is almost nothing to do (just check the reference exist).

- archive comparison objective has a different target, as you know. You 
can restore a backup on a different filesystem partition. Assuming the 
original data was on a single partition and now /usr/share and 
/usr/local are mount points of two different partitions. Suppose 
/usr/share/example.txt and /usr/local/example.txt were hard links to the 
same inode. Restoring on this new partition layout will work, but a hard 
link will not be possible to setup between /usr/local/example.txt and 
/usr/share/example.txt as these paths point to two different 
filesystems/partitions. In that case dar duplicates the inodes (issuing 
a warning).

You can then still compare the backup with what you just restored and 
dar will compare the content of each occurrence of what was originally 
hard links which is what most users will expect. Just checking whether 
the hard link is properly setup will lead comparison to fail while 
restoration suceeded. This to explain why I wrote above: "but only as an 
option".

- at restoration time, for each inode found linked multiple time to the 
directory tree (what is commonly called "a" hard link), dar keep a 
mapping between an internal inode identifier and the first path it has 
been restored to. If a second occurrence of this same inode (same 
internal reference) is to be restored, dar blindly tries to restore the 
inode by creating a hard link to the first place it has been 
linked/restored. If that fails, it restore the inode content as a normal 
plain file.



> The code is there already (more or
> less, not sure about point 3), so all that is needed is another option. Here's
> my reasoning:

All that previous explanations (sorry, I'm still too talkative), to 
explain that the code which you think to be present in dar is not 
present at all in fact:

> 
>    1.) The actual data has already been tested once (the first time the file has
>        been tested).
>    2.) Subsequent hard links only point to already tested data, so they can't be
>        corrupt if the originally tested data tested successfully (i.e., if it
>        wasn't corrupt).

this is wrong, if restoration occurred across two partitions as 
explained above.

>    3.) The only thing that dar needs to make sure is that additional hard links
>        point to the correct inode, and that's easily tested by comparing the
>        current hard link file's inode against the inode it should point to.

unlike soft links, hard links do not "point to" but are just an entry in 
a directory that refer to an inode that is also an entry somewhere else 
(in the same directory under another name or in another directory 
possibly also under another name).

>        Dar
>        only needs to know where the link points to (i.e., file path) 

You cannot know from a first path of an inode linked several times, 
which other paths it has also been linked to in that filesystem. However 
as each inode has an unique inode number in a given filesystem, and as 
each filesystem can be identified uniquely (even if mounted several 
times by mean of bind-mounts) under an Linux operating system, you can 
scan the whole mounted filesystems for the same (inode number, 
filesystem id) couple.

>        on the file
>        system you test against and that information should already be part of the
>        archive at the very point where the file was saved to, so no previous data
>        is necessary.

So in short, the feature you ask is possible, but to know whether a 
directory entry points to the same inode as another directory entry 
(which creates what is commonly designated as a hard link) cannot be 
done based on paths, but on a the couple of (inode number, filesystem id):
  from #include <sys/stat.h>
  this is:
  - struct stat.st_ino
  - struct stat.st_dev

The algorithm is thus a bit different from what you propose.

> 
> 
> Or, in pseudo code:
> 
> file_a: regular;
> file_b: hardlink (file_a);
> 
> Example for file_b:
> 
> if (file_b.is_hardlink ()) {
>    /*
>     * Only check if the metadata is correct, file content must be the same as
>     * for file_a.
>     */
>    if (inode (file_b) == inode (file_b.hardlink_pathto ())) {

here you cannot compare on-fly, you need to record a mapping between the 
dar archive internal inode reference when met the first time, and the 
(stat.st_ino, stat.st_dev) information couple of the corresponding inode 
found at the expected filesystem path for this first time.

Then for each subsequent time this inode has to be compared, the 
operation to perform is to fetch from the mapping the coupe information 
associated to the dar archive internal inode number and to compare it 
withe couple information of the existing inode on filesystem located at 
the path where the hard linked inode should be restore/compared.

>      /*
>       * We know that file_b points to file_a in this example, but that's
>       * not generally true of course.
>       */
>      assert (file_a.path () == file_b.hardlink_pathto ());
> 
>      return (success);
>    }
>    else {
>      /*
>       * inodes of file_b and the file it points to are different - metadata
>       * corruption!
>       */
>      return (error);
>    }
> }
> else {
>    return (check_data (file_b));
> }
> 
> 
> Does that make sense to you?

Yep no worries. And sorry to be talkative... as usual.
I'll add this as feature request.

> 
> 
> 
> Mihai
> 
> 

Denis
OpenPGP_signature.asc (application/pgp-signature, 840 B)