Re: Dar iterating between "random" and last slice in diff mode

* On 2/2/24 22:35, Denis Corbin wrote:
> Well it could be added but only as an option:
> 
> - archive testing objective is to validate the archive structure, data 
> and internal coherence. When it comes to read a hard-linked inode [...]
> there is almost nothing to do (just check the reference exist).

Right, and it didn't jump "backwards" in testing mode due to this.


> - archive comparison objective has a different target, as you know. You 
> can restore a backup on a different filesystem partition.

And that's the thing that I didn't have in mind, but it totally makes sense, yes.

Skipping hard link content checking is only possible if the layout of the root
directory you test the archive against has the same file system layout, true.


> [...] In that case dar duplicates the inodes (issuing 
> a warning).
> 
> You can then still compare the backup with what you just restored and 
> dar will compare the content of each occurrence of what was originally 
> hard links which is what most users will expect. Just checking whether 
> the hard link is properly setup will lead comparison to fail while 
> restoration suceeded. This to explain why I wrote above: "but only as an 
> option".

Hm, an option might be good to have, but can't dar handle this in a smarter way?

I'll explain how exactly later.


> - at restoration time, for each inode found linked multiple time to the 
> directory tree (what is commonly called "a" hard link), dar keep a 
> mapping between an internal inode identifier and the first path it has 
> been restored to.

Yes, that makes sense. Dar needs to keep a reference to some other linked file
to handle inodes correctly, and taking the first seen one isn't a bad choice.


> this is wrong, if restoration occurred across two partitions as 
> explained above.

Yep, ignore that part. Or don't! Because it's actually something that one can use.


> unlike soft links, hard links do not "point to" but are just an entry in 
> a directory that refer to an inode that is also an entry somewhere else 
> (in the same directory under another name or in another directory 
> possibly also under another name).

I know that. With "points to", I was really referring to dar's internal data
about the hard link, i.e., the first file in the archive that has been added and
has the same inode - the "internal reference", as you called it.


> [...] under an Linux operating system, you can 
> scan the whole mounted filesystems for the same (inode number, 
> filesystem id) couple.

Yeah, that's the only way to find all hard linked files, but luckily we don't
need this.


> So in short, the feature you ask is possible, but to know whether a 
> directory entry points to the same inode as another directory entry 
> (which creates what is commonly designated as a hard link) cannot be 
> done based on paths, but on a the couple of (inode number, filesystem id):
>   from #include <sys/stat.h>
>   this is:
>   - struct stat.st_ino
>   - struct stat.st_dev
> 
> The algorithm is thus a bit different from what you propose.

That's fine, I didn't want to get into too many details with my pseudo code. The
imaginary inode() function was meant to abstract exactly that. :)


>> Or, in pseudo code:
>>
>> file_a: regular;
>> file_b: hardlink (file_a);
>>
>> Example for file_b:
>>
>> if (file_b.is_hardlink ()) {
>>    /*
>>     * Only check if the metadata is correct, file content must be the same as
>>     * for file_a.
>>     */
>>    if (inode (file_b) == inode (file_b.hardlink_pathto ())) {
> 
> here you cannot compare on-fly, you need to record a mapping between the 
> dar archive internal inode reference when met the first time, and the 
> (stat.st_ino, stat.st_dev) information couple of the corresponding inode 
> found at the expected filesystem path for this first time.
> 
> Then for each subsequent time this inode has to be compared, the 
> operation to perform is to fetch from the mapping the coupe information 
> associated to the dar archive internal inode number and to compare it 
> withe couple information of the existing inode on filesystem located at 
> the path where the hard linked inode should be restore/compared.

Yes, this is exactly what I meant! I merely tried to describe it concisely
without delving too deeply into technical aspects.

Wouldn't it be generally beneficial to have dar check hard links in this way
first, and only fallback to content checking if the inode/dev id couples on the
target system do not match (and if not in sequential read mode, of course)? That
would save a lot of going "back and forth" in non-sequential read mode and has
the benefit of supporting basic hard link checking even in sequential read mode,
since you don't have to access earlier data.

The only issue is that I don't know what the internal reference dar saves
contains. If it's essentially just a pointer to another place in the catalogue
and additional information must be loaded by reading from the referenced slice,
we're not winning anything.

However, if the reference also contains the first file's full path within the
archive, we'd have all information that we need for checking a hard link without
needing to look up any additional information. If it currently doesn't contain
the first file's full path, maybe that can be added in newer data layout
formats? I realize that this would duplicate this kind of information, but it's
probably worth the overhead, since we're gaining better (and, for big files even
much faster) hard link handling in non-sequential read mode and new basic hard
link checking in sequential read mode.

If dar were to handle hard links this smartly, there's probably no need to add
an option to completely skip hard link checking in non-sequential read mode. At
least I can't think of any situation where that would be helpful. If the layout
is the same as it were when the backup was created, the inode/dev id couple
check would make sure that the data is sane and if it the layout is different,
the content must be checked anyway.



Mihai