Re: Dar iterating between "random" and last slice in diff mode

Denis Corbin <[email protected]> Mon, 5 Feb 2024 21:17:02 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>

On 04/02/2024 19:59, Mihai Moldovan wrote:

[...]
> > Wouldn't it be generally beneficial to have dar check hard links in 
this way
> first, and only fallback to content checking if the inode/dev id couples on the
> target system do not match (and if not in sequential read mode, of course)?

Yes, that's a good idea, and this will also avoid the addition of yet 
another option :)

> That
> would save a lot of going "back and forth" in non-sequential read mode and has
> the benefit of supporting basic hard link checking even in sequential read mode,
> since you don't have to access earlier data.

totally correct, your proposal is adopted :)

> 
> The only issue is that I don't know what the internal reference dar saves
> contains.

this is just an integer (called an "etiquette" which means "label" in 
French).

> If it's essentially just a pointer to another place in the catalogue
> and additional information must be loaded by reading from the referenced slice,

When building the catalog in memory during the backup process, there is 
of course pointers to inodes (libdar's own smart pointers structure that 
existed long before the std::shared_ptr now commonly used smart pointer 
in C++).

But at the end of the backup, when it comes to drop the catalog to file, 
an unique integer (an "étiquette" thus) is assigned to the multiple time 
linked inode and a small data structure containing that integer is 
written to file before that inode metadata (inode data has been written 
earlier during the backup process). When a hard link to this inode is 
found again in the in-memory catalog, a small data structure also 
containing the "etiquette" number associated with that inode is written 
in turn to the backup file...

When comes the time to read the backup (restoration, archive testing, 
comparison...), the process first fetch the catalog and rebuilds the 
smart pointers from the integer numbers found in the on-file catalog 
data structures.

in sequential-read mode, the process is similar (mostly identical), 
except that the file metadata and data are interleaved.

> we're not winning anything.

almost all needed info about the inode's metadata is loaded in memory 
the first time an multiple linked inode is found while reading a dar 
backup. Subsequent link to this inode gives an integer that is mapped to 
the in-memory inode metadata information... What is missing is just the 
path to this first link to the inode which is not stored in the catalog 
(not that way).

But no worries, here the same process (and probably even the same code) 
as the one used for restoration, can be used to map each "etiquette" to 
a set of  { inode object, path of first occurrence,  and eventually also 
adding the the inode_id, and filesystem_id } to have everything to make 
the hard link comparison possible without requiring any additional read 
from backup file.

So your idea stays valid :) ... and is adopted :)

> 
> However, if the reference also contains the first file's full path within the
> archive, we'd have all information that we need for checking a hard link without
> needing to look up any additional information. If it currently doesn't contain
> the first file's full path, maybe that can be added in newer data layout
> formats? I realize that this would duplicate this kind of information, but it's
> probably worth the overhead, since we're gaining better (and, for big files even
> much faster) hard link handling in non-sequential read mode and new basic hard
> link checking in sequential read mode.
> 
> If dar were to handle hard links this smartly, there's probably no need to add
> an option to completely skip hard link checking in non-sequential read mode. 

yes, I agree

> At
> least I can't think of any situation where that would be helpful. If the layout
> is the same as it were when the backup was created, the inode/dev id couple
> check would make sure that the data is sane and if it the layout is different,
> the content must be checked anyway.

there is no problem here *I guess*: if the filesystem_id/inode_id is 
different (which can results from a pre-existing file or splited hard 
link at restoration time due to filesystem boundary), we fallback to the 
current operation mode (comparing the file content). If it is equal to 
the first occurence found and compared, we avoid comparing data as we 
know the entry points to the expected existing inode.

> 
> 
> 
> Mihai
> 
> 

Cheers,
Denis
OpenPGP_signature.asc (application/pgp-signature, 840 B)