Re: MAYBE BUG in dar 2.7.2 with hardlinks handling

On 03/12/2021 22:04, Alexey Loukianov via Dar-support wrote:
> On 03.12.2021 23:25, Denis Corbin wrote:
>> 2/ Now, you have guessed, when the first subdirectory returned by the
>> operating system is the new one ...
> 
>> The order of entries in a directory is no linked to the lexicographical
>> order of their name. It is completely filesystem dependent.
> 
> Gosh, silly me! These were "the missing pieces" I was failing to think
> about when been trying to get a grasp on what's going on here. I was
> even thinking about creating "-O0" build of dar tomorrow evening to give
> it a gdb treatment accompanied by a pint of beer :-). You literally
> saved me from doing that so Saturday's evening would be more of a
> traditional "reading Phoronix" style.

Well, I'm sorry, I just hope you could find another argument to drink
your pint of beer ;)

> 
>> Improving this behavior is quite tricky...
> 
> TBH now when I understand what's going on I don't think that this
> behaviour is really "in needs" to be improved. There's another possible
> approach here though. If it would be possible to pass the explicit list
> of files to be backed up (with names relative to the specified -R) to
> dar then it would be somewhat easier to have it optimized like this:

I'm not sure to follow what you plan to do, but I guess it is possible:

> 
> 1. Read the file list from the reference archive and match it with the
> passed list of files "to be archived". Include all matched files into
> archive in a usual manner, i.e. save or delta-diff-save files that had
> their content changed, do not "save" data for all unchanged. For files
> that are in the archive of reference but are not in the list to archive
> - either mark them as "deleted" or don't do anything about them at all
> depending on some control flag passed by the user to control this.

--exclude-from-file/--include-from-file option can be used to instruct
dar of a list of file to exclude or include in the backup
(full/incremental/differential/decremental). It can cohesist with -g
options as you were using. See also -am option for even more flexibility
about the filtering mechanism

> 
> 2. Do a second pass over the list of files to backup but this time only
> taking into account those which were not found in the archive of
> reference. At this time we would have all the required information on
> hands to be able to find and properly (i.e _not_store_) handle all
> hardlinks.

not sure to understand what you want to do here, but I guess you could
also use here --exclude-from-file/--include-from-file

> 
> I'm not that familiar with the implementation details of the dar so
> can't judge if:
> a) the approach above aligns well with the archive file format (i.e. if
> it is mandatory or not to have all files stored in the specific
> subdirectory to be places inside the archive right after the data
> structure/entry for that directory);
> b) whether it is easy to bypass filesystem traversing code and replace
> it with a feeder coming from the user-specified list;
> 
> But in general such approach feels like something that might be possible
> and not _that_hard_ to implement.

it completely depends on the implementation, and for what you describe i
don't thing it to be not related to the internal structure of the backup
file.

> 
>> But note that this behavior is not new, I've tested 2.7.2 and 2.6.15:
>> they behave the same.
> Yep, can confirm it too. I had started my experiments with "new
> approach" using 2.6.8 and hitting "unexpected behaviour" was yet another
> reason for me to build fresh libthreadar and dar RPMs for CentOS 7.
> 
> 
>> This is my turn to be sorry for this long explanation! :)
> 
> No need, it was a really great explanation done in a form that made it
> clear to me what is happening and why. What a great teaching skills
> you've got, congrats!
> 

Thanks!

I tried playing with the merging operation
- making full backup of N and N-1 at each step
        dar -c tmp-N -g N -g N-1 ...
- then merging with previous backup to passing the entries found in N-1
as "already saved"
	dar -+ backup-N -A tmp-N -@ backup-N-1 -/ Ss

If the new backup do not contain the old hardlinked inodes, at
restoration time, the hard links may not be restored (the hard link is
missing in the restored directory): this is because the order in which
the directory content was returned by the operating system is kept in
the backup...

so I'm not sure it would match your need

Cheers,
Denis