Re: --sequential-read not sequential???

Denis Corbin <[email protected]> Tue, 25 Feb 2020 13:54:23 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>

On 24/02/2020 22:55, Graham Cobb wrote:
>>> 1) Using dar_xform and an actual pipe to feed "dar -t". That caused
>>> dar to emit a hard error because it cannot go backwards on a pipe.
>>> So the "dar -t" failed. It didn't emit any message explaining why
>>> it wanted to go backwards.
>>
>> strange, which version of dar are you using by the way?
> 
> 2.6.6

OK so this explains one point, as described on the Changelog:

from 2.6.6 to 2.6.7
[...]
- fixed bug: dar failed creating an archive on its standard output
  reporting the error message "Skipping backward is not possible on a pipe"



> 
>> Just double check: have you used
>> --sequential-read option?
> 
> yes
> 
>>
>> I Must find a way (and time) to reproduce it...
> 
> In case it is helpful, these archives are created and managed by
> DAR_automatic_backup.sh (probably an old version as I have not touched
> it for many years).

it does not matter how dar is ran, the archive format is totally managed
by libdar... here 2.6.6 so there is no issue using an "old" script with dar.

> 
> I have now found an archive with a single slice which reproduces the
> reading problem. This eliminates the problem being anything to do with
> multiple slices, or dar_xform. Unfortunately this archive is 1.3GB and
> contains private data so I cannot make it available to you. With this
> particular archive,

No problem, I have to try reproducing in my side, good to know this
has nothing related with slicing.

> 
> dar -t - --sequential-read </var/tmp/DARsystemDiff06.1.dar
> 
> gives the following output...
> 
> ERR <ROOT>/var/indexes/cobb : Skipping backward is not possible on a pipe
> ERR <ROOT>/var/indexes/cobb/.Trash : Skipping backward is not possible
> on a pipe
> ERR <ROOT>/home/cobb : Skipping backward is not possible on a pipe
> ERR <ROOT>/home/cobb/dovecot : Skipping backward is not possible on a pipe
> ERR <ROOT>/home/cobb/dovecot/cur : Skipping backward is not possible on
> a pipe
> 
>  --------------------------------------------
>  5903243 item(s) treated
>  0 item(s) with error
>  0 item(s) ignored (excluded by filters)
>  --------------------------------------------
>  Total number of items considered: 5903243
>  --------------------------------------------

this is the bug that has been fixe in 2.6.7 ad following versions

> 
>>
>> if you still have the whole dar ouput, can you provide the output of
>> this command:
>>
>> grep "/var/lib/samba/private" <dar output>
> 
> 
> Yes. I will send it as an attachment to a separate message off-list.

Yes, thanks I've received it. My understanding is the following:
the test routing reads archive contents from a escape_catalogue object
that looks or so called "tape marks" along the archive to get the
archive content. This is a purely sequential reading up to the time
the escape_catalogue reach a mark telling it has reached the catalogue
stored at end of archive. At that time the escape_catalogue loads this
catalogue into memory, compares it with what it has gathered along the
pass (it should be the same thing except for deleted files). If OK,
it continue further the reading by providing the deleted files to the
test routine. At that time, this leads dar to read the directories
leading to these deleted files which trigger the skip back to already
inspected slices, but once the directory leading to each deleted file
have been inspected the test routine does nothing about the deleted
files (because there is nothing to do, as the file is deleted, dar only
records few metadata it has, like filename, inode type and the like).

To validate his hypothesis, if you list the content of your archive,
for /var/lib/samba/private/msg.sock with something like

dar -l <archive> -g var/lib/samba/private/msg.sock
(don't use the --sequential-read mode for that, dar should just need
the last slice, or use an extracted catalogue of the archive if you have
one)

you should see both:

/var/lib/samba/private/msg.sock/3238
/var/lib/samba/private/msg.sock/3331

being deleted files and none of the other entries of that msg.sock
directory being recorded as deleted.


So to summarize:

- you have been hitting a bug fixed in 2.6.7 that avoid you outputing
archive on stdout, thus the dar_xform hint should workaround the issue
using 2.6.7 or 2.6.8.
- there is probably an inconsistency between the testing routing and
the escape_catalogue object that reads archive content sequentially from
the archive... I let you confirm that or not.

> 
> Graham
> 
> 

Cheers,
Denis

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support
signature.asc (application/pgp-signature, 833 B)