Re: --sequential-read not sequential???

>>>> 1) Using dar_xform and an actual pipe to feed "dar -t". That caused
>>>> dar to emit a hard error because it cannot go backwards on a pipe.
>>>> So the "dar -t" failed. It didn't emit any message explaining why
>>>> it wanted to go backwards.
>>>
>>> strange, which version of dar are you using by the way?
>>
>> 2.6.6
> 
> OK so this explains one point, as described on the Changelog:
> 
> from 2.6.6 to 2.6.7
> [...]
> - fixed bug: dar failed creating an archive on its standard output
>   reporting the error message "Skipping backward is not possible on a pipe"
> 

I have now upgraded to 2.6.8 but it hasn't changed the results in my
cases. Maybe because I am not doing archive creation?

>> I have now found an archive with a single slice which reproduces the
>> reading problem. This eliminates the problem being anything to do with
>> multiple slices, or dar_xform. Unfortunately this archive is 1.3GB and
>> contains private data so I cannot make it available to you. With this
>> particular archive,
> 
> No problem, I have to try reproducing in my side, good to know this
> has nothing related with slicing.
> 
>>
>> dar -t - --sequential-read </var/tmp/DARsystemDiff06.1.dar
>>
>> gives the following output...
>>
>> ERR <ROOT>/var/indexes/cobb : Skipping backward is not possible on a pipe
>> ERR <ROOT>/var/indexes/cobb/.Trash : Skipping backward is not possible
>> on a pipe
>> ERR <ROOT>/home/cobb : Skipping backward is not possible on a pipe
>> ERR <ROOT>/home/cobb/dovecot : Skipping backward is not possible on a pipe
>> ERR <ROOT>/home/cobb/dovecot/cur : Skipping backward is not possible on
>> a pipe
>>
>>  --------------------------------------------
>>  5903243 item(s) treated
>>  0 item(s) with error
>>  0 item(s) ignored (excluded by filters)
>>  --------------------------------------------
>>  Total number of items considered: 5903243
>>  --------------------------------------------
> 
> this is the bug that has been fixe in 2.6.7 ad following versions

I still get this with 2.6.8. Was the fixed bug only to do with archive
creation? I assume this is actually an instance of the bug we have been
working here.

>>> if you still have the whole dar ouput, can you provide the output of
>>> this command:
>>>
>>> grep "/var/lib/samba/private" <dar output>
>>
>>
>> Yes. I will send it as an attachment to a separate message off-list.
> 
> Yes, thanks I've received it. My understanding is the following:
> the test routing reads archive contents from a escape_catalogue object
> that looks or so called "tape marks" along the archive to get the
> archive content. This is a purely sequential reading up to the time
> the escape_catalogue reach a mark telling it has reached the catalogue
> stored at end of archive. At that time the escape_catalogue loads this
> catalogue into memory, compares it with what it has gathered along the
> pass (it should be the same thing except for deleted files). If OK,
> it continue further the reading by providing the deleted files to the
> test routine. At that time, this leads dar to read the directories
> leading to these deleted files which trigger the skip back to already
> inspected slices, but once the directory leading to each deleted file
> have been inspected the test routine does nothing about the deleted
> files (because there is nothing to do, as the file is deleted, dar only
> records few metadata it has, like filename, inode type and the like).
> 
> To validate his hypothesis, if you list the content of your archive,
> for /var/lib/samba/private/msg.sock with something like
> 
> dar -l <archive> -g var/lib/samba/private/msg.sock
> (don't use the --sequential-read mode for that, dar should just need
> the last slice, or use an extracted catalogue of the archive if you have
> one)
> 
> you should see both:
> 
> /var/lib/samba/private/msg.sock/3238
> /var/lib/samba/private/msg.sock/3331
> 
> being deleted files and none of the other entries of that msg.sock
> directory being recorded as deleted.

I saw:

[--- REMOVED ENTRY ----] (Tue Jan 21 03:49:58 2020) [s]
var/lib/samba/private/msg.sock/3238
[--- REMOVED ENTRY ----] (Tue Jan 21 03:49:58 2020) [s]
var/lib/samba/private/msg.sock/3331

No other REMOVED ENTRY, and no other mentions of those files. So, I
think that is consistent with your hypothesis.

> So to summarize:
> 
> - you have been hitting a bug fixed in 2.6.7 that avoid you outputing
> archive on stdout, thus the dar_xform hint should workaround the issue
> using 2.6.7 or 2.6.8.

Unfortunately, no. Even with 2.6.8 it seems that the bug you outline
below means "dar -t" still tries (and fails) to go backwards on the
pipe, even when dar_xform is being used. In particular, using dar_xform
with the archive mentioned above (DARsystemDiff06.1.dar) to pipe into
"dar -t - --sequential-read" still gets these errors:

ERR <ROOT>/var/indexes/cobb : Skipping backward is not possible on a pipe
ERR <ROOT>/var/indexes/cobb/.Trash : Skipping backward is not possible
on a pipe
ERR <ROOT>/home/cobb : Skipping backward is not possible on a pipe
ERR <ROOT>/home/cobb/dovecot : Skipping backward is not possible on a pipe
ERR <ROOT>/home/cobb/dovecot/cur : Skipping backward is not possible on
a pipe

> - there is probably an inconsistency between the testing routing and
> the escape_catalogue object that reads archive content sequentially from
> the archive... I let you confirm that or not.

Your hypothesis seems likely given the results on the msg.sock directory
I include above.

Graham