Re: Getting "Skipping backward is not possible on a pipe" after upgrade

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 17/10/2019 22:21, Vladimir Mosgalin wrote:
> Hello everybody.

Hello Vladimir,

> 
> Got a question (well, mostly to Denis, as usual :). I was invoking
> dar like this for a long time:
> 
> sudo dar -c - -B ~/.darrc -R / exclude-path-washi-other >
> /mnt/storage/backup/washi-linux-2019.10.17-full.1.dar
> 
> There are really no relevant options in RC file other than
> compression / silenced warnings / path and compression excludes.
> 
> The reason why I was running it like this is because I had it to
> read the whole filesystem (thus root needed), but on the other had
> I needed to write backup as normal user, as the /mnt/storage being
> a kerberized NFS filesystem which root cannot write to.
> Unfortunately I lost ability to split archives by size this way but
> I just had to deal with it.

since 2.6.0 you have the ability to write the archive to a remote
server using sftp or ftp. But you can also run dar in an ssh session
and output the archive through the ssh session on a remote system
(since version 1.0.0)

here is the new syntax brought by 2.6.0:

dar -c sftp://user@remote_host/some/where/washi-linux-2019.10.17-full
- -B ~/.darrc -R / washi-linux-2019.10.17-full

there is however some issues and workaround if the your
~/.ssh/known_hosts contains some ecdsa-* entries (limitation due to
libssh2 on which libcurl relies, on which libdar in turn relies...)
more simple stays to use dar from ssh command espetially for writing
archive as there is not penalties compared to this new syntax (see
examples at end of this mail for that old way of using dar with ssh).

Using pipes for reading an archive instead, has huge penalties as it
sends the whole archive in sequence and dar has to wait for the data
you want to restore to pass after all the rest it may not need. Then
it also  has to wait for the rest of the archive to be received. In
that case using ssh from dar with the syntax described above is very
interesting as dar is able to directly read the portion of
slice/archive that is needed to restore any particular file: less
time, less network load.
> 
> However after some updates (currently there is dar 2.6.6 in my
> distro) this command stopped working, producing error like this
> after some time (backup is partially created at this point):
> 
> Detected Cache Directory Tagging Standard for /var/cache/man, the
> contents of that directory will not be saved Final memory
> cleanup... Error while saving data: Cannot write down the archive:
> Skipping backward is not possible on a pipe Command exited with
> non-zero status 5

strange, there is only three cases dar tries to skips backward
- - bad compression / tries to resave uncompressed
- - file has changed (dirty file) / retry to save it (see
- --retry-on-change option)
- - when repairing an archive dropping a file that could not be read
completely or which lack a CRC.

anyway, putting appart the third case, before skipping backward dar
tests whether it is possible to skip backward. If not it flags file as
dirty or keep it with a bad compression ratio.

So this is probably a bug, I will chase it

> 
> I have to admit I never understood chaining of multiple dar's or
> using dar_xform so I'm a bit at loss at what to do now. This
> command was working with dar 2.5 for sure. According to current 2.6
> documentation (http://dar.linux.free.fr/doc/usage_notes.html -
> "Single pipe") this command still supposed to work. Only it
> doesn't.

yes it is supported, you've probably hit a bug. I will look for it.

> 
> What is wrong here and is there some other way I'm supposed to be
> doing what I need here (read files as root, write backup as
> non-root user) in 2.6 version? Also, maybe there is some way to
> achieve this so that size splitting will work as well?

if the host you want to backup has an ssh service available you can
run from a remote host the following command (old fashion available
from version 1.0.0):

ssh user@host_to_backup sudo dar -c - -B ~/.darrc -R /
exclude-path-washi-other > /some/where/washi-linux-2019.10.17-full.1.dar

this will:
1 - connect to host_to_backup as login "user" with ssh protocol
2 - run on this host_to_backup the dar commands which will output the
archive on its stdandard output
3 - this stdout is send through the ssh session to the local host and
is available as the stdouput of the ssh command you ran on the local hos
t
4 - this output is reidrected to a file on localhost
5 - once dar will have completed, your ssh session will end and your
local file will be completed.

The issue here is that it sends all data over he network, thus dar
will probably be suspended from time to time if its output is
saturated due to network not being as fast as local disk I/O.
The second issue is that all data sent through ssh is ciphered, this
will slightly increase CPU load on both hosts during the operation.

if you want several slices you have to pipe the ssh output to
dar_xform instead of redirecting it to a file:

ssh user@host_to_backup sudo dar -c - -B ~/.darrc -R /
exclude-path-washi-other | dar_xform -s 1G -S 500M -
/some/where/washi-linux-2019.10.17-full

dar_xform will:
1 - read on its standard input a single sliced archive,
2 - output to /some/where/washi-linux-2019.10.17-full.*.dar the
resliced archive with the first slice having 500 MiB at most and next
slices having 1 GiB at most.

simple no? ;-)


[...]

Cheers,
Denis
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAl2qDRYACgkQCDGwvQPY
sYIRERAArOgFcMorTwjl3C4ium1QhVigbdA3gY12Tl9MlPdePieDcVfd083sGp8R
grZprVFM7Fc2qOizaHTe0C2wPcy+iLosa8gcnF3prrLA/NkdXEfYnjI7gTVFKXhu
frDWe8RERtA5Ye+mTuKyV6UDCFYNB8cs00SykymvChNgqWD5v2CBi6ptUoGdozxO
/2tAVB+FPXSr9Ywt/5QMPbyYc5vcDnvXel9UfKg7VfBqjQIIMAyYVmVGqe16z5b3
WzOQjnOE4G3jHpGSd7ky95ewM7CyfI66wGWPR+qAu4HgJ0A/hS0sQUHJg5h6maXk
uDgzOrR9DwpFXGcv1k1MGC1HN5XYEjHgVPEaOGbyyk4jRKG9KYKnuh4Ecp8oqAHm
LbmZ6jnWcFopj7VNfTmzRbfwLLYnDDv7HpbcNzbVa2tiPjtMJLuOOTZ/yBWTfER/
uCExITaauxC0x3nAnw/xbNyJRdTmB0nJpG3lmcYrRraIvYY1Jp1RQ2IeV13DdFlg
LqFdmaDWQEatZznBVWiDJl2sooJkW9E+2izy69cSIxOlmK3NJYDqltBnFCd/T6da
Qag7qQJwCCBw8I5kmrCio0h9OIKjoVm9UXuoVV/rdKzch4NtD0PQ9BBVbhUknT9R
0OKIjHwU2CCJ1oWv5271TU5RF8N3FC53i5u7p3qr0KvkFIWp2vQ=
=zHAh
-----END PGP SIGNATURE-----