Re: Getting "Skipping backward is not possible on a pipe" after upgrade

Hi Denis Corbin!

 On 2019.10.18 at 21:06:05 +0200, Denis Corbin wrote next:

> since 2.6.0 you have the ability to write the archive to a remote
> server using sftp or ftp. But you can also run dar in an ssh session
> and output the archive through the ssh session on a remote system
> (since version 1.0.0)

Actually dar in ssh session always was problematic for me, so I backed
up remote systems through the use of netcat instead. However, this is
not what I want to use here..

> here is the new syntax brought by 2.6.0:
> 
> dar -c sftp://user@remote_host/some/where/washi-linux-2019.10.17-full
> - -B ~/.darrc -R / washi-linux-2019.10.17-full

Thanks. I'll try it someday!

> 
> there is however some issues and workaround if the your
> ~/.ssh/known_hosts contains some ecdsa-* entries (limitation due to
> libssh2 on which libcurl relies, on which libdar in turn relies...)

Mmm yeah some projects have switched from libssh2 to libssh for ecdsa /
ed25519 support and some other features.

> Using pipes for reading an archive instead, has huge penalties as it

Oh, reading is not a problem for my case; only writing is, because of
permission issue.

> strange, there is only three cases dar tries to skips backward
> - - bad compression / tries to resave uncompressed
> - - file has changed (dirty file) / retry to save it (see
> - --retry-on-change option)
> - - when repairing an archive dropping a file that could not be read
> completely or which lack a CRC.

I am using compression, and retry-on-change option is enabled by default:
--compression=xz:3
-m 256
--cache-directory-tagging
--alter=secu

also some things like compression excludes.

Still, this worked in the past; I remember rare warnings like "file
keeps changing, stop trying to backup new copies" (from default
retry-on-change value of 3) and it did not prevent this pipe thing from
working.

> anyway, putting appart the third case, before skipping backward dar
> tests whether it is possible to skip backward. If not it flags file as
> dirty or keep it with a bad compression ratio.
> 
> So this is probably a bug, I will chase it

Thanks!

I worked around the problem by using local disk to store backup there
and then copied it to the share; but clearly something broke that was
working before (no idea if it's just dar version change or it's built
differently now. My distro is Fedora 30).

> if the host you want to backup has an ssh service available you can
> run from a remote host the following command (old fashion available
> from version 1.0.0):
> 
> ssh user@host_to_backup sudo dar -c - -B ~/.darrc -R /
> exclude-path-washi-other > /some/where/washi-linux-2019.10.17-full.1.dar

I've tried this a long time ago but it didn't work. To be precise, this command
ssh -t gw sudo /usr/bin/dar -c - -B ~mosgalin/.darrc > /mnt/storage/backup/gw-2019.10.17-full.1.dar

doesn't work for me, as it produces broken backup. If trying to list its
contents after that I get

$ dar -l /mnt/storage/backup/gw-2019.10.17-full
Warning: using insecure memory!
Error while reading archive's header, this may be because this archive is an old encrypted archive or that data corruption took place, Assuming it is an old archive, we have to read the header at the beginning of the first slice...
Final memory cleanup...
FATAL error, aborting operation

It was always like this as long as I remember so I'm just using netcat,
too lazy to figure out what's wrong here.

Backup looks the same to my eyes, the only difference is that "file"
reports it as "end slice" (idk why)
/mnt/storage/backup/gw-2019.10.17-full.1.dar:          dar archive, label "7139aa5d 00000000 7210" end slice
/mnt/storage/backup/stor-2019.10.17-full.1.dar:        dar archive, label "6385a95d 00000000 5b0b"


> if you want several slices you have to pipe the ssh output to
> dar_xform instead of redirecting it to a file:
> 
> ssh user@host_to_backup sudo dar -c - -B ~/.darrc -R /
> exclude-path-washi-other | dar_xform -s 1G -S 500M -
> /some/where/washi-linux-2019.10.17-full

Thanks, I'll try dar_xform. Though the ssh part is still broken :)

> simple no? ;-)

Actually, to be completely honest - no :)
Some parts of dar syntax and especially things like its interaction with
piping, dar_xform and few others are just so complicated both in
concepts and syntax (even comparing to lots of other complicated
features, many of which I use!) so I try to avoid them.

Frankly I find dar to have too many options and many of them to be too
complex - that is, actually accessing its features is too hard sometimes.

For me it's dar #1 shortcoming (#2 being lack of multithreaded
compression support.. even though things like xz -T, pigz and pbzip2
work nice, but dar is usually limited by single cpu core doing
compression). Then again, each time I think that it's too complicated
and want to replace it with something else when wanting to back up some
new computer, I discover that I just can't find another tool which does
the same - full and differential backup to single archive with splitting
by size, compression, excludes that can be used from command line in
non-interactive way. So I even started to backup some Windows systems
with dar because... I checked reviews of like "25 best backup tools for
Windows" and they all lack this simple set of features :)

So, this huge set of features also means that everyone can find whatever
they need. Which is very nice :) Thanks again for this tool, it's
awesome.

Oh yeah, #3 dar shortcoming for me is exactly this, problematic
interaction when wanting to backup one system while storing backup on
the other. So far, netcat is the only solution I know but it's far from
elegant..

-- 

Vladimir