Re: questions about on fly isolation with Azure

On 06/10/2021 23:55, Daniel wrote:
> Hi Denis!

Hi Daniel,

> 
> This is such a great thorough answer and you even gave me super hints
> that I didn't even think of!
> Thanks very much!
> 
> You were right at everything (obviously of course), the size issue was
> because of compression difference (no vs gzip) :-)
> Slices are a super idea, I have never bothered to use them but with this
> new Azure method, it is a must go!
> 
> BTW I have been using dar from about 2003, nothing can beat it in
> reliability and flexibility. Such a great thing you have created, thanks
> very much in the name of the whole world!!!

Thank you very much for your words, well appreciated! Funny and nice to
see that there are sill some users from Day 1 among more recent ones (I
think I beat you by slightly more than one year using dar ;^)

...but that's nothing in regards to the soon 20 years of existence this
software has... yes I'm old now

Maybe I should setup a survey like this one: "For how much time do you
use dar?" "For what purpose(s), backup? Archiving? Other?" "With which
feature? compression algorithm? slicing? Encryption? ..." but I have no
time for that and I prefer adding new features :)

And well, the whole world and you are very welcome, this is my pleasure
to share and be useful to others :^)

> 
> Regards,
> Daniel

Cheers,
Denis

> 
> Denis Corbin <[email protected] <mailto:[email protected]>> ezt írta
> (időpont: 2021. okt. 5., K, 20:28):
> 
>     On 05/10/2021 15:37, Daniel wrote:
>     > Hi!
> 
>     Hi!
> 
>     >
>     > I would like to make incremental backups to Azure Cool storage where
>     > uploading costs much less then downloading data, so I would absolutely
>     > try to stay away from downloading data and only keep it for emergency
>     > scenarios.
>     > Previously I haven't used such storage and isolated the catalogue
>     after
>     > the archive creation, so I was happy to see the "on fly" option.
>     >
>     > My questions:
>     >
>     > 1. Am I right that "on fly" isolation solves my problem about minimal
>     > download needed, because the archive is only going to be uploaded to
>     > Azure and the isolation extraction is happening on the host-side
>     without
>     > any data needed from Azure?
> 
>     Short answer: yes
> 
>     Long answer: the backup and its the on-fly isolation are dropped where
>     you instruct dar to drop them, so if the backup is written to a remote
>     storage (unlike the default) and the isolation is done locally, nothing
>     will be read from the remote storage for on-fly isolation.
> 
>     Note that using slices may have some advantages as for a restoration you
>     will need the last slice (containing the catalogue aka metadata) and if
>     restoring only a few files, dar will request only a few other slices to
>     read the data from.
> 
>     You can even go one step further with slices, by using the isolated
>     catalogue you have locally to fetch only the needed slices and thus
>     having a lot of slices will save you money when restoring a few files.
>     (see -E options to automate the slice upload and download).
> 
>     >
>     > 2. The log looks like this sequentally, so it gets me a bit worried,
>     > because according to this the isolation only occurs after the archive
>     > contents have been written. And further more these are real big
>     archives
>     > (300GB+), so memory is out of the question here:
>     > ----
>     > Writing archive contents...
>     > ...
>     > Making room in memory (releasing memory used by archive of
>     reference)...
>     > Now performing on-fly isolation...
>     > Preparing the archive contents for isolation...
>     > Writing archive contents...
>     > Final memory cleanup...
>     > ----
> 
>     short answer: if your RAM resource allow you to make a backup of
>     whatever size (300GB+), you will be able to create an isolated catalogue
> 
>     long answer: the catalog only contains the file's metadata (filename,
>     ownership, permission, dates, ...) not the 300GB+ data of your archive.
> 
>     Long answer: During a normal backup process, the data is treated somehow
>     as a stream (sliced/encrypted/compressed/escaped for sparse file
>     handling/and so on) while the metadata is kept in memory. Memory usage
>     will increase linearly to number of file saved. When all data is
>     treated, dar adds the metadata (the catalogue) into the stream (which
>     ends at the end of the archive/backup)
> 
>     Once the archive is completed, having with its data and metadata, on-fly
>     isolation does not release the memory, but drops the metadata a second
>     time, this time without data in another archive/backup. RAM requirement
>     is not changed with or without on-fly isolation.
> 
>     To simplify this previous explanation, I avoided mentioning that the
>     metadata is also inter-mixed with the data all during the backup to
>     increase reliability and allow the --sequential-read mode. (you can
>     disable it by mean of -at option). The metadata is thus present twice:
>     in-lined with the data and at the end of the backup in the "catalogue".
> 
>     >
>     > 3. It is not clear for me why a normal "after-isolated, -C"
>     catalogue of
>     > the same thing is 380932 bytes when the "on fly" catalogue is 98967
>     > bytes. I am using encryption btw, but at both cases...
> 
>     both catalogues are the same, what may change is the fact you have not
>     been using the same compression algorithm to compress them.
> 
>     You can check that by issuing:
>     dar -l <catalogue name> -q
> 
>     you should see at the top of the input the parameters of the
>     backup/archive
> 
>        [...]
>        Compression algorithm used           : xz
>        [...]
> 
>     OK unfortunately you don't find the compression level because it is not
>     needed to read the archive/backup.
> 
>     and at the end of the output, the composition of the catalogue:
> 
>        CATALOGUE CONTENTS :
> 
>        total number of inode : 61928
>        fully saved           : 61928
>        [...]
> 
>     The first part of the output may differ, the second should stay the same
>     between two isolated catalogue based on the same backup, may one be done
>     on-fly and the other not.
> 
>     >
>     > Thanks all for you help!
>     >
>     > Regards,
>     > Daniel
>     >
>     >
> 
>     Regards,
>     Denis
> 
>