Re: questions about on fly isolation with Azure

Hi Denis!

This is such a great thorough answer and you even gave me super hints that
I didn't even think of!
Thanks very much!

You were right at everything (obviously of course), the size issue was
because of compression difference (no vs gzip) :-)
Slices are a super idea, I have never bothered to use them but with this
new Azure method, it is a must go!

BTW I have been using dar from about 2003, nothing can beat it in
reliability and flexibility. Such a great thing you have created, thanks
very much in the name of the whole world!!!

Regards,
Daniel

Denis Corbin <[email protected]> ezt írta (időpont: 2021. okt. 5., K,
20:28):

> On 05/10/2021 15:37, Daniel wrote:
> > Hi!
>
> Hi!
>
> >
> > I would like to make incremental backups to Azure Cool storage where
> > uploading costs much less then downloading data, so I would absolutely
> > try to stay away from downloading data and only keep it for emergency
> > scenarios.
> > Previously I haven't used such storage and isolated the catalogue after
> > the archive creation, so I was happy to see the "on fly" option.
> >
> > My questions:
> >
> > 1. Am I right that "on fly" isolation solves my problem about minimal
> > download needed, because the archive is only going to be uploaded to
> > Azure and the isolation extraction is happening on the host-side without
> > any data needed from Azure?
>
> Short answer: yes
>
> Long answer: the backup and its the on-fly isolation are dropped where
> you instruct dar to drop them, so if the backup is written to a remote
> storage (unlike the default) and the isolation is done locally, nothing
> will be read from the remote storage for on-fly isolation.
>
> Note that using slices may have some advantages as for a restoration you
> will need the last slice (containing the catalogue aka metadata) and if
> restoring only a few files, dar will request only a few other slices to
> read the data from.
>
> You can even go one step further with slices, by using the isolated
> catalogue you have locally to fetch only the needed slices and thus
> having a lot of slices will save you money when restoring a few files.
> (see -E options to automate the slice upload and download).
>
> >
> > 2. The log looks like this sequentally, so it gets me a bit worried,
> > because according to this the isolation only occurs after the archive
> > contents have been written. And further more these are real big archives
> > (300GB+), so memory is out of the question here:
> > ----
> > Writing archive contents...
> > ...
> > Making room in memory (releasing memory used by archive of reference)...
> > Now performing on-fly isolation...
> > Preparing the archive contents for isolation...
> > Writing archive contents...
> > Final memory cleanup...
> > ----
>
> short answer: if your RAM resource allow you to make a backup of
> whatever size (300GB+), you will be able to create an isolated catalogue
>
> long answer: the catalog only contains the file's metadata (filename,
> ownership, permission, dates, ...) not the 300GB+ data of your archive.
>
> Long answer: During a normal backup process, the data is treated somehow
> as a stream (sliced/encrypted/compressed/escaped for sparse file
> handling/and so on) while the metadata is kept in memory. Memory usage
> will increase linearly to number of file saved. When all data is
> treated, dar adds the metadata (the catalogue) into the stream (which
> ends at the end of the archive/backup)
>
> Once the archive is completed, having with its data and metadata, on-fly
> isolation does not release the memory, but drops the metadata a second
> time, this time without data in another archive/backup. RAM requirement
> is not changed with or without on-fly isolation.
>
> To simplify this previous explanation, I avoided mentioning that the
> metadata is also inter-mixed with the data all during the backup to
> increase reliability and allow the --sequential-read mode. (you can
> disable it by mean of -at option). The metadata is thus present twice:
> in-lined with the data and at the end of the backup in the "catalogue".
>
> >
> > 3. It is not clear for me why a normal "after-isolated, -C" catalogue of
> > the same thing is 380932 bytes when the "on fly" catalogue is 98967
> > bytes. I am using encryption btw, but at both cases...
>
> both catalogues are the same, what may change is the fact you have not
> been using the same compression algorithm to compress them.
>
> You can check that by issuing:
> dar -l <catalogue name> -q
>
> you should see at the top of the input the parameters of the backup/archive
>
>    [...]
>    Compression algorithm used           : xz
>    [...]
>
> OK unfortunately you don't find the compression level because it is not
> needed to read the archive/backup.
>
> and at the end of the output, the composition of the catalogue:
>
>    CATALOGUE CONTENTS :
>
>    total number of inode : 61928
>    fully saved           : 61928
>    [...]
>
> The first part of the output may differ, the second should stay the same
> between two isolated catalogue based on the same backup, may one be done
> on-fly and the other not.
>
> >
> > Thanks all for you help!
> >
> > Regards,
> > Daniel
> >
> >
>
> Regards,
> Denis
>
>