Re: Huge archive and last slice

Denis Corbin <[email protected]> Fri, 15 Dec 2023 18:44:56 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>

On 14/12/2023 18:32, John Goerzen wrote:
> On Wed, Dec 13 2023, Denis Corbin wrote:
> 
>> OK, I could make it. You can checkout the code from git on branch
>> "no_last_slice". I have to test is further but it seems OK so far. I don't know
>> yet where I will merge those changes, probably into branch 2.8.x not 2.7.x, but
>> will keep it updated beside branch_2.7.x until then.
>>
>> This change leads dar to ask the first slice instead of the last slice only when
>> an isolated catalog has been provided and the operation reads a backup in direct
>> access mode (not in sequential read mode). In sequential read mode dar already
>> fetch this info from the first slice, of course.
> 
> I have some questions here:
> 
> 1) When dar_manager is in use, does it always request the first or the
> last slice?  My vague memory was that it doesn't, but this may not have
> been accurate.

dar_manager always read backup or isolated catalogues in direct mode, 
this it fetch the catalogue from the end of the backup (which is located 
on the last slice(s)). Right before that, it also fetch the archive 
version format from the last slice (the "archive trailer"), so usually 
dar_manager only asks the last slice (or all the slices if this is an 
isolated catalogue that has been split in slices, something that is also 
possible).

> 
> 2) If the needed data could be satisfied with either the first or the
> last slice, could this be gated behind an option or something?  For
> consistency, I have built workflow around assuming the last slice is
> always needed.  I suppose I could always just not supply a catalog and
> get that existing behavior.  But does this change how dar_manager
> behaves?

in short: no, it does change how dar_manager behaves.

The change with the "no_last_slice" will only show when:
- you read a backup in direct access mode (by opposition to sequential 
read mode)
- you use a external isolated catalogue (by mean or -A option)

dar_manager always read archive or isolated catalogues in direct access 
mode.


> 
> 3) Could the remaining missing information be added to the catalog so we
> need neither the first nor the last slice when the catalog is supplied?
> 

First some info to clarify the explanations:

Let's take a parallel with a filesystem:
- the filesystem corresponds to the archive format (this info is stored 
in the structure called "archive header" (implemented as the class 
header_version), located at the beginning and at the end of the 
archive): While filesystem manages lists of blocks (which ones are 
free/used...) dar file format defines things like the tape marks used 
between files (if used), the ciphering layer (if present), the 
compression layer (if present) and so on.
- the metadata in a filesystem corresponds to the catalogue (filename, 
permissions, data location, etc.)
- the file data in the filesystem corresponds to the file data in the 
archive

We could also add a layer below the "filesystem" one: the "sar" layer 
(sar stands for segmentation and rassemblement). This one manages 
slicing and exposes to the upper layer a large file containing all the 
slices contents as if it was a single large file... sar layer has its 
own header and metadata in each slices.

As such it does not make sense to hold "filesystem" level information 
inside the catalogue (metadata information)

It does not make sense neither to assume the "filesystem" level 
information from an isolated catalog is the same as the one of the 
archive it refers to. This isolation process can be done long after with 
more recent version of dar.

Now what is an "isolated catalogue"? This is a normal archive with a 
"filesystem" level, having on top the same metadata (aka catalogue) as 
the original backup, but without the data, in other words it does not 
have specific field to store isolated the "archive header" of the 
archive of reference.

So this is not simple, but I'm looking forward for a solution and the 
best place to hold this information inside an isolated Catalog.

>>
>> So there is changes from user point of view. Let me know if someone has any
>> object to this... I can still surface this as a new parameter for user to
>> decide, but would like to avoid this extra work, if it is not necessary.
>>
>> Cheers,
>> Denis
>>
>>>>
>>>>
>>>>
>>>
>>
>> [[End of PGP Signed Part]]
>
OpenPGP_signature.asc (application/pgp-signature, 840 B)