Re: Huge archive and last slice

Denis Corbin <[email protected]> Fri, 15 Dec 2023 19:30:53 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>

On 14/12/2023 22:37, Raoul Bonnal wrote:
> 
[...]
> 
>>> OK, I could make it. You can checkout the code from git on branch
>>> "no_last_slice". I have to test is further but it seems OK so far. I don't know
>>> yet where I will merge those changes, probably into branch 2.8.x not 2.7.x, but
>>> will keep it updated beside branch_2.7.x until then.
> 
> 
> I can confirm that the branch `no_last_slice` works in our setting. We 
> rebuild the source code using the debian test, borrowing 
> the dar_2.7.13-1.dsc and the debian dir. I did a quick and dirty build 
> to get a running dar_no_last_slice.

thanks for confirmation!
non-regression tests are still running and no issue has been met met so far.

> 
> 
> 
>>>
>>> This change leads dar to ask the first slice instead of the last slice only when
>>> an isolated catalog has been provided and the operation reads a backup in direct
>>> access mode (not in sequential read mode). In sequential read mode dar already
>>> fetch this info from the first slice, of course.
>>
>>I have some questions here:
>>
>>1) When dar_manager is in use, does it always request the first or the
>>last slice?  My vague memory was that it doesn't, but this may not have
>>been accurate.
> I did not use dar_manager today (branch no_last_slice). Yesterday 
> (w/2.7.13 backport) it build the `dar -x` which in the end requested the 
> last slice

I have double checked, dar_manager (no_last_slice) works properly with 
dar backups. It only requires the last slices. And for an isolated 
catalogue, of course it requires all the slices.

> 
> 
>>
>>2) If the needed data could be satisfied with either the first or the
>>last slice, could this be gated behind an option or something?  For
>>consistency, I have built workflow around assuming the last slice is
>>always needed.  I suppose I could always just not supply a catalog and
>>get that existing behavior.  But does this change how dar_manager
>>behaves?
> 
>>3) Could the remaining missing information be added to the catalog so we
>>need neither the first nor the last slice when the catalog is supplied?


> 
> This was my original inquiry. I have to admit that the "first slice" is 
> more than reasonable to us.
> I can imagine that injecting all the information into the catalog may 
> require more work and tests.

yes, so it will not be inside the catalogue, but probably beside it

> 
> 
>>> So there is changes from user point of view. Let me know if someone has any
>>> object to this... I can still surface this as a new parameter for user to
>>> decide, but would like to avoid this extra work, if it is not necessary.
> +1 to avoid extra work, but as mentioned by John many may have 
> consolidate pipelines.

yep, that's why I'm asking before merging this change.

Back to the other request asking to store of the archive format 
information in the isolated catalog archive, to avoid dar asking the 
first slice at all while reading an archive, I've hit another rock on 
the road: the sar layer.

This layer is used like a file with an open() phase, a read() phase and 
last a close() phase. At open() time you can specify whether to open it 
by the end of by the beginning. Thus even if we have all the above 
information about the archive format from the isolated catalogue 
archive, when reading it comes to the turn to read the main archive, the 
sar layer has first to be opened() to fetch the slicing layout ( { slice 
size, first slice size, and a few other parameters } ) and thus read 
either the first or the last slice...

A better approach would be to delay the time to fetch the slicing layout 
when a read() request comes, but then at read() time, how to know which 
slice to fetch since the slicing layout is unknown? The catalog only 
stores offset of information as offset in this virtual big file emulated 
by the sar layer, this offset has to be converted into slice number and 
slice offset based on the slicing information.

One could say it could be possible to store the slicing information of 
the archive of reference into the isolated catalogue archive (beside the 
catalogue and archive format information), but this would break the 
ability to re-slice an archive and use it whatever is its slicing 
pattern (see dar_xform) even after the time the isolated catalogue has 
been created. I could even lead dar fetching data from the wrong 
slice/offset while restoring... hopefully the CRC check will trigger 
that but from user standpoint this would be ugly.

In consequence, I see no proper way, without breaking any feature to 
avoid dar asking either the first or the last slice when reading an 
archive using an isolated catalogue, method that is precisely used to 
avoid having to fetch the last slice...

The best way in that context (read with external catalogue) is to fetch 
the first slice instead of the last one, first slice which can be set 
small (see -S option) on purpose. I'll probably make an option of that, 
to not break 21 years of history and usage :)

> 
> Cheers,
> R.
> 
> 

Cheers,
Denis
OpenPGP_signature.asc (application/pgp-signature, 840 B)