Re: Huge archive and last slice

Denis Corbin <[email protected]> Fri, 15 Dec 2023 22:45:27 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>
Hi,

I've added option -affs / --alter=force-first-slice in branch 
"no_first_slice". The man page has also been updated accordingly.

In consequence, the default behavior does not change, but Raoul will 
like using the -affs option.

2.7.x releases will not receive this new feature, but it will be added 
to the next major release: 2.8.0 (already merged into GIT branch master).

I will however keep updated the branch "no_first_slice" for it receives 
all fixes and modifications brought to future 2.7.x releases, up to the 
time release 2.8.0 will be available.

OK this may sound complicated, but you can understand I don't want to 
waste too much time with the release process, which takes several hours 
to be done properly and in coherence between doc / code packaging / 
website(s) / emails...

Cheers,
Denis

On 15/12/2023 19:30, Denis Corbin wrote:
> 
> 
> On 14/12/2023 22:37, Raoul Bonnal wrote:
>>
> [...]
>>
>>>> OK, I could make it. You can checkout the code from git on branch
>>>> "no_last_slice". I have to test is further but it seems OK so far. I 
>>>> don't know
>>>> yet where I will merge those changes, probably into branch 2.8.x not 
>>>> 2.7.x, but
>>>> will keep it updated beside branch_2.7.x until then.
>>
>>
>> I can confirm that the branch `no_last_slice` works in our setting. We 
>> rebuild the source code using the debian test, borrowing 
>> the dar_2.7.13-1.dsc and the debian dir. I did a quick and dirty build 
>> to get a running dar_no_last_slice.
> 
> thanks for confirmation!
> non-regression tests are still running and no issue has been met met so 
> far.
> 
>>
>>
>>
>>>>
>>>> This change leads dar to ask the first slice instead of the last 
>>>> slice only when
>>>> an isolated catalog has been provided and the operation reads a 
>>>> backup in direct
>>>> access mode (not in sequential read mode). In sequential read mode 
>>>> dar already
>>>> fetch this info from the first slice, of course.
>>>
>>> I have some questions here:
>>>
>>> 1) When dar_manager is in use, does it always request the first or the
>>> last slice?  My vague memory was that it doesn't, but this may not have
>>> been accurate.
>> I did not use dar_manager today (branch no_last_slice). Yesterday 
>> (w/2.7.13 backport) it build the `dar -x` which in the end requested 
>> the last slice
> 
> I have double checked, dar_manager (no_last_slice) works properly with 
> dar backups. It only requires the last slices. And for an isolated 
> catalogue, of course it requires all the slices.
> 
>>
>>
>>>
>>> 2) If the needed data could be satisfied with either the first or the
>>> last slice, could this be gated behind an option or something?  For
>>> consistency, I have built workflow around assuming the last slice is
>>> always needed.  I suppose I could always just not supply a catalog and
>>> get that existing behavior.  But does this change how dar_manager
>>> behaves?
>>
>>> 3) Could the remaining missing information be added to the catalog so we
>>> need neither the first nor the last slice when the catalog is supplied?
> 
> 
>>
>> This was my original inquiry. I have to admit that the "first slice" 
>> is more than reasonable to us.
>> I can imagine that injecting all the information into the catalog may 
>> require more work and tests.
> 
> yes, so it will not be inside the catalogue, but probably beside it
> 
>>
>>
>>>> So there is changes from user point of view. Let me know if someone 
>>>> has any
>>>> object to this... I can still surface this as a new parameter for 
>>>> user to
>>>> decide, but would like to avoid this extra work, if it is not 
>>>> necessary.
>> +1 to avoid extra work, but as mentioned by John many may have 
>> consolidate pipelines.
> 
> yep, that's why I'm asking before merging this change.
> 
> Back to the other request asking to store of the archive format 
> information in the isolated catalog archive, to avoid dar asking the 
> first slice at all while reading an archive, I've hit another rock on 
> the road: the sar layer.
> 
> This layer is used like a file with an open() phase, a read() phase and 
> last a close() phase. At open() time you can specify whether to open it 
> by the end of by the beginning. Thus even if we have all the above 
> information about the archive format from the isolated catalogue 
> archive, when reading it comes to the turn to read the main archive, the 
> sar layer has first to be opened() to fetch the slicing layout ( { slice 
> size, first slice size, and a few other parameters } ) and thus read 
> either the first or the last slice...
> 
> A better approach would be to delay the time to fetch the slicing layout 
> when a read() request comes, but then at read() time, how to know which 
> slice to fetch since the slicing layout is unknown? The catalog only 
> stores offset of information as offset in this virtual big file emulated 
> by the sar layer, this offset has to be converted into slice number and 
> slice offset based on the slicing information.
> 
> One could say it could be possible to store the slicing information of 
> the archive of reference into the isolated catalogue archive (beside the 
> catalogue and archive format information), but this would break the 
> ability to re-slice an archive and use it whatever is its slicing 
> pattern (see dar_xform) even after the time the isolated catalogue has 
> been created. I could even lead dar fetching data from the wrong 
> slice/offset while restoring... hopefully the CRC check will trigger 
> that but from user standpoint this would be ugly.
> 
> In consequence, I see no proper way, without breaking any feature to 
> avoid dar asking either the first or the last slice when reading an 
> archive using an isolated catalogue, method that is precisely used to 
> avoid having to fetch the last slice...
> 
> The best way in that context (read with external catalogue) is to fetch 
> the first slice instead of the last one, first slice which can be set 
> small (see -S option) on purpose. I'll probably make an option of that, 
> to not break 21 years of history and usage :)
> 
>>
>> Cheers,
>> R.
>>
>>
> 
> Cheers,
> Denis
>
OpenPGP_signature.asc (application/pgp-signature, 840 B)