Re: Huge archive and last slice

Thanks for all your work on this, Denis.  This all makes good sense to
me!

- John

On Fri, Dec 15 2023, Denis Corbin wrote:

> [[PGP Signed Part:Undecided]]
> Hi,
>
> I've added option -affs / --alter=force-first-slice in branch "no_first_slice".
> The man page has also been updated accordingly.
>
> In consequence, the default behavior does not change, but Raoul will like using
> the -affs option.
>
> 2.7.x releases will not receive this new feature, but it will be added to the
> next major release: 2.8.0 (already merged into GIT branch master).
>
> I will however keep updated the branch "no_first_slice" for it receives all
> fixes and modifications brought to future 2.7.x releases, up to the time release
> 2.8.0 will be available.
>
> OK this may sound complicated, but you can understand I don't want to waste too
> much time with the release process, which takes several hours to be done
> properly and in coherence between doc / code packaging / website(s) / emails...
>
> Cheers,
> Denis
>
> On 15/12/2023 19:30, Denis Corbin wrote:
>> On 14/12/2023 22:37, Raoul Bonnal wrote:
>>>
>> [...]
>>>
>>>>> OK, I could make it. You can checkout the code from git on branch
>>>>> "no_last_slice". I have to test is further but it seems OK so far. I don't
>>>>> know
>>>>> yet where I will merge those changes, probably into branch 2.8.x not 2.7.x,
>>>>> but
>>>>> will keep it updated beside branch_2.7.x until then.
>>>
>>>
>>> I can confirm that the branch `no_last_slice` works in our setting. We
>>> rebuild the source code using the debian test, borrowing the dar_2.7.13-1.dsc
>>> and the debian dir. I did a quick and dirty build to get a running
>>> dar_no_last_slice.
>> thanks for confirmation!
>> non-regression tests are still running and no issue has been met met so far.
>>
>>>
>>>
>>>
>>>>>
>>>>> This change leads dar to ask the first slice instead of the last slice only
>>>>> when
>>>>> an isolated catalog has been provided and the operation reads a backup in
>>>>> direct
>>>>> access mode (not in sequential read mode). In sequential read mode dar
>>>>> already
>>>>> fetch this info from the first slice, of course.
>>>>
>>>> I have some questions here:
>>>>
>>>> 1) When dar_manager is in use, does it always request the first or the
>>>> last slice?  My vague memory was that it doesn't, but this may not have
>>>> been accurate.
>>> I did not use dar_manager today (branch no_last_slice). Yesterday (w/2.7.13
>>> backport) it build the `dar -x` which in the end requested the last slice
>> I have double checked, dar_manager (no_last_slice) works properly with dar
>> backups. It only requires the last slices. And for an isolated catalogue, of
>> course it requires all the slices.
>>
>>>
>>>
>>>>
>>>> 2) If the needed data could be satisfied with either the first or the
>>>> last slice, could this be gated behind an option or something?  For
>>>> consistency, I have built workflow around assuming the last slice is
>>>> always needed.  I suppose I could always just not supply a catalog and
>>>> get that existing behavior.  But does this change how dar_manager
>>>> behaves?
>>>
>>>> 3) Could the remaining missing information be added to the catalog so we
>>>> need neither the first nor the last slice when the catalog is supplied?
>>
>>>
>>> This was my original inquiry. I have to admit that the "first slice" is more
>>> than reasonable to us.
>>> I can imagine that injecting all the information into the catalog may require
>>> more work and tests.
>> yes, so it will not be inside the catalogue, but probably beside it
>>
>>>
>>>
>>>>> So there is changes from user point of view. Let me know if someone has any
>>>>> object to this... I can still surface this as a new parameter for user to
>>>>> decide, but would like to avoid this extra work, if it is not necessary.
>>> +1 to avoid extra work, but as mentioned by John many may have
>>> consolidate pipelines.
>> yep, that's why I'm asking before merging this change.
>> Back to the other request asking to store of the archive format information in
>> the isolated catalog archive, to avoid dar asking the first slice at all while
>> reading an archive, I've hit another rock on the road: the sar layer.
>> This layer is used like a file with an open() phase, a read() phase and last a
>> close() phase. At open() time you can specify whether to open it by the end of
>> by the beginning. Thus even if we have all the above information about the
>> archive format from the isolated catalogue archive, when reading it comes to
>> the turn to read the main archive, the sar layer has first to be opened() to
>> fetch the slicing layout ( { slice size, first slice size, and a few other
>> parameters } ) and thus read either the first or the last slice...
>> A better approach would be to delay the time to fetch the slicing layout when
>> a read() request comes, but then at read() time, how to know which slice to
>> fetch since the slicing layout is unknown? The catalog only stores offset of
>> information as offset in this virtual big file emulated by the sar layer, this
>> offset has to be converted into slice number and slice offset based on the
>> slicing information.
>> One could say it could be possible to store the slicing information of the
>> archive of reference into the isolated catalogue archive (beside the catalogue
>> and archive format information), but this would break the ability to re-slice
>> an archive and use it whatever is its slicing pattern (see dar_xform) even
>> after the time the isolated catalogue has been created. I could even lead dar
>> fetching data from the wrong slice/offset while restoring... hopefully the CRC
>> check will trigger that but from user standpoint this would be ugly.
>> In consequence, I see no proper way, without breaking any feature to avoid dar
>> asking either the first or the last slice when reading an archive using an
>> isolated catalogue, method that is precisely used to avoid having to fetch the
>> last slice...
>> The best way in that context (read with external catalogue) is to fetch the
>> first slice instead of the last one, first slice which can be set small (see
>> -S option) on purpose. I'll probably make an option of that, to not break 21
>> years of history and usage :)
>>
>>>
>>> Cheers,
>>> R.
>>>
>>>
>> Cheers,
>> Denis
>>
>
> [[End of PGP Signed Part]]