Re: Huge archive and last slice

I really appreciate your work.
The deep explanation of the inner mechanisms is very clear and useful.
This project is a gem.

R.

Ottieni Outlook per Android<https://aka.ms/ghei36>

--

Raoul Bonnal
IFOM | Research Computing and Data Science Manager
Ph.: +39 02 57430 3016 | raoul.bonnal-G+fEVLdQ/[email protected]<mailto:raoul.bonnal-G+fEVLdQ/[email protected]>
www.ifom.eu<https://www.ifom.eu> | Linkedin<https://www.linkedin.com/school/ifom/> | Instagram<https://www.instagram.com/ifom_milan/?hl=it> | Facebook<https://www.facebook.com/ifom.eu> | Twitter<https://twitter.com/ifomresearch> | YouTube<https://www.youtube.com/channel/UCOVlG5DhrDWMR1N5NYBB4RQ>
IFOM ETS - The AIRC Institute of Molecular Oncology | Via Adamello 16, 20139 Milan, Italy

[cid:logoIfom_2d2161e0-3259-466c-8931-26f8c2a5979f.png]

________________________________

Confidentiality notice. This message, together with its annexes, contains information to be deemed strictly confidential and is destined only to the addressee(s) identified above who only may use, copy and, under his/their responsibility, further disseminate it. if anyone received this message by mistake or reads it without entitlement is forewarned that keeping, copying, disseminating or distributing this message to persons other than the addressee(s) is strictly forbidden and is asked to transmit it immediately to the sender and to erase the original message received.
Thank you
Save a tree - Do not print this email unless absolutely necessary

________________________________
From: John Goerzen <[email protected]>
Sent: Saturday, December 16, 2023 12:59:49 AM
To: need help compiling or using dar ? This mailing list is for you. <[email protected]>
Cc: Denis Corbin <[email protected]>
Subject: Re: [Dar-support] Huge archive and last slice

Thanks for all your work on this, Denis.  This all makes good sense to
me!

- John

On Fri, Dec 15 2023, Denis Corbin wrote:

> [[PGP Signed Part:Undecided]]
> Hi,
>
> I've added option -affs / --alter=force-first-slice in branch "no_first_slice".
> The man page has also been updated accordingly.
>
> In consequence, the default behavior does not change, but Raoul will like using
> the -affs option.
>
> 2.7.x releases will not receive this new feature, but it will be added to the
> next major release: 2.8.0 (already merged into GIT branch master).
>
> I will however keep updated the branch "no_first_slice" for it receives all
> fixes and modifications brought to future 2.7.x releases, up to the time release
> 2.8.0 will be available.
>
> OK this may sound complicated, but you can understand I don't want to waste too
> much time with the release process, which takes several hours to be done
> properly and in coherence between doc / code packaging / website(s) / emails...
>
> Cheers,
> Denis
>
> On 15/12/2023 19:30, Denis Corbin wrote:
>> On 14/12/2023 22:37, Raoul Bonnal wrote:
>>>
>> [...]
>>>
>>>>> OK, I could make it. You can checkout the code from git on branch
>>>>> "no_last_slice". I have to test is further but it seems OK so far. I don't
>>>>> know
>>>>> yet where I will merge those changes, probably into branch 2.8.x not 2.7.x,
>>>>> but
>>>>> will keep it updated beside branch_2.7.x until then.
>>>
>>>
>>> I can confirm that the branch `no_last_slice` works in our setting. We
>>> rebuild the source code using the debian test, borrowing the dar_2.7.13-1.dsc
>>> and the debian dir. I did a quick and dirty build to get a running
>>> dar_no_last_slice.
>> thanks for confirmation!
>> non-regression tests are still running and no issue has been met met so far.
>>
>>>
>>>
>>>
>>>>>
>>>>> This change leads dar to ask the first slice instead of the last slice only
>>>>> when
>>>>> an isolated catalog has been provided and the operation reads a backup in
>>>>> direct
>>>>> access mode (not in sequential read mode). In sequential read mode dar
>>>>> already
>>>>> fetch this info from the first slice, of course.
>>>>
>>>> I have some questions here:
>>>>
>>>> 1) When dar_manager is in use, does it always request the first or the
>>>> last slice?  My vague memory was that it doesn't, but this may not have
>>>> been accurate.
>>> I did not use dar_manager today (branch no_last_slice). Yesterday (w/2.7.13
>>> backport) it build the `dar -x` which in the end requested the last slice
>> I have double checked, dar_manager (no_last_slice) works properly with dar
>> backups. It only requires the last slices. And for an isolated catalogue, of
>> course it requires all the slices.
>>
>>>
>>>
>>>>
>>>> 2) If the needed data could be satisfied with either the first or the
>>>> last slice, could this be gated behind an option or something?  For
>>>> consistency, I have built workflow around assuming the last slice is
>>>> always needed.  I suppose I could always just not supply a catalog and
>>>> get that existing behavior.  But does this change how dar_manager
>>>> behaves?
>>>
>>>> 3) Could the remaining missing information be added to the catalog so we
>>>> need neither the first nor the last slice when the catalog is supplied?
>>
>>>
>>> This was my original inquiry. I have to admit that the "first slice" is more
>>> than reasonable to us.
>>> I can imagine that injecting all the information into the catalog may require
>>> more work and tests.
>> yes, so it will not be inside the catalogue, but probably beside it
>>
>>>
>>>
>>>>> So there is changes from user point of view. Let me know if someone has any
>>>>> object to this... I can still surface this as a new parameter for user to
>>>>> decide, but would like to avoid this extra work, if it is not necessary.
>>> +1 to avoid extra work, but as mentioned by John many may have
>>> consolidate pipelines.
>> yep, that's why I'm asking before merging this change.
>> Back to the other request asking to store of the archive format information in
>> the isolated catalog archive, to avoid dar asking the first slice at all while
>> reading an archive, I've hit another rock on the road: the sar layer.
>> This layer is used like a file with an open() phase, a read() phase and last a
>> close() phase. At open() time you can specify whether to open it by the end of
>> by the beginning. Thus even if we have all the above information about the
>> archive format from the isolated catalogue archive, when reading it comes to
>> the turn to read the main archive, the sar layer has first to be opened() to
>> fetch the slicing layout ( { slice size, first slice size, and a few other
>> parameters } ) and thus read either the first or the last slice...
>> A better approach would be to delay the time to fetch the slicing layout when
>> a read() request comes, but then at read() time, how to know which slice to
>> fetch since the slicing layout is unknown? The catalog only stores offset of
>> information as offset in this virtual big file emulated by the sar layer, this
>> offset has to be converted into slice number and slice offset based on the
>> slicing information.
>> One could say it could be possible to store the slicing information of the
>> archive of reference into the isolated catalogue archive (beside the catalogue
>> and archive format information), but this would break the ability to re-slice
>> an archive and use it whatever is its slicing pattern (see dar_xform) even
>> after the time the isolated catalogue has been created. I could even lead dar
>> fetching data from the wrong slice/offset while restoring... hopefully the CRC
>> check will trigger that but from user standpoint this would be ugly.
>> In consequence, I see no proper way, without breaking any feature to avoid dar
>> asking either the first or the last slice when reading an archive using an
>> isolated catalogue, method that is precisely used to avoid having to fetch the
>> last slice...
>> The best way in that context (read with external catalogue) is to fetch the
>> first slice instead of the last one, first slice which can be set small (see
>> -S option) on purpose. I'll probably make an option of that, to not break 21
>> years of history and usage :)
>>
>>>
>>> Cheers,
>>> R.
>>>
>>>
>> Cheers,
>> Denis
>>
>
> [[End of PGP Signed Part]]