Re: Huge archive and last slice

--

Raoul Bonnal
IFOM | Research Computing and Data Science Manager
Ph.: +39 02 57430 3016 | raoul.bonnal-G+fEVLdQ/[email protected]<mailto:raoul.bonnal-G+fEVLdQ/[email protected]>
www.ifom.eu<https://www.ifom.eu> | Linkedin<https://www.linkedin.com/school/ifom/> | Instagram<https://www.instagram.com/ifom_milan/?hl=it> | Facebook<https://www.facebook.com/ifom.eu> | Twitter<https://twitter.com/ifomresearch> | YouTube<https://www.youtube.com/channel/UCOVlG5DhrDWMR1N5NYBB4RQ>
IFOM ETS - The AIRC Institute of Molecular Oncology | Via Adamello 16, 20139 Milan, Italy

[cid:logoIfom_2d2161e0-3259-466c-8931-26f8c2a5979f.png]

________________________________

Confidentiality notice. This message, together with its annexes, contains information to be deemed strictly confidential and is destined only to the addressee(s) identified above who only may use, copy and, under his/their responsibility, further disseminate it. if anyone received this message by mistake or reads it without entitlement is forewarned that keeping, copying, disseminating or distributing this message to persons other than the addressee(s) is strictly forbidden and is asked to transmit it immediately to the sender and to erase the original message received.
Thank you
Save a tree - Do not print this email unless absolutely necessary

________________________________
From: John Goerzen <[email protected]>
Sent: Thursday, December 14, 2023 6:32 PM
To: need help compiling or using dar ? This mailing list is for you.
Cc: Denis Corbin
Subject: Re: [Dar-support] Huge archive and last slice

On Wed, Dec 13 2023, Denis Corbin wrote:

>> OK, I could make it. You can checkout the code from git on branch
>> "no_last_slice". I have to test is further but it seems OK so far. I don't know
>> yet where I will merge those changes, probably into branch 2.8.x not 2.7.x, but
>> will keep it updated beside branch_2.7.x until then.


I can confirm that the branch `no_last_slice` works in our setting. We rebuild the source code using the debian test, borrowing the dar_2.7.13-1.dsc and the debian dir. I did a quick and dirty build to get a running dar_no_last_slice.



>>
>> This change leads dar to ask the first slice instead of the last slice only when
>> an isolated catalog has been provided and the operation reads a backup in direct
>> access mode (not in sequential read mode). In sequential read mode dar already
>> fetch this info from the first slice, of course.
>
>I have some questions here:
>
>1) When dar_manager is in use, does it always request the first or the
>last slice?  My vague memory was that it doesn't, but this may not have
>been accurate.
I did not use dar_manager today (branch no_last_slice). Yesterday (w/2.7.13 backport) it build the `dar -x` which in the end requested the last slice


>
>2) If the needed data could be satisfied with either the first or the
>last slice, could this be gated behind an option or something?  For
>consistency, I have built workflow around assuming the last slice is
>always needed.  I suppose I could always just not supply a catalog and
>get that existing behavior.  But does this change how dar_manager
>behaves?

>3) Could the remaining missing information be added to the catalog so we
>need neither the first nor the last slice when the catalog is supplied?

This was my original inquiry. I have to admit that the "first slice" is more than reasonable to us.
I can imagine that injecting all the information into the catalog may require more work and tests.


>> So there is changes from user point of view. Let me know if someone has any
>> object to this... I can still surface this as a new parameter for user to
>> decide, but would like to avoid this extra work, if it is not necessary.
+1 to avoid extra work, but as mentioned by John many may have consolidate pipelines.

Cheers,
R.