Re: Huge archive and last slice

Denis Corbin <[email protected]> Tue, 12 Dec 2023 18:56:03 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<[email protected]>
On 12/12/2023 08:10, Raoul Bonnal wrote:
> Dear Denis,

Hi Raoul,

> 
> thanks for your software.

you're welcome

> 
> My scenario: I must archive scientific data and eventually retrieving a 
> small portion of the data. The data are huge, several tens of TBs so the 
> slices are very useful because we can divide the global archive in 
> pieces, then with the catalog we can identify the exact slice(s) in case 
> of need. Data (archives/slices) can be deposited in a remote site, so 
> optimizing the transfer (retrieving) of the data is a priority.
> 
> At first I thought that the catalog and the slices of interest were 
> enough. Then I realize from the dar messages, and confirmed by a msg on 
> the ml that also the last slice is required.

You can avoid relying on the last slice (which contains the table of 
content or the archive) using -A option while extracting, listing or 
testing your archive:

assuming you have big.238.dar and big.373.dar slices (from a 500 slices 
archive) which slices you know to contain the totality of the files you 
want to extract, and also have cat_big.1.dar the isolated catalog of 
this big archive:

dar -x big [-g/-P/... filtering options] -A cat_big

dar will load the table of content (aka catalogue) from cat_big and try 
to restore the files that match the filter. If they are all located in 
slices 238 or 373 which are present in the local directory, dar will 
find them and use them without asking the user and without the user 
required to download anything else.

You can also automate the process, by mean of -E option and do not 
bother about which slice to download, dar will do it for you:

dar -x big [-g/-P/...filtering options] -A cat_big -E "rm %b.*.%e ; scp 
[email protected]:/some/path/%b.%N.%e ."

-E option will first remove any old slices from the archive which 
basename is %b (substituted by what you provided to -x/-t/-l/... option)
then it will call scp (adapt to the protocol you rely on) to fetch the 
needed slice. As dar reads slices in increasing order, a given slice 
will not be asked more than once (dar could ask the last slice a second 
time if you were not relying on the isolated catalog)

Of course and as you know, you only need the isolated catalogue to know 
the slices needed for a file or even for several files:

assuming you need all files in subidrectory usr/share you get the list 
of slice to fetch by using:

dar -l cat_big -g usr/share -Tslice | tail -n 3


I guess, the confusion cames from the fact the overall size of the 
archive is not know from the isolated catalog and to get this 
information you need the last slice. But except that, the isolated 
cataloge holds all the necessary information needed to restore any file 
data, EA and FSA stored in the archive, including the global offset and 
the slice layout from which dar calculates the slice information (and 
file offset in the slice)

> 
> The point is: if the last slice is 500GB+ I need to keep it available or  > download it every time a file/directory must be recovered.

from the previous explanation, you now know that it is not necessary to 
download the last slice, the isolated catalog is sufficient for that.

> 
> 
> I was wondering if it is possible to "extract" from the last slice only 
> the relevant information needed for the recovery. I can understand that 
> is it convenient to keep them in the last slice because usually it is 
> the smallest one, but in case of huge archives its size can be relevant.

Yes, the is precisely the isolated catalog.

> 
> 
> PS: I must thanks John Goerzen for his support in setting up dar 
> w/threads during my tests.
> 
> 
> Bests,
> 
> R.
>
Cheers,
Denis
OpenPGP_signature.asc (application/pgp-signature, 840 B)