Re: Huge archive and last slice

Patryk Bełzak <[email protected]> Wed, 13 Dec 2023 09:11:09 +0100
Newsgroups	gmane.comp.sysutils.backup.dar.support
Message-ID	<vhbjfu2lwltizx3cjbqc3gstzakesfqwogwmuoedzo3gu77lpp@xzlbet2fjujy>
I guess it's also worth to mention `dar_manager`.
I'm also dealing with PB of scentific data in Hierarchical Storage Management solution, and I'm considering scenario where I'll build a database on some nfs (so it could be accessed by all systems I need, and getting some sort of high availability).
By doing so, searching for files and extracting them wouldn't require to recall large amounts of data from tape library to cache filesystem.
The other trick is that I'll need to provide this information (archive contents) to users, I guess I'll mount the database share in Read Only mode to make sure they won't mess it up.

Kind regards,
Patryk.

On 23/12/12 08:48PM, Denis Corbin wrote:
> Thank you John for your pertinent remarks and continuous support ;)
> 
> On 12/12/2023 20:22, John Goerzen wrote:
> > On Tue, Dec 12 2023, Denis Corbin wrote:
> > 
> > > 
> > > > I was wondering if it is possible to "extract" from the last slice only the
> > > > relevant information needed for the recovery. I can understand that is it
> > > > convenient to keep them in the last slice because usually it is the smallest
> > > > one, but in case of huge archives its size can be relevant.
> > > 
> > > Yes, the is precisely the isolated catalog.
> > 
> > To amplify this point a bit:
> > 
> > Every dar archive contains a "catalog", which is added at the end of the
> > last slice, and is the directory information (including slice number) of
> > every file within the archive and other assorted metadata.
> > 
> > You can also have an "isolated catalog", in which only the catalog
> > portion is saved to a separate file.  This is exactly what you want.
> > 
> > There are two common ways to create this catalog:
> > 
> > 1) At the same time you create the archive itself, with the
> > --on-fly-isolate option
> > 
> > 2) From an existing archive, using the --isolate option.
> > 
> > One use for this is to make incremental backups without needing to have
> > the previous backup available.  You can simply reference its isolated
> > catalog, and then dar will figure out what to do.
> > 
> > I have some examples of this on my blog here:
> > 
> > https://changelog.complete.org/archives/10527-using-dar-for-data-archiving
> > 
> > (ignore the bits about dar_manager as they are irrelevant to what you're
> > asking)
> > 
> > - John
> > 
> > 
> > 
> > > 
> > > > PS: I must thanks John Goerzen for his support in setting up dar w/threads
> > > > during my tests.
> > > > Bests,
> > > > R
signature.asc (application/pgp-signature, 833 B)