Re: isolating catalogs for robust data protection

Robert Dale wrote:
> Hello,

Hello,

> 
> I was disappointed to read that someone on the support list was not
> able to recover his data because he lost his last slice.  I wasn't
> disappointed because he couldn't restore his data, although I feel for
> him - been there, done that -

Same thing to me here,

> but because in response he was told it
> would not have mattered if he had an extra copy of the isolated
> catalog.  This seems to take away greatly one of the benefits of using
> dar:  one corrupt section of data does not ruin the whole archive.
> 
> I have just started using dar and one thing I was expecting to be able
> to do was put a copy of the catalog with each slice onto DVD.  So in
> the event that the last DVD should get destroyed or even just
> scratched in the wrong place, I could restore the majority of my
> backup.

This would not help as you have guessed.

> 
> The -A option is already there!  dar -x archive -A catalog.

-A option with -x does nothing. -A option is only useful with -c -C or 
-+ commands. (You have probably read dar's man page and found that the 
-A option is listed under the "SAVING, ISOLATION AND MERGING OPTIONS" 
title).

> 
> Now, like I said, I'm new to this, but if he would have added that
> catalog to dar_manager,
> would he still need the last slice?

yes. Information stored in dar_manager is a small subset of what you 
will find in a catalogue, it is not enough to restore an archive 
(dar_manager relies on dar and the archive, once it has located and 
sorted archive by archive the files to restore).

>  But is this a solution or a mere
> side-effect?

using Parchive with dar is *the* solution for your need.
for more: http://dar.linux.free.fr/doc/Notes.html#XIV

> 
> I'm _not_ suggesting that a catalog _should_ be maintained with each slice.
> I'm asking that -A lets me use an isolated catalog to restore from any
> given slice.  Then I can choose whether or not I want to isolate the
> catalog and make copies.

I understand. But ...
For homogeneity, an "extracted catalogue" is a real archive. It does not 
contain a copy of the catalogue that is located in your original 
archive. The original purpose of the isolation feature was to be able to 
make a differential backup without the need to provide the archive of 
reference (which may be huge). The isolated archive is exactly the one 
you would get if you did a differential backup just after a full backup 
(all information about the covered files is present except their data 
because the files did not changed since the time archive of reference 
was done).

Extracted catalogue is nothing more nothing less than that.

Now, having an extracted catalogue be used to rescue a corrupted 
catalogue is today not possible, but I add this feature in the list of 
things to do. :-) It seems possible to have this feature but with very 
important changes in the fundamental design of dar. :-/ I think that 
worth it.

> 
> We still have one more problem:  the first slice.  dar requires the
> first slice.  But why do we really need the first slice?  We don't
> need it for decompression.

Wrong, we need it for decompression, as the catalogue is compressed we 
must know the compression algorithm used., this information is stored 
int the only area that is not compressed : the archive header (located 
in the first slice(s))

> We don't need it for decryption.

also wrong, we must know whether an archive is encrypted or not, the 
only place where something is not encrypted is in the archive header 
too. This header also contains the archive revision format which defines 
the structure of the archive (which has changed when new feature have 
been added).

> We don't
> need it to find out if we're dealing with slices or not.

Also wrong. Dar stores the location of a given file's data by an integer 
(= an offset) telling how much bytes must been skipped from the 
beginning of the archive to reach the first byte of the requested data. 
The archive header contains the information of the slice size and first 
slice size. Note that last slice usually has a smaller size than the 
previous one, so in an archive you may have three different slice sizes 
(first, last, and intermediate slices). To know in which slice a given 
offset falls, we must know theses three sizes.


>  We need the
> first slice only to find out four little bits of information -
> version, compression algorithm, command line, and if EA was saved.

in parenthesis, the two latter fields are not more used in recent 
versions of dar.

You also forget a reason : software design. For more about that point 
please read  http://dar.linux.free.fr/doc/Notes.html#IV

> So, if the first slice were to be corrupted or the archive header
> mangled in some way, we then have the same problem with the whole
> archive becoming useless. 

absolutely.

> Replicating the archive header on each
> slice would make a negligible impact on file size while greatly making
> the archive more robust.

but where to replicate it? OK, you will say, at every N bytes of data.
But how to resize an archive then? dar_xform could not survive while it 
is very interesting. (this is the software design referred above).

> 
> With these features,  I could use the first disc to do a full restore
> without having to insert discs _three_times_. 

twice, not three times and only for the first and last slice (or disk if 
disk = slice).

> More importantly, I
> could restore any random, surviving slice without changing discs at
> all.
 >
 > Comments?
 >

This is really rare that a file's data ends at the end of a slice and 
thus that the beginning of each slice as not the end of a file stored in 
the previous slice. right? Having slice able to be autonomous 
extractable structure, would not help theses files, right?

Now, when creating an archive, you fill each slice with the data of 
files. When the size is reached, the slice is closed and the next slice 
is started to be able to drop the rest of the data. If you want to have 
a "autonomous extractable slice", you have to stop filling the slice a 
bit before the end to drop the information necessary to handle the slice 
data. This amount of information depends on the number of file and the 
length of the filenames, etc. How much byte "a bit before" have you to 
stop ? That will make things a bit more complex. right?

Now, assuming you could find an algorithm to stop at as requested above 
a bit before filling the slice to drop the amount of data that will let 
you handle the slice content's. At restoration time, how will you know 
where to start reading this information? Things get even a bit more 
complex right?

Now you must consider another aspect. for a given file, only the 
filename is stored in the catalogue. The directory tree structure is 
stored in the way the file information is put in sequence. (read the 
link given above for detailed information). This makes things very 
efficients in storage and in computation. Now, if you have to store a 
portion of the archive in each slice you must say in which 
directories/subdirectories you are, to be able to know where to restore it.

Now, that you have your catalogue spread at the end of each slice, how 
can you reslice a given archive? Bigger slices will make spread portions 
of catalogue move in random part of each slices (even you will have 
several portion on the same slice), while resizing to smaller slices, 
may lead some slice without any archive portions). Moreover archive 
portions would not match the slice contents ...

Yet another point. As the location of file's data is defined by an 
offset from the beginning of the archive, you would need to replace that 
to a slice number+offset from the beginning of an *slice* to be able to 
restore a single slice. This would here too avoid resizing an archive.

Resizing (what does dar_xform), is an interesting feature: it let you 
manipulate an archive without uncompressing or decrypting it, it let you 
produce an archive to a pipe, which can be sent remotely throw ssh or 
other means to a remote location, where there it can be sliced at will 
and burnt on CD/DVD or stored on floppies if you wish. It also let you 
stick together many slices back to an single huge on-sliced archive, if 
for example you had sent int by email (by 2MB slices) with a script for 
example.

To my point of view, dropping this feature for having something more 
complicated (and thus less robust) does not worth it. While Parchive 
does the exact thing.

Now, in the planned works there is a point that has for object to see 
what's possible to do to have dar able to only request the last slice of 
an archive before requesting the necessary slices to fetch the data. 
This is a major redesign, but that's something I am willing to do.


> Thank you,
> --
> Robert Dale
> 
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEOEKFpC5CI8gYGlIRAg89AKDACvpQ/BHLQ4QyC6fSgjoJvSYqIwCgvxgJ
Zry0oXskWRzPG6DjAxbICZE=
=CUE1
-----END PGP SIGNATURE-----