Re: Stand-alone slices

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[email protected] wrote:
>>> The slicing scheme is retrieved from the first slice, and the
>>> catalogue from the last slice.  If the file in question resides
>>> completely in the 4th slice, then the restore proceeds.
>>> Otherwise, dar asks me to provide the xth slice and I cancel the
>>> operation.  The end result is that I restore all the files in a
>>> single slice.
>> Yep, but this is possible only if you know which files not to
>> restore because they have data located in a missing slice.
>> 
> 
> I try to restore all of the files in the archive.  If dar prompts me
> for a slice, then I know that the file I'm trying to restore does not
> exist wholly in the slice I have.

OK, I see.

> 
> 
>>> My proposition is to replicate my "by hand" method above.  We try
>>> to restore all the files listed in the catalogue.  When a missing
>>> slice is needed dar should not prompt the user, but silently fail
>>> in the restoration and move on to the next file in the catalogue.
>>> This could even be done with an external script providing that
>>> you could set dar to automatically exit rather than ask for a
>>> missing slice.
>> Yes, but even before trying to read data to restore a given file,
>> dar must first create the directory entry (inode) then tries to
>> copy data from the archive into the just created file. At this time
>> only it may fail to proceed due to the lack of slice. which lets in
>> the filesystem a empty file [I must double check what dar does to
>> the already created file, but I guess it is let in place
>> uncompleted or empty]
>> 
>> In your idea, you assume that dar tests the data location before 
>> deciding whether to restore or not a given file. But due to the 
>> implementation in layers (low layer manages files/slices, above
>> which takes place the virtual unsliced data, then above is
>> encryption, then again encryption, then at last file's data and
>> catalogue) It's not straight forward to have the possibility to do
>> this way. In particular, if the start of a Data is possible to
>> locate, due to compression and encryption, it is not easy instead
>> to know where is located its end (same slice or next slice that may
>> be missing for example). Thus you may end with a partially restored
>> file, even having this test implemented before the restoration of
>> each file.
>> 
> 
> I didn't realize that I had gotten so many empty files created in the
> directory.  This is a problem.  But it does seem like it would be
> fairly simple to let the user specify that the files should be
> deleted if the recovery is not successful.

Yes, I agree.

> 
> The other problem would be a file that spans multiple slices and gets
> partially restored.  Would this ever occur more than once per slice?
> 

At most you have two files that span can have partial data in a given
slice. The first has the other part of its data in the previous slice,
the second has part of its data in the next slice (such a slice is thus
neither the first nor the last).

> 
>>> A more efficient way would be to allow the catalogue to be
>>> trimmed to the contents of a specific slice.  That would
>>> eliminate the repeated attempts and failures, but it would
>>> require some extra code and would have to be regenerated in the
>>> event of a reslice.
>> Of course that would require extra code, thus that's an idea to
>> think about. However what I like is the possibility to
>> transparently reslice (using dar_xform) an archive without having
>> to rebuild an extracted catalogue (you can also slice or reslice an
>> extracted catalogue and keep using it against the same original or
>> resliced archive).
>> 
>> IMHO it is much more complicated to correctly trim a extracted
>> catalogue to fit a given slice than to emulate a missing slice
>> (over the fact it does not really match the current software
>> design). The only thing that must be seen more deeply within the
>> current dev. implementation is the possibility to test the start of
>> data location before restoring a given file. I guess (and hope)
>> this should be possible to implement cleanly in coherence with the
>> existing software architecture... Still remains the case of file
>> that could not be restore entirely and the need of a new optional
>> feature to let the user decide whether to remove them from the 
>> filesystem or let them uncompleted restored.
>> 
> 
> The testing of the data location would be good.  Really, couldn't you
> just have an additional prompt for incomplete files.  "dar: provide
> the correct slice"   "user: I don't have it... ESC"  "dar:file not
> complete do you want to delete it?"

Something like that, but I know that some users will also ask for a
command-line switch to not be bothered answering "yes" or "no" many time
to the same question about different files.

> 
> As for the trimming of the file, I have only 1 idea that may not be
> any good.  I'm not sure how the "on-the-fly"  isolation works.

at the end of the archive creation, once it is closed, dar does not free
the space hold by the catalogue in memory and use it to modify it to an
extracted catalogue and dump it into a new archive.


>  I
> have seen that the catalogue file is not written until all slices are
> finished.  But I must assume that the data is collected as the slices
> are made.  Therefore, at the end of the xth slice, you would have all
> the data for the first x slices and no more.  So it might be possible
> to insert a flag when each slice is finished into an on-the-fly
> catalogue.  This is more speculative since I don't know how it's
> implemented.

In theory and if we ignore the problem of files that are span over two
or more slices, it could be possible to generate as many catalogue as
slice generated, but it would break a major feature and design, which
rely on the independence of the slice layout and the archive layout:
this let let dar_xform reslice an archive without any knowledge of the
archive contents or even archive structure, and dar_slave let dar
remotely access an archive without having to transfer it (only what is
necessary is read and transfered). For this reason this feature is not
possible to be implemented.


> 
> Bill


Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJh16UpC5CI8gYGlIRAj3TAJ9k5kVaF7MVe8+yHyJsn5Zjn5tvJQCff2t9
eF9VbRCRlLrfUPqU+O8+Rz0=
=NYht
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com