Re: Parchive building

T wrote:
> Hi,

Hello,

> 
> Have anyone done some statistics on the time it takes to create data
> protection parchive on each single slice or the whole dar backup?

it depends on the CPU power of your computer.

> I mean,
> I know the created parchive size would be not much different in size, but
> how about the creating time?

For what I understand of redundancy, the underlying algorithm build a 
matrix which first dimension is the size of the data to protect, and 
second dimension is equal to the first size plus a percentage 
corresponding on the redundancy ratio.

Calculating a huge matrix takes much more time than calculating several 
smaller matrix. I rawly guess that the Parchive algorithm is in O(n²), 
assuming you have p slices of r bytes in your archive, calculating 
parity data for each slice gives you somthing like O(p.r²) while 
calculating the parity for the whole archive gives you something like 
O(p².r²) ... thus the difference of execution time between the two ways 
of calculating parity data grows proportionnaly with the number of 
slice. In other words:

calculating the parity data slice by slice for an archive of 10 slices 
will take 10 times the amount of time needed to calculate the parity for 
one slice. But calculating the parity data globaly for the same 10 
slices archive will take 100 times the amount of time needed to 
calculate the parity for one slice.

I let you confirm us that the experimentation is in coherence with the 
theory ;-)

> 
> The reason that I'm asking is that, according to the defect theory, the
> defects will mostly likely to happen in concentrated places, instead of
> spreading evenly throughout the whole thing, which would be most unlikely to
> happen.

yes, this makes sens

> 
> So, for a 100 slices backup of 10M each, the maximum error the current
> approach can tolerate is 200K in each slice. But if the parchive is created
> over the whole dar archive, like what the usenet posting is doing, then the
> maximum error a single slice can tolerate is much more than that. When
> defect or scratches happens to CD/DVD backup media, mostly likely, half 
> of a slice will be gone.

No that's wrong. If you have a scratch on a CD/DVD you loose data 
'under' the scratch, not the one after. Of course you need some tool 
that do not abort when the meed the I/O error generated by the 
'scratch'. For that reason I have developped 'dar_cp', which can be used 
  as the unix 'cp' command but instead of it, when it meets an I/O 
error, it skips it replacing the data by zeros and continues reading 
after the error.

I have tested this with a CD-R I gave to my child. I was surprised month 
after month to not meet any I/O error, I just noted that the CD driver 
took more time to complete the reading but did not failed. However after 
the 11th month of bad care (my child --- 4 years old at that time --- 
had it without jacket liying between drawings) the first I/O appeared. I 
tried dar -t on the burnt archive and a file was not possible to 
restore. I then 'cp' all the data to hard disk but as the I/O was at 40% 
of the archive Parchive could not recover the archive. I finally 
'dar_cp' the archive to hard disk, which took almost a night as it takes 
time for the driver report an I/O (it makes several passes before 
aborting), then for the dar_cp to find the end of the I/O, testing 
different positions ... However once the slice was copied only less than 
1% of the data was missing, and Parchive could repair the file (thanks 
to the 6% parity data present on the disk)! I then ran dar -t on the 
repaired archive and all was OK, I could also restore the file that was 
impacted by the corruption.

See the CD aspect, I thought the I/O error would have occurred long 
before that 11th month! It was like an ice ring at the end of a day!

> If we create parchive over the whole dar archive,
> in example I just gave, we can still do a full recovery even if the two
> whole slices are gone.
> 
> Comments?
> 

Yep, that's a good idea if you have space to hold the slices for 
Parchive can work on them, and if you have time for Parchive can compute 
the parity on the whole set of slices.

At repair time, Parchive will ask you all the slices, thus you will have 
to copy them back to hard disk for the repairing can be done.

Regards,
Denis.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
Dar-discussions mailing list
Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/dar-discussions

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFku9gpC5CI8gYGlIRAkhvAJ40/79CNXgmJmpe4a0SCFqe9sx2FQCffkjq
IIZxcC6sCIP1beq6wC4GWDs=
=V7wo
-----END PGP SIGNATURE-----