Re: Future directions: Scalability, partial recoverability & Windows

On Tue, 2010-02-09 at 13:51 +0100, Denis Corbin wrote:

> Well, in brief, seen the dar design, if your RAM is exhausted for your
> backup, better add or increase swap space, it will not take more than
> what dar would need on disk and it would be faster.

In general, I'd agree with the view that the ratio of average file
numbers to available main memory on your average
Desktop PC probably hasn't changed much.

But there other applications. Imagine a standard file server which
contains 10 million small files, most of which are never really looked
at (read email). Not a good setup but cheap, and the customer likes
it that way ;) Now the amount of memory in the system is more
than sufficient to provide appropriate caching for the "hot set"
of data currently being accessed. And the system has a certain
base load, if lots of swap space would be required, that
system will basically become unusable. BTW, in these
kinds of scenarios the system never really can be backed up
in times of no load, so DAR being a good citizen with respect
to the other processes running is important also. 

So the only solution at moment is to increase the amount of
memory in the system just do backups properly. 

> In conclusion for this point: What would be the use of fast but
> unreliable code to restore my data after a crash? What would be
> the interest for me in coding again the same features just for another
> system? What would the use of a software that cannot get any new feature?

None of course, you're priorities are sorted in the right
way, I agree. Just wanted to mention my impression
(which might be totally false :)

> Neither me, so some improvement have been made as last resort, for
> archive corruption recovery, which are now available in dar's
> development code:
> - - you can now use an extracted catalogue to rescue the catalogue inside
> an archive.
> - - you can use the sequential reading mode in addition to direct access
> mode (usual mode of reading), which mode does not rely on the catalogue

Great so the filenames are actually available in the slices as well?

> - - you can read normally or sequentially an archive activating the "lax"
> mode, which when a corruption occurs will try to find other slices to
> recover the data, ask the user for information (compression used, if
> this field is corrupted, dar's version used to create the backup, etc.),
> continue upon error, warning the user, and so on.
> - - In sequential and lax mode, you can use an arbitrary slice of an
> archive set and recover all file that is completely stored in that slice
> without needing the other slices.
> 
> All theses feature are implemented and will be available for 2.4.X but
> are, for the "lax" mode in particular, to be use as very last resort.
> Still better is to test your archives and use Parchive to protect them
> from corruption by media error.

That sounds very interesting, I'll try that out ASAP. And of
course, testing the backups before hand is the better
strategy...

> Anyone is welcome! Anyone will have to test, synchronize, update, the
> feature added in dar into the windows port.

Yeah, the problem in a nutshell, lot's of work to port, and almost
no users on that platform ;)

> C) 2002-2052" ;)...
> 
> ;-)  Well, a small word about this copyright. I did not wanted to change
> it every year, as I have many other thing much more interesting to do
> than that! Thus I found that 50 years would be large enough for the time
> I have planned to allocate to dar. But who knows what I will do
> tomorrow? Not me... For that reason, I have put a lot of documentation
> beside and inside the code.

Hehe, of course. Keep up the good work. Eagerly looking forward to
the 2.4.X release.



------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com