Re: Future directions: Scalability, partial recoverability & Windows

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thomas Jacob a écrit :
> Hello Denis & the list,

Hello Thomas,

> 
> Concerning your invitation for comments on the future development 
> directions of DAR, I am describing a couple of points below I'd be 
> glad to see addressed a bit more in the future.
> 
> Scalability ~~~~~~~~~~~
> 
> This concerns the scalability of DAR to archiving many millions of 
> files.
> 
> One thing here is that the amount of main memory consumed becomes a 
> limiting factor at one point, probably also because of DAR's complex 
> data structures but mostly because it keeps the whole directory tree 
> in main memory. If you want DAR to create an index catalog (as I 
> usually do), it is of course necessary to keep the data around 
> somewhere, but it doesn't have to be main memory which usually is 
> more finite than disk space and also is used for a lot of other 
> things.

Yes, but this is dar's design. 8 years ago, the bet I did was that
average number of file to backup would increase much less faster than
the amount of data to backup (disk space), while the ratio of RAM to the
disk space would at least stay constant or better, increase.

For standard PC and average hardware configuration (thus we do not
consider UFO like Cray and other massively parallel computers) :
* 20 years ago, we could find less than 1 MB of RAM for 40 MB or disk:
ratio = 1/40
* 15 years ago, we could find 4 Mb of RAM for 250 MB of disk: ratio = 1/62
* 10 years ago, we could find 64 Mb of RAM for 2 GB of disk: ratio = 1/32
* 5 years ago, it's was possible to find 1 GB RAM for 80 GB of disk:
ratio = 1/40
* today, 4 GB of RAM and 200 GB of disk: ratio = 1/50

Well, since 20 years, ignoring the non statistical aspect of my samples
 (1/40, 1/62, 1/32, 1/50), this ratio is still in the same range of
values. While the number of file a system contains has probably grown a
bit, the overall average size of file had exploded since 20 years, and I
see no reason for this evolution to revert, as there is too much more
interest for vendors having software more fond of memory both for
software vendors and hardware vendors, under the pretext of "new
features"...

The other aspect is that dar does not need any temporary file, it can
work using pipes on a completely read-only system.

Well, in brief, seen the dar design, if your RAM is exhausted for your
backup, better add or increase swap space, it will not take more than
what dar would need on disk and it would be faster.

> 
> Another thing is that some of the algorithms used by DAR do not scale
>  well if you have millions of files in one directory (which is
> usually a stupid thing to do in the first place, but sometimes you
> have no influence on this, so...). This can easily mean that DAR will
> spent hours with a 100% CPU load sorting thru it's own data
> structures. But as I recall from a previous discussion of this
> problem, this is being addressed already for the the 2.4.X releases.

Yes, it is.

> 
> In general, I also have the impression that DAR is taking 
> significantly longer than some commercial backup software to archive 
> large numbers of small files, but I have no data to back this up, and
>  there are so many things that can influence such a comparison 
> (compression, whether or not the slices are (temporarily) stored on 
> the same system or even the same disk array, so maybe it's just me 
> imagining things here.

Maybe, but in any case, there are choices to do between speed,
portability, ability to evolve and robustness mainly. I will always give
priority to the ability to evolve, then robustness, then portability
only then to speed.

This does not mean that speed is not my concern, I mean that once all
the previous parameters have played their role, I will then optimize the
code for speed. The idea is that CPU power, disk access time do increase
as never, while robustness, portability will never, unless you
re-implement the code properly. For the ability to evolve, just review
the feature that were present in dar-1.0.0.

In conclusion for this point: What would be the use of fast but
unreliable code to restore my data after a crash? What would be
the interest for me in coding again the same features just for another
system? What would the use of a software that cannot get any new feature?

But you are right, there is still room for improvement in dar's code
architecture.

> 
> 
> Partial recoverability ~~~~~~~~~~~~~~~~~~~~~~
> 
> As far as I am aware, if your final slice somehow gets corrupted, all
>  your data basically becomes worthless. For instance, imagine that 
> the backups are stored on an external storage area with a quota limit
>  for each user. If you somehow get to barely over the quota limit, 
> your whole backup is worthless. Yet often one needs to restore only 
> one or two files (backups are not just made for disaster recovery, 
> but also for recovering from "human" errors). Maybe if the directory 
> info were available in the slices, one could still recover the 
> required files even if some parts of the backup have become 
> corrupted.
> 
> A similar issue arises when you store DAR backups on media which 
> might become slightly damaged (DVDs?). Distributing the directory 
> info among the slices would probably reduce the amount of files that 
> cannot be recovered in such a case.

First, in the backup process, as stated in dar's documentation and
repeated all along the support mailing-list, it is asked to test a
backup before relying on it (even better is doing diff, or even better a
restoration + diff of the restored and original filesystems) This is
especially to face human error, media corruption and dar's possible bugs.

Then the second point if you plan to place you backup on unreliable
medium is to make use of parchive to recover the backup in case of
corruption. Would you just jump from an air plain with the back parachute?
Neither me, so some improvement have been made as last resort, for
archive corruption recovery, which are now available in dar's
development code:
- - you can now use an extracted catalogue to rescue the catalogue inside
an archive.
- - you can use the sequential reading mode in addition to direct access
mode (usual mode of reading), which mode does not rely on the catalogue
- - you can read normally or sequentially an archive activating the "lax"
mode, which when a corruption occurs will try to find other slices to
recover the data, ask the user for information (compression used, if
this field is corrupted, dar's version used to create the backup, etc.),
continue upon error, warning the user, and so on.
- - In sequential and lax mode, you can use an arbitrary slice of an
archive set and recover all file that is completely stored in that slice
without needing the other slices.

All theses feature are implemented and will be available for 2.4.X but
are, for the "lax" mode in particular, to be use as very last resort.
Still better is to test your archives and use Parchive to protect them
from corruption by media error.


> 
> 
> Native Windows support ~~~~~~~~~~~~~~~~~~~~~~
> 
> This will probably not interest too many people, but anyway ;)
> 
> At present DAR can be run on Windows systems using the Cygwin 
> compiled binary quit well, if not exactly as fast as on Unix systems 
> (Cygwin eats a lot of resources).

Speed is not my concern when it has a huge cost for portability and when
 I have not the time nor the interest in doing twice the same thing, once
for Unixes a second time for windows. Doing twice is not only
implementing twice, it is also testing twice, maintaining twice, and
keep both implementations compatible. Moreover, I don't think much
people are using command-line tools under windows, moreover for backing
up data, ...

> 
> Unfortunately meaningful backups usually also need all file 
> attributes (ownerships, ACLs etc.), but since Cygwin tries to map the
>  Windows ACL system onto the standard Unix system with rather minimal
>  effort (At least in the Cygwin versions I've been using), the 
> permission data in DAR backups is basically worthless if you really 
> want to recover more than a single file.
> 
> If one would port DAR to run on Windows natively storing Windows ACLs
>  probably wouldn't be that hard (after all POSIX ACLs are already 
> being stored by DAR). Also this might make porting DAR to non-Unix, 
> non-Windows architectures more easy (If anyone ever wanted to do that
>  ;). DAR would also run faster on Windows because of that.
Anyone is welcome! Anyone will have to test, synchronize, update, the
feature added in dar into the windows port.

> 
> But maybe I am underestimating the difficulty of such a port, since 
> DAR isn't just relying on C++ standards but also on things like GNU 
> gettext.

Dar also relies in external libraries, for compression (libz, libbz2,
liblzo) or cyphering.

> 
> 
> ~~~~~~~~~~~~~~~
> 
> Still, I realize that DAR isn't (yet ;) meant to be a complete 
> replacement for commercial backup software in all respects, and Denis
>  is doing almost all of the work, and his spare time too.

Dar's target is not to replace commercial backup! It is to be useful for
anyone sharing the same needs as I have.

> 
> Anyway, in terms of scriptability DAR is already far ahead of all the
>  commercial Backup systems I am aware of.
> 
> But since Denis apparently is planing ahead quite far into the future
>  (cf. infinint or the "(C) 2002-2052" ;)...

;-)  Well, a small word about this copyright. I did not wanted to change
it every year, as I have many other thing much more interesting to do
than that! Thus I found that 50 years would be large enough for the time
I have planned to allocate to dar. But who knows what I will do
tomorrow? Not me... For that reason, I have put a lot of documentation
beside and inside the code.

> 
> Thomas
> 
> 

Regards,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFLcVplpC5CI8gYGlIRAnFBAKCdXwgoYvrxKdb6gaRIS3GkqjEqcwCfYuEa
KoaB2JM5eieMS+bPaFT27nI=
=0Dhb
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com