Future directions: Scalability, partial recoverability & Windows

Hello Denis & the list,

Concerning your invitation for comments on the future
development directions of DAR, I am describing
a couple of points below I'd be glad to see addressed a bit more
in the future.

Scalability
~~~~~~~~~~~

This concerns the scalability of DAR to archiving many millions
of files. 

One thing here is that the amount of main memory consumed
becomes a limiting factor at one point, probably also because of DAR's
complex data structures but mostly because it keeps the whole
directory tree in main memory. If you want DAR to create an index
catalog (as I usually do), it is of course necessary to keep
the data around somewhere, but it doesn't have to be main memory
which usually is more finite than disk space and also is used
for a lot of other things.

Another thing is that some of the algorithms used by DAR do not
scale well if you have millions of files in one directory (which
is usually a stupid thing to do in the first place, but sometimes
you have no influence on this, so...). This can easily mean
that DAR will spent hours with a 100% CPU load sorting thru
it's own data structures. But as I recall from 
a previous discussion of this problem, this is being addressed
already for the the 2.4.X releases.

In general, I also have the impression that DAR is taking
significantly longer than some commercial backup software
to archive large numbers of small files, but I have no
data to back this up, and there are so many things that
can influence such a comparison (compression, whether or not the slices
are (temporarily) stored on the same system or even the same disk array,
so maybe it's just me imagining things here.


Partial recoverability
~~~~~~~~~~~~~~~~~~~~~~

As far as I am aware, if your final slice somehow gets corrupted, all
your data basically becomes worthless. For instance, imagine that
the backups are stored on an external storage area with a quota limit
for each user. If you somehow get to barely over the quota limit, your
whole backup is worthless. Yet often one needs to restore only
one or two files (backups are not just made for disaster recovery,
but also for recovering from "human" errors). Maybe if the
directory info were available in the slices, one could still recover
the required files even if some parts of the backup have become
corrupted.

A similar issue arises when you store DAR backups on media which
might become slightly damaged (DVDs?). Distributing the directory
info among the slices would probably reduce the amount of files
that cannot be recovered in such a case.


Native Windows support
~~~~~~~~~~~~~~~~~~~~~~

This will probably not interest too many people, but anyway ;)

At present DAR can be run on Windows systems using the Cygwin
compiled binary quit well, if not exactly as fast as
on Unix systems (Cygwin eats a lot of resources).

Unfortunately meaningful backups usually also need all
file attributes (ownerships, ACLs etc.), but since
Cygwin tries to map the Windows ACL system onto
the standard Unix system with rather minimal effort
(At least in the Cygwin versions I've been using), the permission
data in DAR backups is basically worthless if you really
want to recover more than a single file.

If one would port DAR to run on Windows natively storing
Windows ACLs probably wouldn't be that hard (after all
POSIX ACLs are already being stored by DAR). Also this
might make porting DAR to non-Unix, non-Windows architectures
more easy (If anyone ever wanted to do that ;). DAR would
also run faster on Windows because of that.

But maybe I am underestimating the difficulty of such
a port, since DAR isn't just relying on C++ standards
but also on things like GNU gettext.


~~~~~~~~~~~~~~~

Still, I realize that DAR isn't (yet ;) meant to be a complete
replacement for commercial backup software in all respects, and Denis
is doing almost all of the work, and his spare time too.

Anyway, in terms of scriptability DAR is already far ahead
of all the commercial Backup systems I am aware of.

But since Denis apparently is planing ahead quite far
into the future (cf. infinint or the "(C) 2002-2052" ;)...

        Thomas


------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com