Re: Dar architecture questions

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 13/10/2012 20:37, Kevin Wormington wrote:
> Hi,

Hi,

> 
> I have recently started using dar to backup a fairly large number of 
> files (it's maildir stuff) which consists of approx 136G of data in 
> ~1.68 million files.  I must complement dar since it is one of the few 
> programs I tried that could actually complete an initial backup in a 
> fairly reasonable period of time...it took just under 10 hours.

Interesting feedback, thanks,

> 
> I have just started testing making a differential backup using the 
> catalog from the full backup that was generated on the fly.  It works 
> quite well but I do have some questions:
> 
> 1) It takes 46 minutes if I am generating an on-the-fly catalog and 30 
> minutes if not.  I haven't looked through the source yet, but I'm 
> curious why there would be such a difference since it appears the 
> catalog is kept in memory.

Thanks pointing at this. After reviewing the code I found that
implementation could probably be simplified about this feature.

In release older than 2.4.0, the isolation process was a modification of
the archive contents (aka "catalogue"). We speak here about C++ classes
and objects: the "archive" objects owns a "catalogue" object. Modifying
that catalog would lead to an incoherence between the catalog and the
other "archive" class's object (objects for slicing, compression,
encryption, etc.). So the isolation operation is realized by the
creation of a new archive object with partial data from the original
"catalogue" object.

Since release 2.4.0, the archive isolation still creates an new archive
object which this time rawly clones the whole catalogue object because
it must now be able to be used as backup of the internal catalogue in
case of corruption in the original archive (and not only as a
replacement of the original catalogue for differential backup as it was
before 2.4.0).

So the isolated catalogue must keep the information of the offset where
to find a particular file's data and EA in the archive. There is just a
flag and the "internal archive signature" taken from the original
archive that are added into the isolated archive header. The flag avoids
a user trying to restore data from this archive alone (as it does not
carry any data). The signature matches the one of the original archive
which can then be used to get the data from, and avoid user trying to
restore data from an archive that the isolated catalogue has not been
taken from.

So, the simplification could result in the replacement of the special
"isolation" constructor for class archive, by an isolation method that
would create a isolated archive from the in-memory archive object (no
more need to copy anything). This would reduce memory and time
requirements for the isolation operation.

> 
> 2) On both test runs I made the stats were displayed after ~21 minutes 
> and the .dar slice does not appear to grow or be modified after this 
> time.  What is dar doing during this period?

I'm pretty sure this time is taken by memory management (releasing
memory here). Dar is very careful in regard to memory allocation and
release. So it takes time releasing memory and checks that everything
has been released once and only once. This has a cost, but the gain is
about bug detection and validation that the operation was done properly.

Yes, dar could simply stop and let the system cleanup the whole mess,
but that's my way of making things, sorry.

> 
> I included the stats display and the time for each run below.  This is 
> dar v 2.4.8.
> 
> Thanks for a great software!

You are welcome! :)

> 
> Kevin
> 

Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUHrC0QgxsL0D2LGCAQIQYhAAmAljPUSkn32eMJMAS2tZdLYBJxmD3SiH
mZu/Xq5HX+SrXm0Mm92nz1gBeI+VOiWpIG25P3B6BylVa41vPIe7eZ0egfTQc/xH
/ofrWV97olOQneGgTc2xeVLn/6zngIk7MqoWW4xohg+PAw47liSRGICcWr92XN9s
csEMWqKlY220wAg8LQP3FTpXpEqPE/vwh/mfpw9Sv4m0Bv8xM+vEprIjEQpVcZrI
qdTZ/Kd2xDcBXK6DJyC9zb1CA9sjY8ZMOPHouc4/U5tqhz1F+N9JAx3jqAEu37tv
5FBvU9vmwkrUFy6KUUCcybvegaSe0Hkh72WHfVfLuMuIROl9pBMwL+khCRfIoAFO
vSeemLb7T3JKn0plGpaxttS/FSdjhNTrZsZ+1+YahNM4ndWglsHL0OyrBc2IdYw5
PPdv31Zm58VsdsEG//aM274iDFjlP+RtFIyes3fdZYZn83z8255cFajRUerWrl5v
omR70PWJS3tvRRhh96DXB5VewvgxMxEFrdBsysOCm69u8T58tFMzg6FEc/PE6Z/Z
zi4s6ARpHXhG4Ne0/sQ2O6m0ciEnZ/lHZa9LqWu1UY6+0QeO8nk2vSgps8KzMeO7
Wfzjz12WWJCC3pX5Wwpwwxh8SqIsGEBoh+M9KzGfzfjOBr08Ujt+x+FG19R7og+B
E0YdaGxUthQ=
=pZpF
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev