Re: Dar architecture questions

Dennis,

Thank you for the detailed reply.  I have done some more testing and you 
are correct that memory deallocation consumes a considerable amount of 
time after the stats are displayed.  I re-configured with 
--disable-special-alloc and the results (below) are quite dramatic. 
There was a ~600MB increase in memory usage but the execution time is 
down to less than 10 minutes.  It appears that memory allocation must be 
playing a significant part in the total backup time as well.  I am going 
to re-run the full backup and see what kind of improvement is has as 
well.  Hopefully I will also get a chance in the next few days to see if 
I can determine what it is about the special_alloc that causes the 
slowdown vs system libraries at least for my use case.

Kevin



stats display after ~8 minutes 2.5GB memory used
  --------------------------------------------
  24317 inode(s) saved
    including 6693 hard link(s) treated
  0 inode(s) changed at the moment of the backup and could not be saved 
properly
  0 byte(s) have been wasted in the archive to resave changing files
  1661495 inode(s) not saved (no inode/file change)
  0 inode(s) failed to be saved (filesystem error)
  2 inode(s) ignored (excluded by filters)
  7382 inode(s) recorded as deleted from reference backup
  --------------------------------------------
  Total number of inode(s) considered: 1693196
  --------------------------------------------
Freeing Memory
Freeing Memory Done
Start on-fly isolation
Isolation complete
X1 memory free
X2 return

real	9m42.090s
user	8m20.500s
sys	0m28.610s



On 10/14/2012 08:49 AM, Denis Corbin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Le 13/10/2012 20:37, Kevin Wormington wrote:
>> Hi,
>
> Hi,
>
>>
>> I have recently started using dar to backup a fairly large number of
>> files (it's maildir stuff) which consists of approx 136G of data in
>> ~1.68 million files.  I must complement dar since it is one of the few
>> programs I tried that could actually complete an initial backup in a
>> fairly reasonable period of time...it took just under 10 hours.
>
> Interesting feedback, thanks,
>
>>
>> I have just started testing making a differential backup using the
>> catalog from the full backup that was generated on the fly.  It works
>> quite well but I do have some questions:
>>
>> 1) It takes 46 minutes if I am generating an on-the-fly catalog and 30
>> minutes if not.  I haven't looked through the source yet, but I'm
>> curious why there would be such a difference since it appears the
>> catalog is kept in memory.
>
> Thanks pointing at this. After reviewing the code I found that
> implementation could probably be simplified about this feature.
>
> In release older than 2.4.0, the isolation process was a modification of
> the archive contents (aka "catalogue"). We speak here about C++ classes
> and objects: the "archive" objects owns a "catalogue" object. Modifying
> that catalog would lead to an incoherence between the catalog and the
> other "archive" class's object (objects for slicing, compression,
> encryption, etc.). So the isolation operation is realized by the
> creation of a new archive object with partial data from the original
> "catalogue" object.
>
> Since release 2.4.0, the archive isolation still creates an new archive
> object which this time rawly clones the whole catalogue object because
> it must now be able to be used as backup of the internal catalogue in
> case of corruption in the original archive (and not only as a
> replacement of the original catalogue for differential backup as it was
> before 2.4.0).
>
> So the isolated catalogue must keep the information of the offset where
> to find a particular file's data and EA in the archive. There is just a
> flag and the "internal archive signature" taken from the original
> archive that are added into the isolated archive header. The flag avoids
> a user trying to restore data from this archive alone (as it does not
> carry any data). The signature matches the one of the original archive
> which can then be used to get the data from, and avoid user trying to
> restore data from an archive that the isolated catalogue has not been
> taken from.
>
> So, the simplification could result in the replacement of the special
> "isolation" constructor for class archive, by an isolation method that
> would create a isolated archive from the in-memory archive object (no
> more need to copy anything). This would reduce memory and time
> requirements for the isolation operation.
>
>>
>> 2) On both test runs I made the stats were displayed after ~21 minutes
>> and the .dar slice does not appear to grow or be modified after this
>> time.  What is dar doing during this period?
>
> I'm pretty sure this time is taken by memory management (releasing
> memory here). Dar is very careful in regard to memory allocation and
> release. So it takes time releasing memory and checks that everything
> has been released once and only once. This has a cost, but the gain is
> about bug detection and validation that the operation was done properly.
>
> Yes, dar could simply stop and let the system cleanup the whole mess,
> but that's my way of making things, sorry.
>
>>
>> I included the stats display and the time for each run below.  This is
>> dar v 2.4.8.
>>
>> Thanks for a great software!
>
> You are welcome! :)
>
>>
>> Kevin
>>
>
> Denis.
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQIVAwUBUHrC0QgxsL0D2LGCAQIQYhAAmAljPUSkn32eMJMAS2tZdLYBJxmD3SiH
> mZu/Xq5HX+SrXm0Mm92nz1gBeI+VOiWpIG25P3B6BylVa41vPIe7eZ0egfTQc/xH
> /ofrWV97olOQneGgTc2xeVLn/6zngIk7MqoWW4xohg+PAw47liSRGICcWr92XN9s
> csEMWqKlY220wAg8LQP3FTpXpEqPE/vwh/mfpw9Sv4m0Bv8xM+vEprIjEQpVcZrI
> qdTZ/Kd2xDcBXK6DJyC9zb1CA9sjY8ZMOPHouc4/U5tqhz1F+N9JAx3jqAEu37tv
> 5FBvU9vmwkrUFy6KUUCcybvegaSe0Hkh72WHfVfLuMuIROl9pBMwL+khCRfIoAFO
> vSeemLb7T3JKn0plGpaxttS/FSdjhNTrZsZ+1+YahNM4ndWglsHL0OyrBc2IdYw5
> PPdv31Zm58VsdsEG//aM274iDFjlP+RtFIyes3fdZYZn83z8255cFajRUerWrl5v
> omR70PWJS3tvRRhh96DXB5VewvgxMxEFrdBsysOCm69u8T58tFMzg6FEc/PE6Z/Z
> zi4s6ARpHXhG4Ne0/sQ2O6m0ciEnZ/lHZa9LqWu1UY6+0QeO8nk2vSgps8KzMeO7
> Wfzjz12WWJCC3pX5Wwpwwxh8SqIsGEBoh+M9KzGfzfjOBr08Ujt+x+FG19R7og+B
> E0YdaGxUthQ=
> =pZpF
> -----END PGP SIGNATURE-----
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> Dar-discussions mailing list
> Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/dar-discussions
>

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev