Re: Dar architecture questions

On 10/15/2012 02:21 PM, Denis Corbin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 14/10/2012 22:37, Kevin Wormington wrote:
>> Dennis,
>>
>> Thank you for the detailed reply.  I have done some more testing and you
>> are correct that memory deallocation consumes a considerable amount of
>> time after the stats are displayed.  I re-configured with
>> --disable-special-alloc and the results (below) are quite dramatic.
>> There was a ~600MB increase in memory usage but the execution time is
>> down to less than 10 minutes.
>
> Well I'm surprised by this result. The objective was rather to speedup
> the memory allocation than to reducing its usage! When it had been
> implemented, it did reduce execution time on the available systems...
>

I re-ran my full backup and it ran in 7.9 hours with special-alloc 
disabled so there was a gain of about 20% vs the 10 hours it took before.

I also ran a small backup (1.2GB, 14,000 files; ran each twice and used 
the lowest number always from the 2nd run where everything was likely in 
read cache) with the results:

		special-alloc		disable-special-alloc
full		1 min 53 secs		1 min 53 secs
incremental	3.329 secs		2.925 secs

So the smaller backup is about even between the two.

I'm going to try and fire it up in gdb and see if I can catch it in the 
deletes.  I speculate that the standard library version is make a single 
(or very few) calls to free() when it sees that the entire object is 
being deleted.

Kevin

>>   It appears that memory allocation must be
>> playing a significant part in the total backup time as well.  I am going
>> to re-run the full backup and see what kind of improvement is has as
>> well.  Hopefully I will also get a chance in the next few days to see if
>> I can determine what it is about the special_alloc that causes the
>> slowdown vs system libraries at least for my use case.
>
> I will double check on my side using a gprof. Thanks for your feedbacks!
>
>>
>> Kevin
>>
>
> Regards,
> Denis.
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQIVAwUBUHxiQAgxsL0D2LGCAQKJeg/6AyoSxurS3E+olrYjqjjOQBS1mywXz6Cs
> EC5Dv3vvO7xo4r8WYx/dT5IjMG7FUmRLTTYedtEiR3cp+TFokcQ4cjk9voN4EjVv
> 27rzXLvAn5d/9kv5a4QEGWYnp4RH3R/ybcWXdAlk2RVnAqJikgwtLi/otI1X1Ayg
> Sv100cbw+nBTwuK60ZqjAY1pXYXC4vQ03ZGRmY9XfvXZ/Eo6e7Shcy3dY+k6aPjB
> LP33BarjiACsbtaI88pSv+9shdOqvpqJOrzEzXFYxzbpz9McKgirRaYhCz12SDFM
> IBUV+8iyvdclLLfizW08SyKl7348xCYF2FkikzhlSBrE5qdMqDSvo+7FU5CuSRWV
> 445YrgdP2Fs7UU+ed1T5irRchlsaWgLelVTepqvRF6bqr8y1OWPg3ZxnNSAcGjPy
> UvnCLDr6OFuM94Wml9KRQCLWYyW8UlFB6Lpezo/CXL/hlUIkcJGiu8jZfHr6x+00
> 0VcRMAdEx+JrOSgxysWsgm2mb+XQQICVSk3n41HUhf7C/QzmQvuKM/fd1axmeacJ
> axDXt+r1mH0pGXjwob/08Nwr67kNmefHIbiqiUePjydIaWXeJYeP0AGbixkujxui
> JZWlz9hCRUuapa3pxpOf5DHGj5Y0SzB9u+4ko/KYzGElrPMwtpLWKTUcuUOD7KZE
> zZpnOPWYQoQ=
> =58FQ
> -----END PGP SIGNATURE-----
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> Dar-discussions mailing list
> Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/dar-discussions
>

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev