Re: Dar architecture questions

On 10/16/2012 02:22 AM, Denis Corbin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Le 16/10/2012 00:30, Kevin Wormington a =E9crit :
>> I strace'd the small backup (14k files) both with and without
>> --disable-special-alloc and it appears the difference is in the huge
>> number of rt_sigprocmask calls made by special-alloc.
>
> interesting... I have tests currently running on my side...
>
>>   Am I correct that
>> the amount of memory used by dar is mostly related to the number of
>> files backed up and not the sizes of the files backed up?
>
> yes this is correct.
>
>> Ie, I wonder
>> if a different workload of small number of large files would be effected
>> by the use of special-malloc.   If it could be confirmed that it's a win
>> to disable special-malloc I would suggest that you consider disabling it
>> by default.  I only have recent ubuntu and debian servers to test with
>> so I'm not sure what the result would be on the other flavors of linux
>> and/or different os platforms.
>
> Yes, that's a point to take into account. And you also reported that
> with special-alloc uses dar uses less virtual memory, right? Sometimes I
> get feedback about lack of memory (huge number of files under backup).
> So I guess its better to have by default a slower binary less fond of
> memory than a faster one using more memory.
>
Yes, special-alloc uses about 600MB less memory for ~1.7 million files =

than the system new/delete routines.  That could definitely be an issue =

for a user with memory constraints.  If your test results confirm mine =

then documenting it in the release/install notes and perhaps in the man =

page would probably be best.

I have also included a patch below that prints some status info before =

the memory frees/on-fly-catalog since these can take several minutes to =

complete after the archive stats print.

diff -c dar.cpp.orig dar.cpp
*** dar.cpp.orig	2012-10-16 09:51:37.157093322 -0500
--- dar.cpp	2012-10-16 09:55:24.467092154 -0500
***************
*** 270,275 ****
--- 270,277 ----

   		    // making some room in memory

+ 		if(!param.quiet)
+ 		    dialog.warning(gettext("Making room in memory..."));
   		if(arch !=3D NULL)
   		{
   		    delete arch;
***************
*** 295,301 ****
   			throw SRC_BUG;
   		    if(param.op =3D=3D create)
   		    {
! 			if(param.info_details)
   			    dialog.warning(gettext("Now performing on-fly isolation..."));
   			if(cur =3D=3D NULL)
   			    throw SRC_BUG;
--- 297,303 ----
   			throw SRC_BUG;
   		    if(param.op =3D=3D create)
   		    {
! 		        if(!param.quiet)
   			    dialog.warning(gettext("Now performing on-fly isolation..."));
   			if(cur =3D=3D NULL)
   			    throw SRC_BUG;
***************
*** 609,614 ****
--- 611,618 ----
   	}
   	catch(...)
   	{
+ 	    if(!param.quiet)
+ 	        dialog.warning(gettext("Final memory cleanup..."));
   	    if(arch !=3D NULL)
   	    {
   		delete arch;
***************
*** 626,631 ****
--- 630,637 ----
   	    }
   	    throw;
   	}
+ 	if(!param.quiet)
+ 	    dialog.warning(gettext("Final memory cleanup..."));
   	if(arch !=3D NULL)
   	{
   	    delete arch;


> However, I must document the result of your tests (and mine) about this
> subject. This might be of interest for the ones that are not concerned
> by memory limitation and want to speed up their backup.
>
>   Perhaps someone else on the list that
>> makes large backups either in size or number of files could confirm if
>> they see the same behavior.
>>

---------------------------------------------------------------------------=
---
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev