Re: Multiprocess

Hi Denis

I was thinking on something like the N+M threads in the faq. But with a
restriction on the buffer size - e.g. on 32MB for each thread. And I
would only have one process reading files and only one writing the output file.
This would give a large speedup as long as the files are not larger than 32MB
compressed. If the files are only a little larger it will speedup some. If the
files are e.g. 2GB the speedup will be insignificant. 

Also a buffer in the writing process on e.g. 128MB would help to even
out the gaps that happens in reading/compression.

Most files that are larger than 32MB is in my case files that are not
compressed anyway because of their extention (.gz,.jpg,.cr2, etc) so I
think all this would give a significant speedup. 

For files larger than 32MB something like 
http://www.zlib.net/pigz/
could be used?

/Anders
On Fri, Sep 24, 2010 at 07:15:19PM +0200, Denis Corbin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi Anders,
> 
> please read the FAQ about that subject,
> 
> 	http://dar.linux.free.fr/doc/FAQ.html#parallel
> 
> if you have any idea or objection, feel free to present them here.
> 
> Kind Regards,
> Denis.
> 
> Anders Bo Rasmussen a écrit :
> > Hi.
> > 
> > I'm using dar to backup up my private files and I'm happy about it. 
> > 
> > I however see that when I take backup up of my home directory to a usb
> > 2.0 disk, dar uses on average 50% cpu, but sometimes peaks at 100% cpu
> > of one core. When it peaks at 100% cpu the backup to the usb is slower
> > than optimal - this can easily be seen by delays in the blinking LED. I
> > use both encryption and compression, so this probably happens when there
> > is data that can be compressed a lot.
> > 
> > If I bought a USB 3.0 drive it would probably wait for the CPU all the
> > time. 
> > 
> > Of course I could set the compression level lower (it is currently 6),
> > which would make less cpu usage and at the same time write more data to
> > the drive.
> > 
> > But wouldn't it be good to have dar multiprocess? I see some possible
> > benefits:
> > 
> > * Multiple processes that could compress and encrypt (I don't know if
> > the encryption is possible or it has to be encrypted as one big stream)
> > would increase the available cpu power.
> > * Possible buffered output, so there would be e.g. 500MB of unwritten
> > data, that could be used for when the "hard cpu times" comes.
> > * Possible buffered input, so small files could be read ahead, and
> > therefore not make a gab in feeding the compression/encryption.
> > 
> > /Anders
> > 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iD8DBQFMnNympC5CI8gYGlIRArrEAKCcN502Si0AoNgVs90AhD4GhNT0/QCeK8nQ
> O7PIpQpBKA1K6bPAJKwUrow=
> =iJFh
> -----END PGP SIGNATURE-----
> 
> ------------------------------------------------------------------------------
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> Dar-discussions mailing list
> Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/dar-discussions

------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev