Re: Multiprocess

Hi,

Just for the reference, there is also a parallel implementation of bzip2
compressor. Not sure if the code is useable from dar...

http://compression.ca/pbzip2/

If I remember correctly, the algorithm compresses the file into several
independent bzip2 archives which are concatenated together and seen as a
single archive by the decompressor. Again, not sure how useable this is for
dar, but it would be nice to have bzip2 threaded :)

Best regards,

-----Original Message-----
From: Denis Corbin [mailto:[email protected]] 
Sent: Monday, September 27, 2010 1:48 PM
To: For general discussions about Disk ARchive
Subject: Re: [Dar-discussions] Multiprocess

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Anders,

Anders Bo Rasmussen a écrit :
> Hi Denis
> 
> I was thinking on something like the N+M threads in the faq. But with a
> restriction on the buffer size - e.g. on 32MB for each thread. And I
> would only have one process reading files and only one writing the output
file.
> This would give a large speedup as long as the files are not larger than
32MB
> compressed. If the files are only a little larger it will speedup some. If
the
> files are e.g. 2GB the speedup will be insignificant. 
> 
> Also a buffer in the writing process on e.g. 128MB would help to even
> out the gaps that happens in reading/compression.
> 
> Most files that are larger than 32MB is in my case files that are not
> compressed anyway because of their extention (.gz,.jpg,.cr2, etc) so I
> think all this would give a significant speedup. 
> 
> For files larger than 32MB something like 
> http://www.zlib.net/pigz/
> could be used?

You remind me a feature request about pigz, and fetching the reference
to it:
https://sourceforge.net/tracker/index.php?func=detail&aid=2908554&group_id=6
5612&atid=511615
I noticed that there had been very interesting feedback since the last
time I discussed about it there. So if some improvement has to be done
for parallel computing it will be done thanks pigz I hope.

> 
> /Anders

Thanks for reminding me this point!

Regards,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFMoIRkpC5CI8gYGlIRAtNpAKChjHYotWZTSv6dfn+uzk7O7x89xwCgxRzq
eZFJGx9mRZp1KFOxfIYeajg=
=xZBg
-----END PGP SIGNATURE-----

----------------------------------------------------------------------------
--
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Dar-discussions mailing list
Dar-discussions-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/dar-discussions


------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev