Re: Multiprocess

Just to be extra-complete, I made a parallel implementation of xz (aka
lzma, aka 7zip): http://github.com/vasi/pixz , It currently only runs
on tar files, but that's trivially changed.

-Dave

2010/9/27 Saso Slavicic <[email protected]>:
> Hi,
>
> Just for the reference, there is also a parallel implementation of bzip2
> compressor. Not sure if the code is useable from dar...
>
> http://compression.ca/pbzip2/
>
> If I remember correctly, the algorithm compresses the file into several
> independent bzip2 archives which are concatenated together and seen as a
> single archive by the decompressor. Again, not sure how useable this is for
> dar, but it would be nice to have bzip2 threaded :)
>
> Best regards,
>
> -----Original Message-----
> From: Denis Corbin [mailto:[email protected]]
> Sent: Monday, September 27, 2010 1:48 PM
> To: For general discussions about Disk ARchive
> Subject: Re: [Dar-discussions] Multiprocess
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello Anders,
>
> Anders Bo Rasmussen a écrit :
>> Hi Denis
>>
>> I was thinking on something like the N+M threads in the faq. But with a
>> restriction on the buffer size - e.g. on 32MB for each thread. And I
>> would only have one process reading files and only one writing the output
> file.
>> This would give a large speedup as long as the files are not larger than
> 32MB
>> compressed. If the files are only a little larger it will speedup some. If
> the
>> files are e.g. 2GB the speedup will be insignificant.
>>
>> Also a buffer in the writing process on e.g. 128MB would help to even
>> out the gaps that happens in reading/compression.
>>
>> Most files that are larger than 32MB is in my case files that are not
>> compressed anyway because of their extention (.gz,.jpg,.cr2, etc) so I
>> think all this would give a significant speedup.
>>
>> For files larger than 32MB something like
>> http://www.zlib.net/pigz/
>> could be used?
>
> You remind me a feature request about pigz, and fetching the reference
> to it:
> https://sourceforge.net/tracker/index.php?func=detail&aid=2908554&group_id=6
> 5612&atid=511615
> I noticed that there had been very interesting feedback since the last
> time I discussed about it there. So if some improvement has to be done
> for parallel computing it will be done thanks pigz I hope.
>
>>
>> /Anders
>
> Thanks for reminding me this point!
>
> Regards,
> Denis.
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFMoIRkpC5CI8gYGlIRAtNpAKChjHYotWZTSv6dfn+uzk7O7x89xwCgxRzq
> eZFJGx9mRZp1KFOxfIYeajg=
> =xZBg
> -----END PGP SIGNATURE-----
>
> ----------------------------------------------------------------------------
> --
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> Dar-discussions mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dar-discussions
>
>
> ------------------------------------------------------------------------------
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> Dar-discussions mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dar-discussions
>

------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
Spend less time writing and  rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb
_______________________________________________
Dar-discussions mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-discussions