Re: Multiprocess

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Dave,

Thanks for the info. I have noted the URL in the lzma feature request
for further reference.

Regards,
Denis.

Dave Vasilevsky a écrit :
> Just to be extra-complete, I made a parallel implementation of xz (aka
> lzma, aka 7zip): http://github.com/vasi/pixz , It currently only runs
> on tar files, but that's trivially changed.
> 
> -Dave
> 
> 2010/9/27 Saso Slavicic <[email protected]>:
>> Hi,
>>
>> Just for the reference, there is also a parallel implementation of bzip2
>> compressor. Not sure if the code is useable from dar...
>>
>> http://compression.ca/pbzip2/
>>
>> If I remember correctly, the algorithm compresses the file into several
>> independent bzip2 archives which are concatenated together and seen as a
>> single archive by the decompressor. Again, not sure how useable this is for
>> dar, but it would be nice to have bzip2 threaded :)
>>
>> Best regards,
>>
>> -----Original Message-----
>> From: Denis Corbin [mailto:[email protected]]
>> Sent: Monday, September 27, 2010 1:48 PM
>> To: For general discussions about Disk ARchive
>> Subject: Re: [Dar-discussions] Multiprocess
>>
> Hello Anders,
> 
> Anders Bo Rasmussen a écrit :
>>>> Hi Denis
>>>>
>>>> I was thinking on something like the N+M threads in the faq. But with a
>>>> restriction on the buffer size - e.g. on 32MB for each thread. And I
>>>> would only have one process reading files and only one writing the output
> file.
>>>> This would give a large speedup as long as the files are not larger than
> 32MB
>>>> compressed. If the files are only a little larger it will speedup some. If
> the
>>>> files are e.g. 2GB the speedup will be insignificant.
>>>>
>>>> Also a buffer in the writing process on e.g. 128MB would help to even
>>>> out the gaps that happens in reading/compression.
>>>>
>>>> Most files that are larger than 32MB is in my case files that are not
>>>> compressed anyway because of their extention (.gz,.jpg,.cr2, etc) so I
>>>> think all this would give a significant speedup.
>>>>
>>>> For files larger than 32MB something like
>>>> http://www.zlib.net/pigz/
>>>> could be used?
> You remind me a feature request about pigz, and fetching the reference
> to it:
> https://sourceforge.net/tracker/index.php?func=detail&aid=2908554&group_id=6
> 5612&atid=511615
> I noticed that there had been very interesting feedback since the last
> time I discussed about it there. So if some improvement has to be done
> for parallel computing it will be done thanks pigz I hope.
> 
>>>> /Anders
> Thanks for reminding me this point!
> 
> Regards,
> Denis.
> 
>>
-
----------------------------------------------------------------------------
- --
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Dar-discussions mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-discussions
>>
>>
-
------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Dar-discussions mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-discussions
>>

> ------------------------------------------------------------------------------
> Beautiful is writing same markup. Internet Explorer 9 supports
> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
> Spend less time writing and  rewriting code and more time creating great
> experiences on the web. Be a part of the beta today.
> http://p.sf.net/sfu/beautyoftheweb
> _______________________________________________
> Dar-discussions mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/dar-discussions
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFMq4uCpC5CI8gYGlIRAtIBAKCFKIIkTWUjmczdxw+X9ih210woswCgus7Q
F0o2h8ACEglnANcs71KIURQ=
=BoYK
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
Spend less time writing and  rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb
_______________________________________________
Dar-discussions mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-discussions