Re: Multiprocess

Cyril Russo wrote:
> In the github code, I can see the 2 threads, but I wonder how to make x
> threads of them.

http://github.com/vasi/pixz/blob/master/write.c#L109

You can see that there's one reader-thread, one writer-thread (the
main thread) and N encoder-threads. Currently N is the number of
processors online as returned by sysconf(_SC_NPROCESSORS_ONLN) . One
day I'll make it a setting, or for bonus points make it use only
"available processors".

> Sorry, I meant rsync-able, like the -rsyncable option in gzip.
> The idea is to empty the entropy encoder history, so that while
> decoding, you can start from that position without requiring the
> previous data / encoder state.
[snip]
> This doesn't solve the rsync issue, I meant: let's say if you add 1
> bytes in the compressed block i, then it'll still modify compressed
> block i+1.

Right, currently pixz is fine for rsync only if the size stays the
same. I really question under what circumstances someone would have
large xz archives which change just a little bit, I guess if you have
a cron that creates tarballs for some reason?

Anyhow, if it is important to you, it should be quite possible to add
a pre-processing step to pixz so that it starts a new block based on a
pattern/checksum rather than on size alone. You'll want a much bigger
block size than the 6k that gzip --rsyncable uses, so that xz can use
its full power. There are some potential downsides to this: slightly
lower compression; more code to handle degenerate files that never
match the pattern; splitting logical blocks (eg: files in a tarball)
between multiple pixz blocks. But they're not very serious problems.


A further thought about rsync in dar. One great feature in dar is that
you can detach the catalogue from an archive, and future incremental
archives require only the catalogue to be present. The same sort of
thing could be done with dar+rdiff: Allow the catalogue+signatures to
be split off from the full archive, and then future incremental,
rdiff-delta archives will not require the full archive to be present.

-V

------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
Spend less time writing and  rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb