Re: 2.7 pre-release: Virtual memory exploding and dar locks up

On 01/04/2021 21:39, Carsten Aulbert wrote:
> Hi again
> 
> On 01.04.21 20:40, Denis Corbin wrote:
>> Let's hope then the 1.3.1 release of libthreadar will fix the issue!
> 
> and you are spot on - compiling against libthreadar v1.3.1 fixes this
> issue.
> 
> Darn, why was the search result for
> 
> https://github.com/Edrusb/libthreadar
> 
> (which had v1.3.0 as the latest release) higher then the sourceforge
> project...

this is my bad, I just realized I had not pushed the tag 1.3.1 to git
(both github and sourceforge), while the source code was up to date in
those repositories and packaging at sourceforge is done out of git. This
avoided you identifying this release on github... Sorry for that, the
problem is now fixed.

> 
> Sorry, for wasting everyone's time - I will continue testing/evaluating
> pre4 for now. Anything in particular I should test for? Large data sets,
> many small files, huge files, massively parallel?

Well any positive or negative feedback is interesting, use pre4 as best
fits your needs, this would already bring interesting results as you
seem having uncommon (big) data set.

The tests done to validate this feature were done at the scale of 8 to
12 threads at most. Up to 6 to 8 threads the performance was very good
(execution time was near divided by the number of thread). As far as I
remember the multi-thread management overhead became visible over 10
threads, though the overall execution time was still much faster than
the single thread execution.

these tests were done based on a tmpfs filesystem to avoid having
hard-disk I/O becoming the bottleneck.

Last there is also the nature of the files that can impact performances:
as by design dar compress per file (to be able to directly access each
of them and to avoid compressing already compressed ones [-Z and -Y
options]), many small files will not give the same performance as a few
large files... You can play with the compression block size (within -z
option) but also the --mincompr option independently.

So if you have the opportunity to test at various scales, that would be
interesting, I'm just afraid you will rather test your storage
performance than the compression/decompression parallelization
performance in libdar!  :^)

> 
> Cheers
> Carsten
> 

Cheers,
Denis

_______________________________________________
Dar-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dar-support