Re: zip decompression and multithreading

Hi Uwe,

Are you perhaps reading the image a scanline at a time, for example by 
calling readPixels(y) ? If so, try pulling in the entire image in one go 
- i.e. use readPixels(minY,maxY) and see if that changes the threading 
behaviour. EXR only multithreads efficiently when you request enough 
scanlines in a single call for it to spread the work across all threads.

If you simultaneously read multiple EXR files but only a single scanline 
from each, you may still only get one thread decompressing. Some updates 
are on their way to make that possible by setting the globalThreadCount 
to 1 in that case, but I suspect you'd still get better performance by 
reading entire images one at a time.

Peter



On 05/06/18 04:22, Uwe wrote:
> *Originates outside of Weta Digital
>
>
>
> Hello everyone,
>
> I am using openexr-2.2.1 on Linux and OSX.
>
> I'm developing an application which loads a sequence of zip-compressed
> EXR-files
> and converts them into some kind of archive. I would like to load and
> dispatch
> images multithreaded. I'm testing my application on a 12-core machine
> running linux,
> so, watching performance by means of 'top' (the shell command) should
> ideally give a performance of nearly 1200%, but I get only 150%. If I
> analyze this
> with gdb, I see that in fact I have 12 threads (the ones I created myself
> outside openexr), but 11 of them are waiting at some mutex within
> openexr, while the 12th
> is busy decompressing somewhere inside zlib. What I would like to have
> is 12 threads
> decompressing...
>
> One solution seems to be to compile openexr with option
> --enable-threading=false,
> so that openexr uses a single thread for everything, but my applications
> runs 12 threads reading exr-files.
> If I do this, I nearly get the full performance, something around 1050%
> for 12 cores,
> but this might slow down other decompression modes.
>
> My question is:
> 1. Is there some reason for openexr to use only one thread for
> decompressing
> with zip? As far as I know zlib is thread-safe, so I'm puzzled...
> 2. Or is there a way to enforce multithreading during zip-decompression?
>
> Best regards
> Uwe Sassenberg
>
>
> _______________________________________________
> Openexr-devel mailing list
> [email protected]
> https://lists.nongnu.org/mailman/listinfo/openexr-devel