Re: Multithreading with GM

On Fri, 14 Oct 2011, Chris Healer wrote:
> -The sheer quantity of images seems strain the cache, causing the cache to search for de-allocatable things...

The only thing I can think of in the cache which might work like this 
is if the pixel cache is using disk files.  With 256GB of memory you 
probably should not usually be using disk files.  The reason why there 
is an issue with disk files is that there is a limit on the number of 
file handles which may be allocated to a process at one time. Once the 
file handle limit has been hit, then cache files need to be opened and 
closed as they are used (done in a rather clumsy way).

% gm convert -list resource
Resource Limits (Q16, 64 bits/pixel, 64bit address)
----------------------------------------------------
     Disk:  Unlimited (MAGICK_LIMIT_DISK)
    Files:        256 (MAGICK_LIMIT_FILES)  <------- this one
      Map:      7.7GB (MAGICK_LIMIT_MAP)
   Memory:      3.9GB (MAGICK_LIMIT_MEMORY)
   Pixels:  Unlimited (MAGICK_LIMIT_PIXELS)
  Threads:          4 (OMP_NUM_THREADS)

The process file handle limit is increased by GM to 256 if needed. 
Depending on the OS it might be possible to increase to a much higher 
value (e.g. 1024).  The limit may be adjusted via the 
MAGICK_LIMIT_FILES environment variable.

Regardless, if cache files are being used, it is important to make 
sure that it is because you have already used up your RAM rather than 
due to an improperly low 'memory' resource limit.

Make sure that the above command indicates a memory limit which is 
appropriate for the memory on your system.

Lastly, if you export the environment variable MAGICK_DEBUG=Cache then 
your program should start logging how it is using the pixel cache. 
There is also a Resource debug option which you can add 
(comma-delimited if more than one) to see how resource consumption is 
being tallied. If there is a resource consumption "leak" (due to a 
bug) then GM might stop using a fast resource (e.g. RAM) because it 
thinks it is expended.

It does seem like you are really exercising the software.  Please send 
fixes for any bugs you find. :-)

Bob
-- 
Bob Friesenhahn
[email protected], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct