Re: Multithreading with GM

On Tue, 11 Oct 2011, Chris Healer wrote:
>
> The problem I'm having is that no combination of Mutexes seems to
> protect the image cache from flushing unpredictably.  As an example, there
> may be 8 or more threads trying to call something like:
>
> Pixels view(img);
> const PixelPacket *pixels = view.getConst(0,row,img.columns(),1);

The Pixels view is just a handle to a heap-allocated structure.  It 
does not maintain a reference count to ensure that the underlying 
image is not destroyed if there are still allocated views.  If the 
underlying pixel cache is destroyed, then it is important not to use 
any still-allocated pixel view handles since bad things would surely 
happen.  It is wise to scope/order things so that allocated views are 
released before the referenced image.  In other words, deallocation 
should be in reverse order of allocation.

Besides the Image reference counting that the Magick++ Image class 
does, the underlying Image structure uses reference counting to the 
pixel cache representation which actually stores the pixels.  Both of 
these forms of reference counting are designed to be thread safe.

> concurrently.  It works fine for a while but eventually it crashes.
> Running with MAGIC_DEBUG=all reveals:
>
> destroy cache [path]

This message comes right before:

   cache_info->signature=0;
   MagickFreeMemory(cache_info);

> Destroy blob, image=0x10282b200, filename="[path]"

The blob is a structure referenced by the Image.  This one is also 
reference counted.  I believe that a bug was fixed which sometimes 
broke the blob reference counting since the last release.

> are reliably the culprits, and:
>
> Assertion failed: (image->signature == MagickSignature), function
> OpenCacheView, file magick/pixel_cache.c, line 3193.

This particular assertion would happen if the underlying C API Image 
structure was already deallocated (or corrupted somehow) when the 
cache view was opened.

> This makes sense... anyone know of a way to control the flushing, or create
> a mutex of some kind so I can grant permission for the cache to do its thing?
> I can't seem to find anything in the documentation like this...

The cache is not destroyed until all of the Image references are gone. 
There is nothing similar to garbage collection used.

Multithreaded programming takes more discipline than normal 
programming.  While you reported an error with OpenCacheView(), I do 
think that you need to check to make sure that all Pixel view 
references to an image are destroyed before the referenced Image.

It is certainly possible that Magick++ or GraphicsMagick has a bug. 
Make sure that your build only disabled OpenMP and not the thread 
locking support.

What GraphicsMagick version are you using?

If your system is able to run valgrind (does support AMD64/Darwin) 
then its 'helgrind' tool (and 'drd' tool) can be very useful to 
discover any locking bugs.  As delivered, the development version of 
GraphicsMagick completely passes the valgrind 'memcheck' and 
'helgrind' tests when driven by OpenMP.

See "http://www.valgrind.org/" for valgrind.

Bob
-- 
Bob Friesenhahn
[email protected], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct