Re: Using Magick++ to convert large PDFs into individual PNGs

On Mon, 8 Mar 2021, Lukas Studer wrote:
>
> I did notice though that setting the "subimage" and "subrange"
> attributes in ImageInfo was significantly slower than using the file
> name specifier. Do you have any idea as to why this might be?

I have no idea other than if for some reason Ghostscript is being 
invoked more often or that something you are doing is causing more RAM 
to be consumed.

> On a slightly different topic: It appears that the writing of PDF files
> suffers the same limitations as the reading thereof. Writing large
> documents causes the RAM to overflow crashing the application. Do you
> have any insights on how to keep the RAM usage in check while writing
> the PDF files? Is there a way to write them in chunks? Further, I don't
> quite understand why this happens in general. How is it that a PDF file
> of about 100MB can overflow my RAM of about 64GB (when
> importing/exporting at 300 DPI). Is there something I'm doing horribly
> wrong? Why is it bloated this much?

The internal image model used by GraphicsMagick (inherited from 
ImageMagick) is not very efficient.

The documentation in www/INSTALL-unix.rst says that the memory usage 
for an image may be computed as:

   (QuantumDepth*Rows*Columns*5)/8

Notice that QuantumDepth is a constant for the build ("Q8", "Q16", 
"Q32") and that the memory required for one pixel has no relationship 
with how the pixels may be stored in a file.

Your PDF files might use just a single bit (where one byte can store 8 
bits) to represent a pixel (for black/white).  Furthermore, the PDF 
file may use extremely effective compression so that many pixels may 
be stored in very little space.

While the simple representation in memory simplifies things and 
reduces the amount of implementation code required, it is grossly 
inefficient as pertains to memory usage.  Even the Q8 build may 
consume 5 bytes of memory to store one bilevel pixel.

The format writer suffers from some of the same issues as the reader. 
All of the frames need to be built up in memory (or also as temporary 
disk files) before the writer writes anything.

There are two major issues to solve in GraphicsMagick (but also in 
ImageMagick in case you were wondering) in order to reduce the memory 
consumption for each pixel, but also to be able to "stream" image 
frames from the source to the destination.

Bob
-- 
Bob Friesenhahn
[email protected], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
Public Key,     http://www.simplesystems.org/users/bfriesen/public-key.txt