Re: Using Magick++ to convert large PDFs into individual PNGs

On Tue, 9 Feb 2021, Lukas Studer wrote:
>
> First of all, I'm new to GraphicsMagick and the management thereof, so I
> am not quite sure if this is the correct mailing list.
>
> I'm am using the Magick++ API to read large PDF files (ballpark of about
> 2000 to 3000 pages, size of 100MB and upwards) and store each page in a
> database (not shown in snippet). While doing so I run into resource
> issues, namely not enough RAM, when reading the PDF file resulting in
> basically, or what feels like, an infinite runtime.

You are facing a long-term design issue regarding the interfacing of 
GraphicsMagick and Ghostscript as well as how GraphicsMagick works in 
general.  GraphicsMagick asks Ghostscript to convert all (or a range) 
of pages to an intermediate file format (e.g. a PNM sub-format) and 
then reads all of the pages before it writes any output.  This 
approach does not scale since the system will run out of resources.

> My question now boils down to, what is the proper API for such a use
> case, i.e. how does one handle such big PDF files using the Magick++
> API? Is there a way to do this in batches or in a streamed fashion,
> where I read page by page and directly store them?

It is possible to succeed by iteratively setting the 'subimage' and 
'subrange' properties so that only a few pages are included in the 
conversion at a time.  It turns out that Ghostscript is pretty 
efficient at finding the first specified page in a PDF sequence.

> I know of the syntax involving the indexOf operator to only read a range
> of pages, e.g. "myFile.pdf[0-100]". I was hoping to solve the issue
> directly via the Magick++ API.

Given the advice to set the 'subimage' and 'subrange' properties you 
would be able to succeed, but then the next problem is hit in that I 
see this FIXME: statement at the top of the readImages() 
implementation in Magick++/STL.h:

"FIXME: need a way to specify options like size, depth, and density."

The FIXME: statement should also have mentioned 'subimage' and 
'subrange'.

There is yet hope given that the readImages() template function is in 
a header file and is very short so a modified implementation can be 
added.  The next trick is that the Magick::Options implementation is 
described as "This is an internal implementation class and is not part 
of the Magick++ API" so it can't be used.  There are also 
subImageImage() and subRangeImage() STL function objects, but they are 
not intended for this purpose.

Ultimately, what is needed is to update 'subimage' and 'subrange' in 
this ImageInfo instance similar to how filename is updated.

MagickLib::ImageInfo *imageInfo = MagickLib::CloneImageInfo(0);

So, what can be done is to duplicate the existing readImages() 
template function and create a new one with two more integer arguments 
for 'subimage' and 'subrange'.  Then assign those arguments into the 
ImageInfo struct.  This would allow you to request a reasonable number 
of pages at a time.  The last tricky thing is to detect when the end 
of the document has been reached.  If just one page at a time is 
requested, then this is not so hard, but it is less efficient than 
several pages at a time.  Perhaps when an error occurs, then switch to 
one page at a time.  A more sophisticated readImages() could add a 
loop so that the implementation does the per-page iteration itself.

Lastly, there is a completely different solution, which is to ignore 
the existing STL implementation.  Instead, use the Image class and its 
read method which is only willing to read one image at a time from the 
sequence.  The Image class provides subImage() and subRange() so you 
should be able to read one page at a time.

I apologize for such a lame implementation.

Bob
-- 
Bob Friesenhahn
[email protected], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
Public Key,     http://www.simplesystems.org/users/bfriesen/public-key.txt