RE: Fix for 687696 PDF 1.4 transparency can use largechunks of memory.

Igor,

>I viewed your patch briefly in 15 minutes,
>and don't see any problem in code level.
>Let me know if you expect me to spend more time for reviewing it.

No.  Since this is not a piece of code with which you have worked,
I do not expect you to spend any more time on this review.


>One thing I'm worried about.
>I'd like to understand, what happens when the
>target device is another high level device, such as pdfwrite.
>In this case there is no clist device.
>Where do you insert two pdf14 devices in this case ?

Only one PDF compositing device is inserted with high level devices.
This is the same as the previous version of Ghostscript.


>Actually pdfwrite needs 2 modes :
>1. "Planarize" the transparency (when converting into PDF <=1.3,
>particularly for ps2write).
>2. Pass the transparency through (when converting into PDF >1.3)
>(not yet inplemented).

I agree that the handling of PDF 1.4 transparency in the pdfwrite
device needs to be enhanced.  (There is already an open bug report
for this issue.)  The work which has been done for 687696 is a first
step in that process.  Since this change uses the create_compositor
device proc to implement transparency, the pdfwrite device can now
control what is being done with transparency.  By simply using the
default PDF 1.4 create compositor logic, the current action of
flattening the transparency data is accomplished.  However the pdfwrite
device can also do its own implementation of the PDF 1.4 create
compositor.  This would give the pdfwrite device access to the
various PDF 1.4 transparency actions.  The pdfwrite device can then
place the required info into the output file.


>Did you test the your patch with pdfwrite ?

Yes.  A full regression test has been done on the code.  This
includes testing for pdfwrite.


Dan


-----Original Message-----
From: Igor V. Melichev [mailto:[email protected]]
Sent: Thursday, March 10, 2005 3:16 AM
To: [email protected]; Gs-Code-Review
Subject: Re: [gs-code-review] Fix for 687696 PDF 1.4 transparency can
use largechunks of memory. 


Dan,

I viewed your patch briefly in 15 minutes,
and don't see any problem in code level.
Let me know if you expect me to spend more time for reviewing it.

One thing I'm worried about.
I'd like to understand, what happens when the
target device is another high level device, such as pdfwrite.
In this case there is no clist device.
Where do you insert two pdf14 devices in this case ?
Thank you.

Actually pdfwrite needs 2 modes :
1. "Planarize" the transparency (when converting into PDF <=1.3,
particularly for ps2write).
2. Pass the transparency through (when converting into PDF >1.3)
(not yet inplemented).

Did you test the your patch with pdfwrite ?

Igor.

----- Original Message ----- 
From: "Dan Coby" <[email protected]>
To: "Gs-Code-Review" <[email protected]>
Sent: Thursday, March 10, 2005 12:01 PM
Subject: [gs-code-review] Fix for 687696 PDF 1.4 transparency can use 
largechunks of memory.


> Fix for 687696 PDF 1.4 transparency can use large chunks of memory.
> Prior to this fix the handling of PDF 1.4 transparency features used one
> or more full size image buffers.  With high resolutions and large images,
> these buffers can be multiple gigabytes.  As a result, we were not able
> to handle these files.
>
> DETAILS:
>
> PDF 1.4 transparency is handled by a device which is inserted into the
> device chain.  Prior to this fix, this device was via the 'device filter'
> logic.  With this fix, the PDF 1.4 transparency device is inserted via
> the compositor device mechanism.  This change was made since the device
> filter logic did not allow other devices to control of where the PDF 1.4
> transparency device is placed in the device chain.  There is a device
> proc which implements the create compositor action.  This allows more
> control than the device filter logic.
>
> The first step in this fix was to convert the PDF 1.4 transparency
> logic to use the create compositor mechanism.  For situations in which
> the clist logic is not being used, the PDF 1.4 transparency device is
> created and placed in the same location in the device chain as it was put
> by the device filter logic.  In this situation, there is very little
> logical difference in the operation of the PDF 1.4 transparency.
>
> For situations in which the clist is used, there are two PDF 1.4
> compositing devices created.  There is one device before the clist when 
> the
> clist is being written.  There is a second device after the clist when
> the clist data is being read.
>
> The device before the clist serves two purposes, it provides a means for
> implementing the process color model for the PDF 1.4 compositing.  (The
> PDF 1.4 compositing may used a different process color model from the 
> output
> device.)  Thus this device has color_info and encode/decode color procs
> which match the PDF 1.4 blending color space.  This device also ensures
> that the PDF 1.4 blending parameters (blend mode, shape, opacity, etc.)
> are passed through the clist to the second PDF 1.4 device.
>
> The second PDF 1.4 device (the one when the clist is being read) is the
> same PDF 1.4 compositing device which is used when the clist is not being
> used.  This is the device which does the blending operations.  Since the
> device is after the clist, the buffers that it needs are the size of a
> single band instead of the full page.  This device is placed prior to
> the memory device which is used for drawing the raster data.
>
> Two changes were made in the 'create compositor' logic.  The imager state
> was being passed as a 'const'.  It is no longer const since the PDF 1.4
> transparency compositing changes the cmap procs in the imager state to not
> use transfer functions during color calculations.  The transfer functions
> are applied later (and the cmap procs restored) when blended image data
> is output from the PDF 1.4 compositing buffers.
>
> The second change to the create compositor logic is the addition of two
> more procedures to the gs_composite_type_t.procs list.  These procedures
> are used by the clist device create compositor routines.  The first
> is used to create the PDF 1.4 clist write device.  The second saves and
> restores the color_info field for the clist device (see below).
>
> Default versions of the added gs_composite_t.procs were also added for
> use by the other compositor devices.  The defaults are no-ops.
>
> The clist logic has a few changes.  Since the PDF 1.4 compositor device
> may use a different process color model from the output device, the number
> of colorants and the depth (bits per pixel) can be different for the PDF
> 1.4 compositor and the output device.  Thus changes were needed in parts
> of the clist logic which assumed that these items were constant during the
> processing of the clist.  An extra field was added to the device halftone
> to indicate the number of colorants in effect when the device halftone was
> created.  This data is included when the halftone is serialized for the
> clist.  (Fortunately the PDF 1.4 compositing does not use halftones or
> transfer functions.  So the device halftones and transfer functions do
> not need to be changed when the PDF 1.4 compositor is installed.)
>
> As previously mentioned the create compositor procedures for the clist
> writer and reader now call a couple of compositor specific routines. 
> These
> allow the compositor to implement actions related to the clist device.
> (The PDF 1.4 compositor uses these routines to create the 'write clist'
> PDF 1.4 compositor and to save and restore the clist device color info.
> The color_info field of the clist device is changed to match the process
> color model of the PDF 1.4 compositor while the PDF 1.4 compositor is 
> active.
>
> A additional device parameter (PageUsesTransparency) was added.  This
> parameter is sent by the PDF interpreter to indicate if PDF 1.4 
> transparency
> will be used on the page.  The banding/no banding decision and the size
> of a band includes an estimate of the size of the PDF 1.4 blending buffers
> if this parameter is true.  Note:  The estimate of the size of the
> blending buffers is not exact since this calculation is made before the
> actual number of buffers, the number of blending colors and alpha channels
> is known.  Fixed values are used for the estimated buffer size.  This is
> still much better than the pervious situation, in which the only the
> output raster size was being used.  The blending buffers can easily be
> 30 or more times larger than the raster buffer.
>
> This fix also involves the removal of the PDF 1.4 'marking' devices.
> The marking devices were used to actually implement the pixel marking.
> The marking devices were created at the start of each high level drawing
> operation and deleted after the completion of the operation.  The marking
> device action has been merged into the primary PDF 1.4 compositing device.
> This change eliminates some device creation overhead.  The overhead would
> have been worse since most information about high level drawing operations
> is not passed through the clist.  As a result it would have been necessary
> to create marking devices for each low level operations.
>
>
>
> The following issues are not included in this fix but need to be 
> considered.
>
> 1)  The device filter logic is no longer used.  It may be desired to 
> remove
> this logic.
>
> 2)  We currently have a 'create_compositor' device proc.  However there
> is not a 'delete_compositor' device proc.  As a result, compositor devices
> can stay in the device chain even when they are no longer needed.  The
> current fix turns the PDF 1.4 compositor into a 'forwarding' device.  This
> is the same method used by the overprint compositor device.  I do not like
> leaving these devices since there is some extra overhead and there may be
> possible side effects.  (Testing has been done to try to find side effects
> and none has been found with the current code.)
>
> 3)  This change only affects devices which can use the clist logic.  This
> excludes the 'high' level devices.  These include the 'display', 'x11',
> and 'pdfwrite' devices.  Thus these device will continue to use a full
> image buffer.  It is possible to extend this fix to use a clist device
> for the PDF 1.4 device for these devices.
>
>
> Dan


--------------------------------------------------------------------------------


> _______________________________________________
> gs-code-review mailing list
> [email protected]
> http://www.ghostscript.com/mailman/listinfo/gs-code-review
>