Re: Fix for 687696 PDF 1.4 transparency can use largechunks of memory.

Dan,

Thank you for explanation.

> Only one PDF compositing device is inserted with high level devices.
> This is the same as the previous version of Ghostscript.

Please add it to the log message of the patch.

Igor.


----- Original Message ----- 
From: "Dan Coby" <[email protected]>
To: "Igor V. Melichev" <[email protected]>; "Gs-Code-Review" 
<[email protected]>
Sent: Thursday, March 10, 2005 9:24 PM
Subject: RE: [gs-code-review] Fix for 687696 PDF 1.4 transparency can use 
largechunks of memory. (xefitra)


>
> Igor,
>
>>I viewed your patch briefly in 15 minutes,
>>and don't see any problem in code level.
>>Let me know if you expect me to spend more time for reviewing it.
>
> No.  Since this is not a piece of code with which you have worked,
> I do not expect you to spend any more time on this review.
>
>
>>One thing I'm worried about.
>>I'd like to understand, what happens when the
>>target device is another high level device, such as pdfwrite.
>>In this case there is no clist device.
>>Where do you insert two pdf14 devices in this case ?
>
> Only one PDF compositing device is inserted with high level devices.
> This is the same as the previous version of Ghostscript.
>
>
>>Actually pdfwrite needs 2 modes :
>>1. "Planarize" the transparency (when converting into PDF <=1.3,
>>particularly for ps2write).
>>2. Pass the transparency through (when converting into PDF >1.3)
>>(not yet inplemented).
>
> I agree that the handling of PDF 1.4 transparency in the pdfwrite
> device needs to be enhanced.  (There is already an open bug report
> for this issue.)  The work which has been done for 687696 is a first
> step in that process.  Since this change uses the create_compositor
> device proc to implement transparency, the pdfwrite device can now
> control what is being done with transparency.  By simply using the
> default PDF 1.4 create compositor logic, the current action of
> flattening the transparency data is accomplished.  However the pdfwrite
> device can also do its own implementation of the PDF 1.4 create
> compositor.  This would give the pdfwrite device access to the
> various PDF 1.4 transparency actions.  The pdfwrite device can then
> place the required info into the output file.
>
>
>>Did you test the your patch with pdfwrite ?
>
> Yes.  A full regression test has been done on the code.  This
> includes testing for pdfwrite.
>
>
> Dan
>
>
> -----Original Message-----
> From: Igor V. Melichev [mailto:[email protected]]
> Sent: Thursday, March 10, 2005 3:16 AM
> To: [email protected]; Gs-Code-Review
> Subject: Re: [gs-code-review] Fix for 687696 PDF 1.4 transparency can
> use largechunks of memory.
>
>
> Dan,
>
> I viewed your patch briefly in 15 minutes,
> and don't see any problem in code level.
> Let me know if you expect me to spend more time for reviewing it.
>
> One thing I'm worried about.
> I'd like to understand, what happens when the
> target device is another high level device, such as pdfwrite.
> In this case there is no clist device.
> Where do you insert two pdf14 devices in this case ?
> Thank you.
>
> Actually pdfwrite needs 2 modes :
> 1. "Planarize" the transparency (when converting into PDF <=1.3,
> particularly for ps2write).
> 2. Pass the transparency through (when converting into PDF >1.3)
> (not yet inplemented).
>
> Did you test the your patch with pdfwrite ?
>
> Igor.
>
> ----- Original Message ----- 
> From: "Dan Coby" <[email protected]>
> To: "Gs-Code-Review" <[email protected]>
> Sent: Thursday, March 10, 2005 12:01 PM
> Subject: [gs-code-review] Fix for 687696 PDF 1.4 transparency can use
> largechunks of memory.
>
>
>> Fix for 687696 PDF 1.4 transparency can use large chunks of memory.
>> Prior to this fix the handling of PDF 1.4 transparency features used one
>> or more full size image buffers.  With high resolutions and large images,
>> these buffers can be multiple gigabytes.  As a result, we were not able
>> to handle these files.
>>
>> DETAILS:
>>
>> PDF 1.4 transparency is handled by a device which is inserted into the
>> device chain.  Prior to this fix, this device was via the 'device filter'
>> logic.  With this fix, the PDF 1.4 transparency device is inserted via
>> the compositor device mechanism.  This change was made since the device
>> filter logic did not allow other devices to control of where the PDF 1.4
>> transparency device is placed in the device chain.  There is a device
>> proc which implements the create compositor action.  This allows more
>> control than the device filter logic.
>>
>> The first step in this fix was to convert the PDF 1.4 transparency
>> logic to use the create compositor mechanism.  For situations in which
>> the clist logic is not being used, the PDF 1.4 transparency device is
>> created and placed in the same location in the device chain as it was put
>> by the device filter logic.  In this situation, there is very little
>> logical difference in the operation of the PDF 1.4 transparency.
>>
>> For situations in which the clist is used, there are two PDF 1.4
>> compositing devices created.  There is one device before the clist when
>> the
>> clist is being written.  There is a second device after the clist when
>> the clist data is being read.
>>
>> The device before the clist serves two purposes, it provides a means for
>> implementing the process color model for the PDF 1.4 compositing.  (The
>> PDF 1.4 compositing may used a different process color model from the
>> output
>> device.)  Thus this device has color_info and encode/decode color procs
>> which match the PDF 1.4 blending color space.  This device also ensures
>> that the PDF 1.4 blending parameters (blend mode, shape, opacity, etc.)
>> are passed through the clist to the second PDF 1.4 device.
>>
>> The second PDF 1.4 device (the one when the clist is being read) is the
>> same PDF 1.4 compositing device which is used when the clist is not being
>> used.  This is the device which does the blending operations.  Since the
>> device is after the clist, the buffers that it needs are the size of a
>> single band instead of the full page.  This device is placed prior to
>> the memory device which is used for drawing the raster data.
>>
>> Two changes were made in the 'create compositor' logic.  The imager state
>> was being passed as a 'const'.  It is no longer const since the PDF 1.4
>> transparency compositing changes the cmap procs in the imager state to 
>> not
>> use transfer functions during color calculations.  The transfer functions
>> are applied later (and the cmap procs restored) when blended image data
>> is output from the PDF 1.4 compositing buffers.
>>
>> The second change to the create compositor logic is the addition of two
>> more procedures to the gs_composite_type_t.procs list.  These procedures
>> are used by the clist device create compositor routines.  The first
>> is used to create the PDF 1.4 clist write device.  The second saves and
>> restores the color_info field for the clist device (see below).
>>
>> Default versions of the added gs_composite_t.procs were also added for
>> use by the other compositor devices.  The defaults are no-ops.
>>
>> The clist logic has a few changes.  Since the PDF 1.4 compositor device
>> may use a different process color model from the output device, the 
>> number
>> of colorants and the depth (bits per pixel) can be different for the PDF
>> 1.4 compositor and the output device.  Thus changes were needed in parts
>> of the clist logic which assumed that these items were constant during 
>> the
>> processing of the clist.  An extra field was added to the device halftone
>> to indicate the number of colorants in effect when the device halftone 
>> was
>> created.  This data is included when the halftone is serialized for the
>> clist.  (Fortunately the PDF 1.4 compositing does not use halftones or
>> transfer functions.  So the device halftones and transfer functions do
>> not need to be changed when the PDF 1.4 compositor is installed.)
>>
>> As previously mentioned the create compositor procedures for the clist
>> writer and reader now call a couple of compositor specific routines.
>> These
>> allow the compositor to implement actions related to the clist device.
>> (The PDF 1.4 compositor uses these routines to create the 'write clist'
>> PDF 1.4 compositor and to save and restore the clist device color info.
>> The color_info field of the clist device is changed to match the process
>> color model of the PDF 1.4 compositor while the PDF 1.4 compositor is
>> active.
>>
>> A additional device parameter (PageUsesTransparency) was added.  This
>> parameter is sent by the PDF interpreter to indicate if PDF 1.4
>> transparency
>> will be used on the page.  The banding/no banding decision and the size
>> of a band includes an estimate of the size of the PDF 1.4 blending 
>> buffers
>> if this parameter is true.  Note:  The estimate of the size of the
>> blending buffers is not exact since this calculation is made before the
>> actual number of buffers, the number of blending colors and alpha 
>> channels
>> is known.  Fixed values are used for the estimated buffer size.  This is
>> still much better than the pervious situation, in which the only the
>> output raster size was being used.  The blending buffers can easily be
>> 30 or more times larger than the raster buffer.
>>
>> This fix also involves the removal of the PDF 1.4 'marking' devices.
>> The marking devices were used to actually implement the pixel marking.
>> The marking devices were created at the start of each high level drawing
>> operation and deleted after the completion of the operation.  The marking
>> device action has been merged into the primary PDF 1.4 compositing 
>> device.
>> This change eliminates some device creation overhead.  The overhead would
>> have been worse since most information about high level drawing 
>> operations
>> is not passed through the clist.  As a result it would have been 
>> necessary
>> to create marking devices for each low level operations.
>>
>>
>>
>> The following issues are not included in this fix but need to be
>> considered.
>>
>> 1)  The device filter logic is no longer used.  It may be desired to
>> remove
>> this logic.
>>
>> 2)  We currently have a 'create_compositor' device proc.  However there
>> is not a 'delete_compositor' device proc.  As a result, compositor 
>> devices
>> can stay in the device chain even when they are no longer needed.  The
>> current fix turns the PDF 1.4 compositor into a 'forwarding' device. 
>> This
>> is the same method used by the overprint compositor device.  I do not 
>> like
>> leaving these devices since there is some extra overhead and there may be
>> possible side effects.  (Testing has been done to try to find side 
>> effects
>> and none has been found with the current code.)
>>
>> 3)  This change only affects devices which can use the clist logic.  This
>> excludes the 'high' level devices.  These include the 'display', 'x11',
>> and 'pdfwrite' devices.  Thus these device will continue to use a full
>> image buffer.  It is possible to extend this fix to use a clist device
>> for the PDF 1.4 device for these devices.
>>
>>
>> Dan
>
>
> --------------------------------------------------------------------------------
>
>
>> _______________________________________________
>> gs-code-review mailing list
>> [email protected]
>> http://www.ghostscript.com/mailman/listinfo/gs-code-review
>>
>
>
>