RE: The right formula for rounding

Igor,

I said that the addition of 0.5 for routing in following equation
was 'minor'.

> >    (int) (pdev->HWResolution[0] * 0x10000 + 0.5);


I DID NOT CLAIM THAT ROUNDING WAS MINOR IN ALL SITUATIONS!!!


In the given situation, we are rounding only after we have
multiplied by 0x10000.  Assuming that the given resolution
is around 72 dpi (which is on the lower end of the range of
typical values) the effective error is 0.5 / ( 72 * 0x10000 )
	= 0.000000106 or 0.0000105 %


Dan


-----Original Message-----
From: Igor V. Melichev [mailto:[email protected]]
Sent: Sunday, February 15, 2004 12:47 AM
To: [email protected]; tech; [email protected];
[email protected]
Subject: The right formula for rounding


Developers,

Many of Ghostscript's quirks happen due to an inaccurate rounding.
The correct rounding formula from math is

       i =  (int)(x + 0.5)

rather than

       i = (int)x

The constant 0.5 is not a "minor detail".
Missing it, the result is biased in average.
This is especially danger with iterations,
for example with a subdivision of a curve while flattening.

With HWResolution an iteration is rare
but it is still possible if an user iteratively runs Photoshop
and Ghostscript multiple times while designing a document.
He'll wonder why the document page becomes smaller
and smaller in each next revision.

Please be accurate developing Ghostscript.
Save our time from maintenance of a poor code.

Igor.


----- Original Message -----
From: "Dan Coby" <[email protected]>
To: "Igor V. Melichev" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, February 15, 2004 12:34 AM
Subject: RE: [gs-code-review] Fix for 687294 psdcmyk device does not
writeresolution correctly to PSD files


>
> Igor,
>
> >I can't review your algorithm because I have no knowledge about the  PSD
> >format.
> >The syntax looks perfect.
>
> If you are really interested, I can send you a copy of the specifications.
> My copy of Photoshop includes a copy of the Photoshop SDK.  The SDK
> includes the specifications.  If you have Photoshop then you probably
> also have the specifications.
>
>
> >Don't know where it's fine to assume that HWResolution is an integer.
> >I would code
> >
> >    (int) (pdev->HWResolution[0] * 0x10000 + 0.5);
>
> There is no assumption that HWResolution is an integer.  It is defined
> as a 'float'.  The value is cast to an integer only after multiplying
> by 0x10000.  The cast is needed because Photoshop uses a 'fixed' format
> for storing resolution.  The 'fixed' format consists of a 16 bit integer
> and a 16 bit fraction.  The addition of 0.5 for rounding is minor but I
> have added it to the code.
>
>
> Dan