Re: png++: diffs in test bench libpng12 vs. libpng16

On Sat, Nov 26, 2016 at 11:38 PM, Alex Shulgin <[email protected]> wrote:
> The way that test suite is designed is that it runs a number of conversions
> using png++ over about a 100 of pre-selected test files from the official
> pngsuite and compares the resulting files with the expected "standard"
> output files using cmp(1).

You mean direct comparison of numerical pixel values?

That doesn't work between major releases and sometimes fails within a
single major release.  It also doesn't work between different libpng
configurations; changing from floating to fixed point changes the
results as does changing the speed/accuracy configuration parameters.

Major releases do three major things in this area:

1) Fix bugs; 1.6 still has a long-standing really bad math error
(doing the gamma correction twice sometimes) and there are other,
lesser, errors.

2) Improve accuracy: this lead to small changes.  I think you saw the
effects of one change to how the RGB->gray coefficients are stored
that increased accuracy a very small amount.

3) Improve speed: sometimes this lessens accuracy and sometimes it
doesn't alter it but it does change the result; e.g. if a calculation
is accurate to +/-10 in the final number then a different calculation
can make two libpng versions different by up to 21!

> There seems to be a diff in the latest byte of the gAMA chunk (why?)

I couldn't tell from your email what changed (reformated diff -u of
two od streams is kind of difficult to decode).   The question is,
what is the *right* answer.  I'm assuming you set the gamma value in
the output and can look at the value as it passes through your hands.

Regardless of whether you use the FP or the fixed APIs PNG gamma
values should round-trip perfectly, but if you use an FP value like
1.000005, which is the PNG gamma value 100000.5, then I can imagine
that 1.2 would end up with 100,000 (round to 0), 1.6 should end up
with 100,001 (round to nearest) and another version could go back to
100,000 (round to even; the traditional high accuracy math choice).

I think it is quite likely that 1.2 didn't round in some cases; gamma
values are really only accurate to 1dp, so having 5dp accuracy is
definitely overkill.

> that is probably the reason why the complete IDAT chunk is completely
> different as well.

IDAT chunks depend on PNG line filter settings, zlib settings, IDAT
chunk size settings and potentially zlib version.  Those things can
potentially change between identical builds on different machines.
Expect IDAT chunks to change even between minor releases.

Still from sng, both files produce identical output:
...
> gAMA {1.0000}

You need five DP - the accuracy of a gAMA chunk - to know if it changed.

> 3. More interestingly, there is one failure where the differences are only
> in the IDAT chunk (but only in first 2 and latest 4 bytes before the final
> zero quadruplet):

The last 4 are the Adler32 checksum of the uncompressed data (used to
detect otherwise undetectable errors in the zlib stream if the PNG CRC
is ignored, which it often is on the last chunk because of the way
libpng works.)

The first 2 have been changed because the zlib window size has been
optimized.  I think that was done in 1.4.  The optimization is another
thing that can change between major releases.

> - Is it worth trying to find the root causes for these differences or was it
> never supposed to be 100% stable?

The PNGs will change with every major release and at least some of the
above can change in minor releases.  (Particularly as security fixes
are done in minor releases).

Other than as a way of discovering what might have changed in a
release I can't see any point:

> - Should I change my approach to testing or am I missing something?

Well, for sure test the *uncompressed* output.

When you do math using libpng expect it to change; even if 1.6 met the
requirements of human vision (+/-1%) there are major speed/time
tradeoffs.  Anyway, 8-bit encoded values cannot meet the requirements
for all images; 256 levels just aren't enough, so big errors creep in
to the darker colors regardless of the accuracy.

The only way to check the math stuff is to check it for the accuracy
you need, bearing in mind that libpng doesn't always do that well ;-)
People needing accuracy need to use 16-bit encodings and do the math
themselves if the accuracy doesn't correspond to the underlying libpng
assumption that the data is for human consumption.

> - Is there a way to adapt the png++ code to produce the same output both
> with libpng12 and 16?

Don't ask libpng to do math.  Most important of all, handle any gamma
yourself; it's easy to get an effectively exact answer if you have
floating point (ANSI-C float is sufficient), but bear in mind that the
actual accuracy of the data is severely limited with an 8-bit
encoding.

Any transform which changes the encoding is doing math, for example
converting a 16-bit value to an 8-bit value does math.  In most cases
(RGB to gray is an exception) the math is documented in the spec but
even so that doesn't help because:

1) libpng doesn't always do the right math; that's a bug.
2) The math produces exact encoded values, but doing it is slow, so
libpng currently uses approximations; that's a deliberate compromise.

John Bowler

------------------------------------------------------------------------------