Re: 4 byte length storage

On Mon, Dec 12, 2016 at 5:45 PM, Ruben Safir <ruben-v1XHP5QhJixWk0Htik3J/[email protected]> wrote:
> So now I have a question about the data in the IDAT chuncks.
>
> do they likiwise need to be reversed in the byte order before sending
> them to be decompressed with zlib?

The zlib specification defines the compressed data as a *byte* stream.
This is then split into sections (arbitrarily, except that each
section must be less than 2^31 bytes long) and sections are stored in
a sequence of IDAT chunks (no intervening chunks permitted).

The only 8/32 bit issue here is that each *section* is checksumed
using CRC32 (the cyclic redundancy code defined for Ethernet packets)
and, while this also checksums a *byte* stream, the result is a 32-bit
number which is encoded at the end of each chunk as a big-endian
sequence of four bytes.  This is exactly the same as the check sum on
every other chunk.

Internally the zlib stream is defined to end with an Adler32 checksum.
This is another 32-bit checksum of a sequence of bytes but now over
the whole data stream *before* compression.  This is, in turn, encoded
as a sequence of 4 bytes at the end of the *compressed* stream but
that definition comes from the zlib spec.  IRC it is big-endian (i.e.
the same as the CRC32 that immediately follows it.)

Anyway, the internals of zlib are not part of the PNG spec and they
have to deal with the simple fact that the zlib stream (well, the
deflate sub-part) is actually a *bit* stream that has to be mapped to
*bytes* to make a byte stream.

Roll-your-own zlib decompression is, in fact, very straight-forward
however nowhere near as simple as decoding a PNG stream using an
existing decompression implementation.

The relevant references are, high level to low:

PNG specification: http://www.libpng.org/pub/png/spec/iso/
    (Glenn gave an equivalent link to the W3C web site; I prefer the
ISO spec because it is the normative version, it also has lots of nice
explanation and pretty pictures.)
Zlib specification: https://www.ietf.org/rfc/rfc1950.txt
    Defines a big-endian multi-byte format
Deflate specification: https://www.ietf.org/rfc/rfc1951.txt
    Defines a little-endian multi-byte format

Notice that the zlib and deflate specifications use different ordering
for multi-byte numbers.  It is important to read the specification and
to read the *correct* specification when decoding data!  Indeed, as
the deflate specification points out, different numbers within the
different parts of the spec are encoded in different ways; in some
cases numbers are written as Huffman codes and, in that case, the most
significant parts of the Huffman code are written first; the code is
effectively big-endian (although that description has its own set of
assumptions.)

John Bowler

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot