Re: Technical Introduction update ?

I agree its a little funky to have a compression scheme (or 4 in this case)
within a compression type, but its much nicer than accidently lossy
compressing your z channel.

There isn't a great way to introspect it, short of a hex editor (the
choices made are recorded at the start of the compressed block).

I've had to use that giant comment myself a few times to refresh my memory
after being away from the code a while.

Karl

On Monday, March 28, 2016, Deke Kincaid <[email protected]> wrote:

> Ahh got it.  I thought exrheader would explicitly say "zip, individual
> scanlines" next to those parts and not "dwa, small scanline". blocks".  Is
> there a way to introspect the compression for each part that dwaa is doing
> auto-magically?
>
> Richard: sorry for hijacking your thread 😈  It appears a lot of this is
> explained in the header for ImfDwaCompressor.cpp.  Most of the time the top
> of the EXR source is just copy-write data and basic comments so I didn't
> realize there as a whole book chapter in there on the topic. 😀
>
> On Mon, Mar 28, 2016 at 4:43 PM, Karl Rasche <[email protected]
> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>
>>
>> Ahh; I see what you are saying.
>>
>> The filtering we're talking about happens internally when you set the
>> compressor = DWAA or DWAB. You won't see the affect in exrheader.
>>
>> So in your example there are multiple parts, all set to DWAB. But only
>> part 3 will be lossy compressed, and of that, only the A channel.
>>
>> Parts 0-2 will be lossless.
>>
>> Karl
>>
>> On Mon, Mar 28, 2016 at 4:35 PM, Deke Kincaid <[email protected]
>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>>
>>> The channel filtering you and Piotr mentioned as being automatic does
>>> not occur.  Every part whether it has XYZ, UV, Z, etc... is written with
>>> dwa compression.
>>>
>>> On Mon, Mar 28, 2016 at 4:29 PM, Karl Rasche <[email protected]
>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>>>
>>>>
>>>> So.. what's the issue at hand then?
>>>>
>>>> On Mon, Mar 28, 2016 at 4:17 PM, Deke Kincaid <[email protected]
>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>>>>
>>>>> They must be implementing something incorrectly then.  Below is a
>>>>> truncated exrheader on a file out of Nuke with the "interleave" knob set to
>>>>> "channels" which means each layer is put into a separate part..
>>>>>
>>>>> file testMultiPartChannelsdwaa.exr:
>>>>>
>>>>> file format version: 2, flags 0x1000
>>>>>
>>>>>  part 0:
>>>>> channels (type chlist):
>>>>>     x, 16-bit floating-point, sampling 1 1
>>>>>     y, 16-bit floating-point, sampling 1 1
>>>>>     z, 16-bit floating-point, sampling 1 1
>>>>> chunkCount (type int): 34
>>>>> compression (type compression): dwa, small scanline blocks
>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>> dwaCompressionLevel (type float): 45
>>>>> lineOrder (type lineOrder): increasing y
>>>>> name (type string): "N.left"
>>>>> nuke/full_layer_names (type int): 0
>>>>> nuke/node_hash (type string): "368b9d1b87a4b787"
>>>>> nuke/version (type string): "9.0v8"
>>>>> pixelAspectRatio (type float): 1
>>>>> screenWindowCenter (type v2f): (0 0)
>>>>> screenWindowWidth (type float): 1
>>>>> type (type string): "scanlineimage"
>>>>> version (type int): 1
>>>>> view (type string): "left"
>>>>>
>>>>>  part 1:
>>>>> channels (type chlist):
>>>>>     x, 16-bit floating-point, sampling 1 1
>>>>>     y, 16-bit floating-point, sampling 1 1
>>>>>     z, 16-bit floating-point, sampling 1 1
>>>>> chunkCount (type int): 34
>>>>> compression (type compression): dwa, small scanline blocks
>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>> dwaCompressionLevel (type float): 45
>>>>> lineOrder (type lineOrder): increasing y
>>>>> name (type string): "P.left"
>>>>> pixelAspectRatio (type float): 1
>>>>> screenWindowCenter (type v2f): (0 0)
>>>>> screenWindowWidth (type float): 1
>>>>> type (type string): "scanlineimage"
>>>>> version (type int): 1
>>>>> view (type string): "left"
>>>>>
>>>>>  part 2:
>>>>> channels (type chlist):
>>>>>     Z, 16-bit floating-point, sampling 1 1
>>>>> chunkCount (type int): 34
>>>>> compression (type compression): dwa, small scanline blocks
>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>> dwaCompressionLevel (type float): 45
>>>>> lineOrder (type lineOrder): increasing y
>>>>> name (type string): "depth.left"
>>>>> pixelAspectRatio (type float): 1
>>>>> screenWindowCenter (type v2f): (0 0)
>>>>> screenWindowWidth (type float): 1
>>>>> type (type string): "scanlineimage"
>>>>> version (type int): 1
>>>>> view (type string): "left"
>>>>>
>>>>>  part 3:
>>>>> channels (type chlist):
>>>>>     u, 16-bit floating-point, sampling 1 1
>>>>>     v, 16-bit floating-point, sampling 1 1
>>>>> chunkCount (type int): 34
>>>>> compression (type compression): dwa, small scanline blocks
>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>> dwaCompressionLevel (type float): 45
>>>>> lineOrder (type lineOrder): increasing y
>>>>> name (type string): "motion.left"
>>>>> pixelAspectRatio (type float): 1
>>>>> screenWindowCenter (type v2f): (0 0)
>>>>> screenWindowWidth (type float): 1
>>>>> type (type string): "scanlineimage"
>>>>> version (type int): 1
>>>>> view (type string): "left"
>>>>>
>>>>>  part 4:
>>>>> channels (type chlist):
>>>>>     A, 16-bit floating-point, sampling 1 1
>>>>>     B, 16-bit floating-point, sampling 1 1
>>>>>     G, 16-bit floating-point, sampling 1 1
>>>>>     R, 16-bit floating-point, sampling 1 1
>>>>> chunkCount (type int): 34
>>>>> compression (type compression): dwa, small scanline blocks
>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>> dwaCompressionLevel (type float): 45
>>>>> lineOrder (type lineOrder): increasing y
>>>>> name (type string): "rgba.left"
>>>>> pixelAspectRatio (type float): 1
>>>>> screenWindowCenter (type v2f): (0 0)
>>>>> screenWindowWidth (type float): 1
>>>>> type (type string): "scanlineimage"
>>>>> version (type int): 1
>>>>> view (type string): "left"
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 28, 2016 at 3:16 PM, Piotr Stanczyk <
>>>>> [email protected]
>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>>>>>
>>>>>> +1 .. there is nothing that the client code really has to do. It's
>>>>>> purely based off the channel names.
>>>>>>
>>>>>> -Piotr
>>>>>>
>>>>>>
>>>>>> On 28 March 2016 at 15:05, Karl Rasche <[email protected]
>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>>>>>>
>>>>>>> Deke -
>>>>>>>
>>>>>>> All the channel rule setup should happen automagically; there isn't
>>>>>>> anything for Nuke to setup.
>>>>>>>
>>>>>>> That said, if your channels are all named RGB going into the write
>>>>>>> node, they'll all be handled the same by the compressor.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>> On Monday, March 28, 2016, Deke Kincaid <[email protected]
>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote:
>>>>>>>
>>>>>>>> Thanks for the info Karl.  Currently Nuke doesn't follow these
>>>>>>>> rules and when you pick DWAA and just compresses all channels the same
>>>>>>>> instead of selectively based on channel names. So we are having to write
>>>>>>>> some hackery to work around it at the moment.
>>>>>>>>
>>>>>>>> I'll put this into my feature request/bug to the Foundry.
>>>>>>>>
>>>>>>>> On Mon, Mar 28, 2016 at 2:15 PM, Karl Rasche <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> This won't help on the documentation front, but for your question
>>>>>>>>> about how channels are handled...
>>>>>>>>>
>>>>>>>>> This is setup in initializeChannelRules()
>>>>>>>>> <https://github.com/openexr/openexr/blob/master/OpenEXR/IlmImf/ImfDwaCompressor.cpp#L3120>.
>>>>>>>>> That logic is basically saying:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - If the channel name is A, use a lossless RLE compression on
>>>>>>>>>    it
>>>>>>>>>    - If the channel name is {Y, BY, RY}, and the data is {FLOAT,
>>>>>>>>>    HALF}, lossy compress the channel
>>>>>>>>>    - If the channel name is {R, G, B}, and the data is {FLOAT,
>>>>>>>>>    HALF}, lossy compress the channels.
>>>>>>>>>    - If we have R, G, and B channels, convert to a color
>>>>>>>>>    difference format prior to quantization. Otherwise, deal with the channels
>>>>>>>>>    independently as-is.
>>>>>>>>>    - Otherwise, use a lossless compression on the data
>>>>>>>>>
>>>>>>>>> So channels named {y, u, U, v, V} should not be lossy compressed.
>>>>>>>>>
>>>>>>>>> The rule set that Jonathan is referring to is in
>>>>>>>>> initializeLegacyChannelRules()
>>>>>>>>> <https://github.com/openexr/openexr/blob/master/OpenEXR/IlmImf/ImfDwaCompressor.cpp#L3149>,
>>>>>>>>> and are slightly different from what 2.2 does by default.
>>>>>>>>>
>>>>>>>>> Most of this is hold-over from the time before multi-part files
>>>>>>>>> allowed more control over compression on different channel sets.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>> On Mon, Mar 28, 2016 at 1:56 PM, Richard Hadsell <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> There has been discussion on the [nuke-prerelease] e-mail list
>>>>>>>>>> about DWAA and DWAB compression with OpenExr 2.2.  I checked the OpenExr
>>>>>>>>>> Technical Introduction, but was disappointed to see that it has not been
>>>>>>>>>> updated since November 2013.  Is there any chance that someone is working
>>>>>>>>>> on an update?
>>>>>>>>>>
>>>>>>>>>> It is not difficult to use DWAA and DWAB compressions, which are
>>>>>>>>>> available in the compression header.  That is not the problem.  The
>>>>>>>>>> questions that I would like to see answered pertain more to lossy vs.
>>>>>>>>>> lossless compression.  The [nuke-prerelease] discussion included
>>>>>>>>>> interesting information like this:
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 17, 2016 at 1:03 PM, Jonathan Egstad <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>> ...
>>>>>>>>>> > 3. Does the lossyness cause issues with data aov's, motion
>>>>>>>>>> vectors, uv, position, normals, etc...?
>>>>>>>>>>
>>>>>>>>>> Yup - it will destroy data AOVs like normals, positions, etc.  I
>>>>>>>>>> believe the codec recognizes typical color-channel names like
>>>>>>>>>> 'r/R/red/RED', etc and only lossy compresses those.  Channels it doesn't
>>>>>>>>>> recognize will get lossless compressed with I believe ZIP - alpha for
>>>>>>>>>> example is lossless compressed.
>>>>>>>>>> One gotcha is I think it traps 'Y/y, u/U, v/V' as valid color
>>>>>>>>>> channels and lossy-compresses those, which is a problem for xyz and uv
>>>>>>>>>> channels...
>>>>>>>>>> This is how our internal codec works so you might want to check
>>>>>>>>>> that the one in the official OpenEXR2.2+ release has these behaviors.
>>>>>>>>>> ...
>>>>>>>>>> (end quote)
>>>>>>>>>>
>>>>>>>>>> It would be great to see a list of those special channel names.
>>>>>>>>>> Are they the only channels that are compressed with DWAA/DWAB?  What
>>>>>>>>>> compression would be applied to the other channels?  Does this behavior
>>>>>>>>>> apply to all lossy compressions?
>>>>>>>>>>
>>>>>>>>>> I hope to see answers to these questions in an updated Technical
>>>>>>>>>> Intro (eventually).
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Dick Hadsell			203-992-6320  Fax: 203-992-6001
>>>>>>>>>> Reply-to:			[email protected]
>>>>>>>>>> Blue Sky Studios                http://www.blueskystudios.com
>>>>>>>>>> 1 American Lane, Greenwich, CT 06831-2560
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Openexr-devel mailing list
>>>>>>>>>> [email protected]
>>>>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Openexr-devel mailing list
>>>>>>>>> [email protected]
>>>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Openexr-devel mailing list
>>>>>>> [email protected]
>>>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>
>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Openexr-devel mailing list
>>>>> [email protected]
>>>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>
>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Openexr-devel mailing list
>>> [email protected]
>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>
>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>
>>>
>>
>

_______________________________________________
Openexr-devel mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/openexr-devel