Re: Technical Introduction update ?

Ahh; I see what you are saying.

The filtering we're talking about happens internally when you set the
compressor = DWAA or DWAB. You won't see the affect in exrheader.

So in your example there are multiple parts, all set to DWAB. But only part
3 will be lossy compressed, and of that, only the A channel.

Parts 0-2 will be lossless.

Karl

On Mon, Mar 28, 2016 at 4:35 PM, Deke Kincaid <[email protected]> wrote:

> The channel filtering you and Piotr mentioned as being automatic does not
> occur.  Every part whether it has XYZ, UV, Z, etc... is written with dwa
> compression.
>
> On Mon, Mar 28, 2016 at 4:29 PM, Karl Rasche <[email protected]> wrote:
>
>>
>> So.. what's the issue at hand then?
>>
>> On Mon, Mar 28, 2016 at 4:17 PM, Deke Kincaid <[email protected]>
>> wrote:
>>
>>> They must be implementing something incorrectly then.  Below is a
>>> truncated exrheader on a file out of Nuke with the "interleave" knob set to
>>> "channels" which means each layer is put into a separate part..
>>>
>>> file testMultiPartChannelsdwaa.exr:
>>>
>>> file format version: 2, flags 0x1000
>>>
>>>  part 0:
>>> channels (type chlist):
>>>     x, 16-bit floating-point, sampling 1 1
>>>     y, 16-bit floating-point, sampling 1 1
>>>     z, 16-bit floating-point, sampling 1 1
>>> chunkCount (type int): 34
>>> compression (type compression): dwa, small scanline blocks
>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>> dwaCompressionLevel (type float): 45
>>> lineOrder (type lineOrder): increasing y
>>> name (type string): "N.left"
>>> nuke/full_layer_names (type int): 0
>>> nuke/node_hash (type string): "368b9d1b87a4b787"
>>> nuke/version (type string): "9.0v8"
>>> pixelAspectRatio (type float): 1
>>> screenWindowCenter (type v2f): (0 0)
>>> screenWindowWidth (type float): 1
>>> type (type string): "scanlineimage"
>>> version (type int): 1
>>> view (type string): "left"
>>>
>>>  part 1:
>>> channels (type chlist):
>>>     x, 16-bit floating-point, sampling 1 1
>>>     y, 16-bit floating-point, sampling 1 1
>>>     z, 16-bit floating-point, sampling 1 1
>>> chunkCount (type int): 34
>>> compression (type compression): dwa, small scanline blocks
>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>> dwaCompressionLevel (type float): 45
>>> lineOrder (type lineOrder): increasing y
>>> name (type string): "P.left"
>>> pixelAspectRatio (type float): 1
>>> screenWindowCenter (type v2f): (0 0)
>>> screenWindowWidth (type float): 1
>>> type (type string): "scanlineimage"
>>> version (type int): 1
>>> view (type string): "left"
>>>
>>>  part 2:
>>> channels (type chlist):
>>>     Z, 16-bit floating-point, sampling 1 1
>>> chunkCount (type int): 34
>>> compression (type compression): dwa, small scanline blocks
>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>> dwaCompressionLevel (type float): 45
>>> lineOrder (type lineOrder): increasing y
>>> name (type string): "depth.left"
>>> pixelAspectRatio (type float): 1
>>> screenWindowCenter (type v2f): (0 0)
>>> screenWindowWidth (type float): 1
>>> type (type string): "scanlineimage"
>>> version (type int): 1
>>> view (type string): "left"
>>>
>>>  part 3:
>>> channels (type chlist):
>>>     u, 16-bit floating-point, sampling 1 1
>>>     v, 16-bit floating-point, sampling 1 1
>>> chunkCount (type int): 34
>>> compression (type compression): dwa, small scanline blocks
>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>> dwaCompressionLevel (type float): 45
>>> lineOrder (type lineOrder): increasing y
>>> name (type string): "motion.left"
>>> pixelAspectRatio (type float): 1
>>> screenWindowCenter (type v2f): (0 0)
>>> screenWindowWidth (type float): 1
>>> type (type string): "scanlineimage"
>>> version (type int): 1
>>> view (type string): "left"
>>>
>>>  part 4:
>>> channels (type chlist):
>>>     A, 16-bit floating-point, sampling 1 1
>>>     B, 16-bit floating-point, sampling 1 1
>>>     G, 16-bit floating-point, sampling 1 1
>>>     R, 16-bit floating-point, sampling 1 1
>>> chunkCount (type int): 34
>>> compression (type compression): dwa, small scanline blocks
>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>> dwaCompressionLevel (type float): 45
>>> lineOrder (type lineOrder): increasing y
>>> name (type string): "rgba.left"
>>> pixelAspectRatio (type float): 1
>>> screenWindowCenter (type v2f): (0 0)
>>> screenWindowWidth (type float): 1
>>> type (type string): "scanlineimage"
>>> version (type int): 1
>>> view (type string): "left"
>>>
>>>
>>>
>>> On Mon, Mar 28, 2016 at 3:16 PM, Piotr Stanczyk <
>>> [email protected]> wrote:
>>>
>>>> +1 .. there is nothing that the client code really has to do. It's
>>>> purely based off the channel names.
>>>>
>>>> -Piotr
>>>>
>>>>
>>>> On 28 March 2016 at 15:05, Karl Rasche <[email protected]> wrote:
>>>>
>>>>> Deke -
>>>>>
>>>>> All the channel rule setup should happen automagically; there isn't
>>>>> anything for Nuke to setup.
>>>>>
>>>>> That said, if your channels are all named RGB going into the write
>>>>> node, they'll all be handled the same by the compressor.
>>>>>
>>>>> Karl
>>>>>
>>>>> On Monday, March 28, 2016, Deke Kincaid <[email protected]> wrote:
>>>>>
>>>>>> Thanks for the info Karl.  Currently Nuke doesn't follow these rules
>>>>>> and when you pick DWAA and just compresses all channels the same instead of
>>>>>> selectively based on channel names. So we are having to write some hackery
>>>>>> to work around it at the moment.
>>>>>>
>>>>>> I'll put this into my feature request/bug to the Foundry.
>>>>>>
>>>>>> On Mon, Mar 28, 2016 at 2:15 PM, Karl Rasche <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> This won't help on the documentation front, but for your question
>>>>>>> about how channels are handled...
>>>>>>>
>>>>>>> This is setup in initializeChannelRules()
>>>>>>> <https://github.com/openexr/openexr/blob/master/OpenEXR/IlmImf/ImfDwaCompressor.cpp#L3120>.
>>>>>>> That logic is basically saying:
>>>>>>>
>>>>>>>
>>>>>>>    - If the channel name is A, use a lossless RLE compression on it
>>>>>>>    - If the channel name is {Y, BY, RY}, and the data is {FLOAT,
>>>>>>>    HALF}, lossy compress the channel
>>>>>>>    - If the channel name is {R, G, B}, and the data is {FLOAT,
>>>>>>>    HALF}, lossy compress the channels.
>>>>>>>    - If we have R, G, and B channels, convert to a color difference
>>>>>>>    format prior to quantization. Otherwise, deal with the channels
>>>>>>>    independently as-is.
>>>>>>>    - Otherwise, use a lossless compression on the data
>>>>>>>
>>>>>>> So channels named {y, u, U, v, V} should not be lossy compressed.
>>>>>>>
>>>>>>> The rule set that Jonathan is referring to is in
>>>>>>> initializeLegacyChannelRules()
>>>>>>> <https://github.com/openexr/openexr/blob/master/OpenEXR/IlmImf/ImfDwaCompressor.cpp#L3149>,
>>>>>>> and are slightly different from what 2.2 does by default.
>>>>>>>
>>>>>>> Most of this is hold-over from the time before multi-part files
>>>>>>> allowed more control over compression on different channel sets.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>> On Mon, Mar 28, 2016 at 1:56 PM, Richard Hadsell <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> There has been discussion on the [nuke-prerelease] e-mail list
>>>>>>>> about DWAA and DWAB compression with OpenExr 2.2.  I checked the OpenExr
>>>>>>>> Technical Introduction, but was disappointed to see that it has not been
>>>>>>>> updated since November 2013.  Is there any chance that someone is working
>>>>>>>> on an update?
>>>>>>>>
>>>>>>>> It is not difficult to use DWAA and DWAB compressions, which are
>>>>>>>> available in the compression header.  That is not the problem.  The
>>>>>>>> questions that I would like to see answered pertain more to lossy vs.
>>>>>>>> lossless compression.  The [nuke-prerelease] discussion included
>>>>>>>> interesting information like this:
>>>>>>>>
>>>>>>>> On Thu, Mar 17, 2016 at 1:03 PM, Jonathan Egstad <
>>>>>>>> [email protected]> wrote:
>>>>>>>> ...
>>>>>>>> > 3. Does the lossyness cause issues with data aov's, motion
>>>>>>>> vectors, uv, position, normals, etc...?
>>>>>>>>
>>>>>>>> Yup - it will destroy data AOVs like normals, positions, etc.  I
>>>>>>>> believe the codec recognizes typical color-channel names like
>>>>>>>> 'r/R/red/RED', etc and only lossy compresses those.  Channels it doesn't
>>>>>>>> recognize will get lossless compressed with I believe ZIP - alpha for
>>>>>>>> example is lossless compressed.
>>>>>>>> One gotcha is I think it traps 'Y/y, u/U, v/V' as valid color
>>>>>>>> channels and lossy-compresses those, which is a problem for xyz and uv
>>>>>>>> channels...
>>>>>>>> This is how our internal codec works so you might want to check
>>>>>>>> that the one in the official OpenEXR2.2+ release has these behaviors.
>>>>>>>> ...
>>>>>>>> (end quote)
>>>>>>>>
>>>>>>>> It would be great to see a list of those special channel names.
>>>>>>>> Are they the only channels that are compressed with DWAA/DWAB?  What
>>>>>>>> compression would be applied to the other channels?  Does this behavior
>>>>>>>> apply to all lossy compressions?
>>>>>>>>
>>>>>>>> I hope to see answers to these questions in an updated Technical
>>>>>>>> Intro (eventually).
>>>>>>>>
>>>>>>>> --
>>>>>>>> Dick Hadsell			203-992-6320  Fax: 203-992-6001
>>>>>>>> Reply-to:			[email protected]
>>>>>>>> Blue Sky Studios                http://www.blueskystudios.com
>>>>>>>> 1 American Lane, Greenwich, CT 06831-2560
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Openexr-devel mailing list
>>>>>>>> [email protected]
>>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Openexr-devel mailing list
>>>>>>> [email protected]
>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>>>
>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Openexr-devel mailing list
>>>>> [email protected]
>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Openexr-devel mailing list
>>> [email protected]
>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>
>>>
>>
>
> _______________________________________________
> Openexr-devel mailing list
> [email protected]
> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>
>

_______________________________________________
Openexr-devel mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/openexr-devel