Re: Technical Introduction update ?

That's correct -- channels named Y with DWAx will be treated as lossy.

So if you don't have a way to stick that pass in a part with a
lossless-only compressor (PIZ, ZIP, ZIPS, etc), you'd probably want to name
the channel y instead.

There's probably other gains (size+perf) to be had by changing the
compression type on data parts to something best suited for your data. The
non-alpha lossless mode in DWAx was never meant to be super efficient, it's
mostly there to minimize foot shooting.

Karl



On Tue, Mar 29, 2016 at 9:12 AM, Deke Kincaid <[email protected]> wrote:

> So dumb question I already know the answer to but I just want to
> confirm.  You scope the lossy channels to R,G,B,Y,BY,RY.  So I'm assuming
> you should make sure your xyz pass channels are never named XYZ?
>
> On Mon, Mar 28, 2016 at 5:31 PM, Karl Rasche <[email protected]> wrote:
>
>> I agree its a little funky to have a compression scheme (or 4 in this
>> case) within a compression type, but its much nicer than accidently lossy
>> compressing your z channel.
>>
>> There isn't a great way to introspect it, short of a hex editor (the
>> choices made are recorded at the start of the compressed block).
>>
>> I've had to use that giant comment myself a few times to refresh my
>> memory after being away from the code a while.
>>
>> Karl
>>
>> On Monday, March 28, 2016, Deke Kincaid <[email protected]> wrote:
>>
>>> Ahh got it.  I thought exrheader would explicitly say "zip, individual
>>> scanlines" next to those parts and not "dwa, small scanline". blocks".  Is
>>> there a way to introspect the compression for each part that dwaa is doing
>>> auto-magically?
>>>
>>> Richard: sorry for hijacking your thread 😈  It appears a lot of this
>>> is explained in the header for ImfDwaCompressor.cpp.  Most of the time the
>>> top of the EXR source is just copy-write data and basic comments so I
>>> didn't realize there as a whole book chapter in there on the topic. 😀
>>>
>>> On Mon, Mar 28, 2016 at 4:43 PM, Karl Rasche <[email protected]>
>>> wrote:
>>>
>>>>
>>>> Ahh; I see what you are saying.
>>>>
>>>> The filtering we're talking about happens internally when you set the
>>>> compressor = DWAA or DWAB. You won't see the affect in exrheader.
>>>>
>>>> So in your example there are multiple parts, all set to DWAB. But only
>>>> part 3 will be lossy compressed, and of that, only the A channel.
>>>>
>>>> Parts 0-2 will be lossless.
>>>>
>>>> Karl
>>>>
>>>> On Mon, Mar 28, 2016 at 4:35 PM, Deke Kincaid <[email protected]>
>>>> wrote:
>>>>
>>>>> The channel filtering you and Piotr mentioned as being automatic does
>>>>> not occur.  Every part whether it has XYZ, UV, Z, etc... is written with
>>>>> dwa compression.
>>>>>
>>>>> On Mon, Mar 28, 2016 at 4:29 PM, Karl Rasche <[email protected]>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> So.. what's the issue at hand then?
>>>>>>
>>>>>> On Mon, Mar 28, 2016 at 4:17 PM, Deke Kincaid <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> They must be implementing something incorrectly then.  Below is a
>>>>>>> truncated exrheader on a file out of Nuke with the "interleave" knob set to
>>>>>>> "channels" which means each layer is put into a separate part..
>>>>>>>
>>>>>>> file testMultiPartChannelsdwaa.exr:
>>>>>>>
>>>>>>> file format version: 2, flags 0x1000
>>>>>>>
>>>>>>>  part 0:
>>>>>>> channels (type chlist):
>>>>>>>     x, 16-bit floating-point, sampling 1 1
>>>>>>>     y, 16-bit floating-point, sampling 1 1
>>>>>>>     z, 16-bit floating-point, sampling 1 1
>>>>>>> chunkCount (type int): 34
>>>>>>> compression (type compression): dwa, small scanline blocks
>>>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>>>> dwaCompressionLevel (type float): 45
>>>>>>> lineOrder (type lineOrder): increasing y
>>>>>>> name (type string): "N.left"
>>>>>>> nuke/full_layer_names (type int): 0
>>>>>>> nuke/node_hash (type string): "368b9d1b87a4b787"
>>>>>>> nuke/version (type string): "9.0v8"
>>>>>>> pixelAspectRatio (type float): 1
>>>>>>> screenWindowCenter (type v2f): (0 0)
>>>>>>> screenWindowWidth (type float): 1
>>>>>>> type (type string): "scanlineimage"
>>>>>>> version (type int): 1
>>>>>>> view (type string): "left"
>>>>>>>
>>>>>>>  part 1:
>>>>>>> channels (type chlist):
>>>>>>>     x, 16-bit floating-point, sampling 1 1
>>>>>>>     y, 16-bit floating-point, sampling 1 1
>>>>>>>     z, 16-bit floating-point, sampling 1 1
>>>>>>> chunkCount (type int): 34
>>>>>>> compression (type compression): dwa, small scanline blocks
>>>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>>>> dwaCompressionLevel (type float): 45
>>>>>>> lineOrder (type lineOrder): increasing y
>>>>>>> name (type string): "P.left"
>>>>>>> pixelAspectRatio (type float): 1
>>>>>>> screenWindowCenter (type v2f): (0 0)
>>>>>>> screenWindowWidth (type float): 1
>>>>>>> type (type string): "scanlineimage"
>>>>>>> version (type int): 1
>>>>>>> view (type string): "left"
>>>>>>>
>>>>>>>  part 2:
>>>>>>> channels (type chlist):
>>>>>>>     Z, 16-bit floating-point, sampling 1 1
>>>>>>> chunkCount (type int): 34
>>>>>>> compression (type compression): dwa, small scanline blocks
>>>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>>>> dwaCompressionLevel (type float): 45
>>>>>>> lineOrder (type lineOrder): increasing y
>>>>>>> name (type string): "depth.left"
>>>>>>> pixelAspectRatio (type float): 1
>>>>>>> screenWindowCenter (type v2f): (0 0)
>>>>>>> screenWindowWidth (type float): 1
>>>>>>> type (type string): "scanlineimage"
>>>>>>> version (type int): 1
>>>>>>> view (type string): "left"
>>>>>>>
>>>>>>>  part 3:
>>>>>>> channels (type chlist):
>>>>>>>     u, 16-bit floating-point, sampling 1 1
>>>>>>>     v, 16-bit floating-point, sampling 1 1
>>>>>>> chunkCount (type int): 34
>>>>>>> compression (type compression): dwa, small scanline blocks
>>>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>>>> dwaCompressionLevel (type float): 45
>>>>>>> lineOrder (type lineOrder): increasing y
>>>>>>> name (type string): "motion.left"
>>>>>>> pixelAspectRatio (type float): 1
>>>>>>> screenWindowCenter (type v2f): (0 0)
>>>>>>> screenWindowWidth (type float): 1
>>>>>>> type (type string): "scanlineimage"
>>>>>>> version (type int): 1
>>>>>>> view (type string): "left"
>>>>>>>
>>>>>>>  part 4:
>>>>>>> channels (type chlist):
>>>>>>>     A, 16-bit floating-point, sampling 1 1
>>>>>>>     B, 16-bit floating-point, sampling 1 1
>>>>>>>     G, 16-bit floating-point, sampling 1 1
>>>>>>>     R, 16-bit floating-point, sampling 1 1
>>>>>>> chunkCount (type int): 34
>>>>>>> compression (type compression): dwa, small scanline blocks
>>>>>>> dataWindow (type box2i): (0 240) - (2047 1315)
>>>>>>> displayWindow (type box2i): (0 0) - (2047 1555)
>>>>>>> dwaCompressionLevel (type float): 45
>>>>>>> lineOrder (type lineOrder): increasing y
>>>>>>> name (type string): "rgba.left"
>>>>>>> pixelAspectRatio (type float): 1
>>>>>>> screenWindowCenter (type v2f): (0 0)
>>>>>>> screenWindowWidth (type float): 1
>>>>>>> type (type string): "scanlineimage"
>>>>>>> version (type int): 1
>>>>>>> view (type string): "left"
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 28, 2016 at 3:16 PM, Piotr Stanczyk <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> +1 .. there is nothing that the client code really has to do. It's
>>>>>>>> purely based off the channel names.
>>>>>>>>
>>>>>>>> -Piotr
>>>>>>>>
>>>>>>>>
>>>>>>>> On 28 March 2016 at 15:05, Karl Rasche <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Deke -
>>>>>>>>>
>>>>>>>>> All the channel rule setup should happen automagically; there
>>>>>>>>> isn't anything for Nuke to setup.
>>>>>>>>>
>>>>>>>>> That said, if your channels are all named RGB going into the write
>>>>>>>>> node, they'll all be handled the same by the compressor.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>> On Monday, March 28, 2016, Deke Kincaid <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the info Karl.  Currently Nuke doesn't follow these
>>>>>>>>>> rules and when you pick DWAA and just compresses all channels the same
>>>>>>>>>> instead of selectively based on channel names. So we are having to write
>>>>>>>>>> some hackery to work around it at the moment.
>>>>>>>>>>
>>>>>>>>>> I'll put this into my feature request/bug to the Foundry.
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 28, 2016 at 2:15 PM, Karl Rasche <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This won't help on the documentation front, but for your
>>>>>>>>>>> question about how channels are handled...
>>>>>>>>>>>
>>>>>>>>>>> This is setup in initializeChannelRules()
>>>>>>>>>>> <https://github.com/openexr/openexr/blob/master/OpenEXR/IlmImf/ImfDwaCompressor.cpp#L3120>.
>>>>>>>>>>> That logic is basically saying:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>    - If the channel name is A, use a lossless RLE compression
>>>>>>>>>>>    on it
>>>>>>>>>>>    - If the channel name is {Y, BY, RY}, and the data is
>>>>>>>>>>>    {FLOAT, HALF}, lossy compress the channel
>>>>>>>>>>>    - If the channel name is {R, G, B}, and the data is {FLOAT,
>>>>>>>>>>>    HALF}, lossy compress the channels.
>>>>>>>>>>>    - If we have R, G, and B channels, convert to a color
>>>>>>>>>>>    difference format prior to quantization. Otherwise, deal with the channels
>>>>>>>>>>>    independently as-is.
>>>>>>>>>>>    - Otherwise, use a lossless compression on the data
>>>>>>>>>>>
>>>>>>>>>>> So channels named {y, u, U, v, V} should not be lossy compressed.
>>>>>>>>>>>
>>>>>>>>>>> The rule set that Jonathan is referring to is in
>>>>>>>>>>> initializeLegacyChannelRules()
>>>>>>>>>>> <https://github.com/openexr/openexr/blob/master/OpenEXR/IlmImf/ImfDwaCompressor.cpp#L3149>,
>>>>>>>>>>> and are slightly different from what 2.2 does by default.
>>>>>>>>>>>
>>>>>>>>>>> Most of this is hold-over from the time before multi-part files
>>>>>>>>>>> allowed more control over compression on different channel sets.
>>>>>>>>>>>
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Mar 28, 2016 at 1:56 PM, Richard Hadsell <
>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> There has been discussion on the [nuke-prerelease] e-mail list
>>>>>>>>>>>> about DWAA and DWAB compression with OpenExr 2.2.  I checked the OpenExr
>>>>>>>>>>>> Technical Introduction, but was disappointed to see that it has not been
>>>>>>>>>>>> updated since November 2013.  Is there any chance that someone is working
>>>>>>>>>>>> on an update?
>>>>>>>>>>>>
>>>>>>>>>>>> It is not difficult to use DWAA and DWAB compressions, which
>>>>>>>>>>>> are available in the compression header.  That is not the problem.  The
>>>>>>>>>>>> questions that I would like to see answered pertain more to lossy vs.
>>>>>>>>>>>> lossless compression.  The [nuke-prerelease] discussion included
>>>>>>>>>>>> interesting information like this:
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 17, 2016 at 1:03 PM, Jonathan Egstad <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>> ...
>>>>>>>>>>>> > 3. Does the lossyness cause issues with data aov's, motion
>>>>>>>>>>>> vectors, uv, position, normals, etc...?
>>>>>>>>>>>>
>>>>>>>>>>>> Yup - it will destroy data AOVs like normals, positions, etc.
>>>>>>>>>>>> I believe the codec recognizes typical color-channel names like
>>>>>>>>>>>> 'r/R/red/RED', etc and only lossy compresses those.  Channels it doesn't
>>>>>>>>>>>> recognize will get lossless compressed with I believe ZIP - alpha for
>>>>>>>>>>>> example is lossless compressed.
>>>>>>>>>>>> One gotcha is I think it traps 'Y/y, u/U, v/V' as valid color
>>>>>>>>>>>> channels and lossy-compresses those, which is a problem for xyz and uv
>>>>>>>>>>>> channels...
>>>>>>>>>>>> This is how our internal codec works so you might want to check
>>>>>>>>>>>> that the one in the official OpenEXR2.2+ release has these behaviors.
>>>>>>>>>>>> ...
>>>>>>>>>>>> (end quote)
>>>>>>>>>>>>
>>>>>>>>>>>> It would be great to see a list of those special channel
>>>>>>>>>>>> names.  Are they the only channels that are compressed with DWAA/DWAB?
>>>>>>>>>>>> What compression would be applied to the other channels?  Does this
>>>>>>>>>>>> behavior apply to all lossy compressions?
>>>>>>>>>>>>
>>>>>>>>>>>> I hope to see answers to these questions in an updated
>>>>>>>>>>>> Technical Intro (eventually).
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Dick Hadsell			203-992-6320  Fax: 203-992-6001
>>>>>>>>>>>> Reply-to:			[email protected]
>>>>>>>>>>>> Blue Sky Studios                http://www.blueskystudios.com
>>>>>>>>>>>> 1 American Lane, Greenwich, CT 06831-2560
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Openexr-devel mailing list
>>>>>>>>>>>> [email protected]
>>>>>>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Openexr-devel mailing list
>>>>>>>>>>> [email protected]
>>>>>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Openexr-devel mailing list
>>>>>>>>> [email protected]
>>>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Openexr-devel mailing list
>>>>>>> [email protected]
>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Openexr-devel mailing list
>>>>> [email protected]
>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel
>>>>>
>>>>>
>>>>
>>>
>

_______________________________________________
Openexr-devel mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/openexr-devel