Re: OpenEXR Perceptual Image Differencing
Larry Gritz <[email protected]> Tue, 30 May 2017 18:22:51 -0700
| Newsgroups | gmane.comp.video.openexr.devel,gmane.comp.video.openexr.user |
|---|---|
| Message-ID | <[email protected]> |
--===============8845638310507557636== Content-Type: multipart/alternative; boundary="Apple-Mail=_19FACB9C-F528-4814-BE9D-96D0BA97AF4C" --Apple-Mail=_19FACB9C-F528-4814-BE9D-96D0BA97AF4C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Oy, have I really had this hanging over my head since 2014? Sorry, = Thiago, if you are reading this. Tip: I think that in addition to --fail and --failpercent, it may also = be wise to set --hardfail as well (generally to a larger value than you = used for --fail). The complexity here reflects that it's quite tricky to = design good "image is close enough" criteria. You want to allow some = pixels to differ by a bit -- but not too many -- but also that if any = one pixel is TOO far off, that's also a failure even if there is just = one. It's tricky to keep straight, and I have to read the docs carefully = every time I use it to remind myself which flags mean what. > On May 30, 2017, at 5:52 PM, Ryan Heniser <[email protected]> wrote: >=20 > Hi Larry, >=20 > Thank you for taking the time to provide such a thorough response. = Sorry, you had to repeat yourself. I just found Thiago=E2=80=99s post = about this from 2014 = (http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/2014-Augu= st/013430.html = <http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/2014-Augu= st/013430.html>). Yes, the --diff option with --fail 0.05 and = --failpercent 0.1 are working well to address LSB error for my automated = tests. For anyone new to oiiotool, --fail and --failpercent must go = before =E2=80=94diff. >=20 > Thank you, > Ryan >=20 >=20 >> On 31 May 2017, at 5:07 am, Larry Gritz <[email protected] = <mailto:[email protected]>> wrote: >>=20 >> I should add that, while -pdiff seemed like a good idea at the time, = I'm not sure it's any better in practice than using regular --diff with = wisely chosen epsilons to allow a certain amount of LSB error to not be = counted as failure. I do that all the time for testsuite image = comparisons, where I expect just a little bit of error against the = reference output, for different platforms or different versions of = dependent libraries. >>=20 >> Oh my, this is embarrassing. I see now that the oiiotool docs do not = even explain how those thresholds can be set. So sorry. The undocumented = commands are --fail, --failpercent, --warn, --warnpercent, --hardfail. = It may be undocumented in the oiiotool section of the OIIO docs, but the = same commands are explained in the "idiff" section of the docs (they = work the same in the two programs). I'll get that oversight fixed. >>=20 >> It's also been suggested that I look into a replacement (or = additional) perceptual diff based on CIEDE2000 or DE2000K (google them = for details). Seems like a good idea, but I haven't had time to do it. = I'd certainly be very supportive if anybody felt like it would be a = project they wanted to work on and submit it as a PR. >>=20 >>=20 >>> On May 29, 2017, at 10:16 PM, Ryan Heniser <[email protected] = <mailto:[email protected]>> wrote: >>>=20 >>> Ugg=E2=80=A6 Sorry, I overlooked >>>=20 >>> oiiotool --help >>> ... >>> --pdiff Print report on the perceptual = difference of two images >>>=20 >>>=20 >>>> On 30 May 2017, at 2:34 pm, Ryan Heniser <[email protected] = <mailto:[email protected]>> wrote: >>>>=20 >>>> Is there a popular or de facto industry standard tool for OpenEXR = perceptual image differencing?=20 >>>>=20 >>>> Kind regards, >>>>=20 >>>> ryan heniser >>>> senior shading technical director | [email protected] = <mailto:[email protected]> >>>> rising sun pictures | http://www.rsp.com.au = <http://www.rsp.com.au/> >>>> +61 8 8400 6400=20 >>>=20 >>> _______________________________________________ >>> Openexr-devel mailing list >>> [email protected] <mailto:[email protected]> >>> https://lists.nongnu.org/mailman/listinfo/openexr-devel = <https://lists.nongnu.org/mailman/listinfo/openexr-devel> >>=20 >> -- >> Larry Gritz >> [email protected] <mailto:[email protected]> >>=20 >>=20 >=20 -- Larry Gritz [email protected] --Apple-Mail=_19FACB9C-F528-4814-BE9D-96D0BA97AF4C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">Oy, have I really had this hanging over my head since 2014? = Sorry, Thiago, if you are reading this.<div class=3D""><br = class=3D""></div><div class=3D"">Tip: I think that in addition to --fail = and --failpercent, it may also be wise to set --hardfail as well = (generally to a larger value than you used for --fail). The complexity = here reflects that it's quite tricky to design good "image is close = enough" criteria. You want to allow some pixels to differ by a bit -- = but not too many -- but also that if any one pixel is TOO far off, = that's also a failure even if there is just one. It's tricky to keep = straight, and I have to read the docs carefully every time I use it to = remind myself which flags mean what.</div><div class=3D""><br = class=3D""><div class=3D""><br class=3D""><div><blockquote type=3D"cite" = class=3D""><div class=3D"">On May 30, 2017, at 5:52 PM, Ryan Heniser = <<a href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><meta = http-equiv=3D"Content-Type" content=3D"text/html charset=3Dutf-8" = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; -webkit-line-break: after-white-space;" class=3D"">Hi Larry,<div = class=3D""><br class=3D""></div><div class=3D"">Thank you for taking the = time to provide such a thorough response. Sorry, you had to repeat = yourself. I just found Thiago=E2=80=99s post about this from 2014 (<a = href=3D"http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/20= 14-August/013430.html" = class=3D"">http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org= /2014-August/013430.html</a>). Yes, the --diff option with --fail 0.05 = and --failpercent 0.1 are working well to address LSB error for my = automated tests. For anyone new to oiiotool, --fail and --failpercent = must go before =E2=80=94diff.</div><div class=3D""><br = class=3D""></div><div class=3D"">Thank you,</div><div = class=3D"">Ryan</div><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""><div class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D"">On 31 May 2017, at 5:07 am, Larry Gritz = <<a href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><meta = http-equiv=3D"Content-Type" content=3D"text/html charset=3Dutf-8" = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; -webkit-line-break: after-white-space;" class=3D"">I should add = that, while -pdiff seemed like a good idea at the time, I'm not sure = it's any better in practice than using regular --diff with wisely chosen = epsilons to allow a certain amount of LSB error to not be counted as = failure. I do that all the time for testsuite image comparisons, where I = expect just a little bit of error against the reference output, for = different platforms or different versions of dependent libraries.<div = class=3D""><br class=3D""></div><div class=3D"">Oh my, this is = embarrassing. I see now that the oiiotool docs do not even explain how = those thresholds can be set. So sorry. The undocumented commands are = --fail, --failpercent, --warn, --warnpercent, --hardfail. It may = be undocumented in the oiiotool section of the OIIO docs, but the same = commands are explained in the "idiff" section of the docs (they work the = same in the two programs). I'll get that oversight fixed.</div><div = class=3D""><br class=3D""></div><div class=3D"">It's also been suggested = that I look into a replacement (or additional) perceptual diff based = on <en-clipboard class=3D""></en-clipboard>CIEDE2000 or DE2000K = (google them for details). Seems like a good idea, but I haven't had = time to do it. I'd certainly be very supportive if anybody felt like it = would be a project they wanted to work on and submit it as a PR.<div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""><div = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On May = 29, 2017, at 10:16 PM, Ryan Heniser <<a = href=3D"mailto:[email protected]" class=3D"">[email protected]</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><meta = http-equiv=3D"Content-Type" content=3D"text/html charset=3Dutf-8" = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; -webkit-line-break: after-white-space;" class=3D""><div = class=3D"">Ugg=E2=80=A6 Sorry, I overlooked</div><div class=3D""><br = class=3D""></div><div class=3D""><div class=3D""><font face=3D"monospace, = monospace" class=3D"">oiiotool --help</font></div><div class=3D""><font = face=3D"monospace, monospace" class=3D""> = ...</font></div><div class=3D""><font face=3D"monospace, monospace" = class=3D""> --pdiff = Print report on the perceptual difference of = two images</font></div></div><div class=3D""><br class=3D""></div><br = class=3D""><div class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On 30 May 2017, at 2:34 pm, Ryan Heniser <<a = href=3D"mailto:[email protected]" class=3D"">[email protected]</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><meta = http-equiv=3D"Content-Type" content=3D"text/html charset=3Dus-ascii" = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; -webkit-line-break: after-white-space;" class=3D""><span = style=3D"font-family: arial, helvetica, sans-serif; font-size: 16px;" = class=3D"">Is there a popular or de facto industry standard tool for = OpenEXR perceptual image differencing? </span><div = style=3D"font-family: arial, helvetica, sans-serif; font-size: 16px;" = class=3D""><div class=3D""><br class=3D""></div><div class=3D"">Kind = regards,</div><div class=3D""><br class=3D""><div class=3D""><span = name=3D"x" class=3D""></span>ryan heniser<br class=3D"">senior shading = technical director | <a href=3D"mailto:[email protected]" = class=3D"">[email protected]</a><br class=3D"">rising sun pictures = | <a href=3D"http://www.rsp.com.au/" = class=3D"">http://www.rsp.com.au</a><br class=3D"">+61 8 8400 = 6400 </div></div></div></div></div></blockquote></div><br = class=3D""></div>_______________________________________________<br = class=3D"">Openexr-devel mailing list<br class=3D""><a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a><br class=3D""><a = href=3D"https://lists.nongnu.org/mailman/listinfo/openexr-devel" = class=3D"">https://lists.nongnu.org/mailman/listinfo/openexr-devel</a><br = class=3D""></div></blockquote></div><br class=3D""><div = apple-content-edited=3D"true" class=3D""> --<br class=3D"">Larry Gritz<br class=3D""><a = href=3D"mailto:[email protected]" class=3D"">[email protected]</a><br = class=3D""><br class=3D""> </div> <br class=3D""></div></div></div></div></blockquote></div><br = class=3D""></div></div></div></blockquote></div><br class=3D""><div = apple-content-edited=3D"true" class=3D""> --<br class=3D"">Larry Gritz<br class=3D""><a = href=3D"mailto:[email protected]" class=3D"">[email protected]</a><br = class=3D""><br class=3D""> </div> <br class=3D""></div></div></body></html>= --Apple-Mail=_19FACB9C-F528-4814-BE9D-96D0BA97AF4C-- --===============8845638310507557636== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Openexr-devel mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/openexr-devel --===============8845638310507557636==--