weakness in 'double to text' conversion
newbie nullzwei via gnumeric-list <[email protected]> Fri, 10 Dec 2021 16:21:02 +0100
| Newsgroups | gmane.comp.gnome.apps.gnumeric |
|---|---|
| Message-ID | <trinity-048a8fc9-31f5-4199-b855-edb22cbdcb14-1639149662564@3c-app-gmx-bap27> |
--===============7201861808252044138== Content-Type: text/html; charset=UTF-8 <html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div style="font-family: Verdana;font-size: 12.0px;"> <div style="font-family: Verdana;font-size: 12.0px;"> <div style="font-family: Verdana;font-size: 12.0px;"> <div style="font-family: Verdana;font-size: 12.0px;"> <div> <br/> hello @all, <br/> <br/> I tried to use '=value( text( x, "0.0" ) )' and similar to check results of my 'improved calculations' and control whether rounding worked correctly or went wrong. <br/> <br/> That worked fine until i had to compare against 1.45 . the binary 'nearest' representation for 1.45 with IEEE doubles is 'undershot' as 1.449999999999999955591~, and thus the rounding performed by 'text' switches to '1.4' rather than '1.5'. <br/> ( same applies to plenty other values, I assume 50% of doubles for ~5 values have undershot representation ) <br/> <br/> That is! correct from a simple POV, but wrong when accounting that the value is the representant for decimal 1.45 .<br/> <br/> ( even pure 'C' calculates 'printf( " test: %.1f \n", 1.45 ) to '1.4' thus this question might be better suited in other forums ... ??? ) <br/> <br/> I could - and did - construct a workaround by evaluating a second string '=value( text( x, "0.000000000000000" ) )', and if i find a '5' there in the position after the rounding limit and the 'text' result is smaller than the origin add 0.1 to it ... but that's ... cumbersome. <br/> <br/> 'round' works better for this simple case, but fails for some other values ( from a decimal POV ), and that's just my task, find the critical values where round fails, thus round doesn't help. <br/> <br/> Does anyone have a good idea for a 'elegant' way to achieve ... 'decimal correctly rounded results for double values'? It won't be a problem if slow or 'string-math', it's just needed for testing.<br/> <br/> Best Regards, <br/> <br/> <br/> <br/> b.<br/> <br/> P.S. please be tolerant, I'm not a professional but just look, poke and try around, sometimes it results in good new things ... but not always ;-)</div> </div> </div> </div> </div></div></body></html> --===============7201861808252044138== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ gnumeric-list mailing list [email protected] https://mail.gnome.org/mailman/listinfo/gnumeric-list --===============7201861808252044138==--