Re: follow up II - newbie questions, sum over range, rounding of special problematic values
Steven D'Aprano <[email protected]> Mon, 24 May 2021 10:07:24 +1000
| Newsgroups | gmane.comp.gnome.apps.gnumeric |
|---|---|
| Message-ID | <[email protected]> |
On Sun, May 23, 2021 at 12:42:29PM -0600, Andreas J. Guelzow wrote: > Hi, > > On 2021-05-22 7:26 p.m., Steven D'Aprano wrote: > > > >input 0.24999999999999997 > > You are playing at the edge of the maximum mantissa length. Did you ever > try using that number in LibreOffice or Excel? I'm not discussing LibreOffice bugs on this gnumeric mailing list, we'd be here for years :-( As far as I can tell, LibreOffice doesn't even allow you to create the number 0.24999999999999997 but immediately rounds it to 0.25. Do you know what is the internal format for floating point numbers in gnumeric? I assume it is 64-bit IEEE-754 floats. Do you know the algorithm used for round-down and round-up? -- Steve