deco-math_step_00_b_sa_code_exact_rounddown_roundup_in_gnumeric | was: Aw: Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).

hello Morten,

acc. to a proposal of John Denker to work around wrong floor() results for 'zombie values' by

floor(x) < x ? floor(x) : floor(x) - 1;

i'd try to put it into goffice/goffice/math/go-math.c, and it seems to work.

would you mind to check the code:

---------------------------

double

go_fake_trunc (double x)

{

x = go_d2d (x);

if (x == floor (x))

return x;

// edit b. wip 2021-07-09: trying to avoid wrong results for rounddown

return (x >= 0)

// ori was: ? floor (go_add_epsilon (x))

? floor ( x ) <= x ? floor( x ) : floor( x ) - 1

// ori was: : -floor (go_add_epsilon (-x));

: -floor ( -x ) >= -x ? -floor( -x ) : -floor ( -x ) - 1;

}

---------------------------

if this (or a variant, i'm not! an experienced coder and open for all improving suggestions) is ok i'd like to help to implement similar for the other affected functions.

attached a sheet with the results patched / standard for a value i tested with.

best regards,

b.

---

Gesendet: Donnerstag, 08. Juli 2021 um 00:16 Uhr

Von: "Morten Welinder" <[email protected]>

An: "Steven D'Aprano" <[email protected]>

Cc: "Gnumeric Mailing List" <[email protected]>

Betreff: Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).

Gnumeric's rounding here is indeed done with

round_to_int(x * 10^d) / 10^d

except that round_to_int deliberately misrounds 0.5-1ulp to 1. That's

not important here.

This means that, in general, you should not expect last-bit accuracy

for this operation. Both the multiplication and the division can

introduce rounding errors. I am willing to entertain patches that fix

that if (a) someone is sufficiently interested to do the work, and (b)

can do it in a sane way that doesn't involve making strings out of

numbers. This is not a beginner's task.

But note that the function implemented would be x ->

round_to_base2(round_to_base10(x,d),53). In other words, you are

inherently double rounding. Allow me to point out that while

well-defined, it is not a very sane thing to do, except in the cases

where the outer round is known to be a no-op.

There are two types of rounding in Gnumeric: the rounding functions

such as ROUND. These change the values. Then there is the rounding

implied by attaching a format to a cell. That does not change the

value, only its display: even if you format A1 to show two decimals,

then another cell containing =A1*42 will still see the full precision

of A1.

Morten

_______________________________________________

gnumeric-list mailing list

[email protected]

https://mail.gnome.org/mailman/listinfo/gnumeric-list

_______________________________________________
gnumeric-list mailing list
[email protected]
https://mail.gnome.org/mailman/listinfo/gnumeric-list