Re: long double losing mantissa bits
Taylor R Campbell <[email protected]>
| Newsgroups | gmane.os.netbsd.devel.userlevel,gmane.os.netbsd.ports.x86-64 |
|---|---|
| Message-ID | <[email protected]> |
> Date: Tue, 19 Jan 2021 22:16:32 +0000 > From: Taylor R Campbell <[email protected]> > > This is not an integer, so conversion to uint64_t rounds it to > nearest, with ties to even, so you get ufrac = 0x80000000000000 = > 36028797018963968 as shown in the result. Oops -- one more correction: conversion to uint64_t rounds toward zero, not to nearest. The result happens to coincide in this case. > It looks like you have an off-by-one error in your exponent handling. > If you want to scale the fractional part into an integer, you need to > multiply by 2^57, not by 2^56. Another possibility is that you expected round-to-odd or something instead of toward-zero or to-nearest -- or perhaps the code works as intended but the test cases are buggy.