Re: long double losing mantissa bits
Rhialto <[email protected]>
| Newsgroups | gmane.os.netbsd.ports.x86-64,gmane.os.netbsd.devel.userlevel |
|---|---|
| Message-ID | <YAh/[email protected]> |
On Tue 19 Jan 2021 at 22:16:32 +0000, Taylor R Campbell wrote: > It looks like you have an off-by-one error in your exponent handling. > If you want to scale the fractional part into an integer, you need to > multiply by 2^57, not by 2^56. Thank you! Your analysis made me realise that I was miscounting the bits, and assumed one too many. So several of the test values were a bit too long and got rounded which confused me. So I did change the multiplication to 2^57. But the 2^56 was correct for the final format, so I just used it to get the lsb for rounding later, and then shifted it a bit right again. Final result is at https://gitlab.com/Rhialto/macro11/-/blob/master/parse.c#L386. -Olaf. -- Olaf 'Rhialto' Seibert -- rhialto at falu dot nl ___ Anyone who is capable of getting themselves made President should on \X/ no account be allowed to do the job. --Douglas Adams, "THGTTG"
signature.asc
(application/pgp-signature, 488 B)
-----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEFVAhiiWjqgwBVdQAmYnGRWHD+9MFAmAIf0kACgkQmYnGRWHD +9Pgngf+JTTSb+d38L6PvA342DbURlQ2TlZGuR6KJ9RGnjqORjCUdJ9xoZIcYpLi R31h+FWls4V/6ArW2Q1pY0OBoWG1f30wAN+LCl9pDP33A6CLiGUAHRVTCNLo2epQ PrbE8s55WOAFRB6ki/55ebuU+MGUFt9RQ+7SDzXkDnYsdlbbM7fm/L/8Ngc3DXwI /9kFtzyKcyqp0VG4rLP/LabkAXX/JGkFebSPwWiTVbDWEqsi489030UjpEaW+Y3R tNf97wGkARABEE9BLPgg7yGBDTWVS94v6knAYSuU18suQtZ3u5UmKZZJyWox+7EX SnccXpFsZiJ/wSzBs+Ft1oAdcStcMA== =vMW7 -----END PGP SIGNATURE-----