Re: long double losing mantissa bits

Rhialto <[email protected]>
Newsgroups gmane.os.netbsd.ports.x86-64,gmane.os.netbsd.devel.userlevel
Message-ID <YAh/[email protected]>
On Tue 19 Jan 2021 at 22:16:32 +0000, Taylor R Campbell wrote:
> It looks like you have an off-by-one error in your exponent handling.
> If you want to scale the fractional part into an integer, you need to
> multiply by 2^57, not by 2^56.

Thank you! Your analysis made me realise that I was miscounting the
bits, and assumed one too many. So several of the test values were a bit
too long and got rounded which confused me.

So I did change the multiplication to 2^57.  But the 2^56 was correct
for the final format, so I just used it to get the lsb for rounding
later, and then shifted it a bit right again.

Final result is at
https://gitlab.com/Rhialto/macro11/-/blob/master/parse.c#L386.

-Olaf.
-- 
Olaf 'Rhialto' Seibert -- rhialto at falu dot nl
___  Anyone who is capable of getting themselves made President should on
\X/  no account be allowed to do the job.       --Douglas Adams, "THGTTG"
signature.asc (application/pgp-signature, 488 B)
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEFVAhiiWjqgwBVdQAmYnGRWHD+9MFAmAIf0kACgkQmYnGRWHD
+9Pgngf+JTTSb+d38L6PvA342DbURlQ2TlZGuR6KJ9RGnjqORjCUdJ9xoZIcYpLi
R31h+FWls4V/6ArW2Q1pY0OBoWG1f30wAN+LCl9pDP33A6CLiGUAHRVTCNLo2epQ
PrbE8s55WOAFRB6ki/55ebuU+MGUFt9RQ+7SDzXkDnYsdlbbM7fm/L/8Ngc3DXwI
/9kFtzyKcyqp0VG4rLP/LabkAXX/JGkFebSPwWiTVbDWEqsi489030UjpEaW+Y3R
tNf97wGkARABEE9BLPgg7yGBDTWVS94v6knAYSuU18suQtZ3u5UmKZZJyWox+7EX
SnccXpFsZiJ/wSzBs+Ft1oAdcStcMA==
=vMW7
-----END PGP SIGNATURE-----
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.