Aren
/
NetBSD
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

tests/lib/libc/stdio/t_printf: Fix another rounding error. 

Noted by kre.

This doesn't break a passing test or fix a failed test, at least on
x86 -- our printf produces `0x1.533p+3' for the double case and
`0xa.99ap+0' for the long double case.  But of the hexadecimal number
literals that that start with 0x5 having three hexadigits to the
right of the fractional point, 0x5.4cdp+1 closest to the IEEE 754
binary64, VAX D, x86 extended precision, and IEEE 754 binary128
floating-point numbers closest to 10.6.

The reason is that the number 10.6 (or the nearest floating-point
number in any format with enough precision) is:

101.0100 1100 1100|1100... * 2^1 = 0x5.4cc|c...p+1

If we round at the vertical bar to the _nearest_ output with three
hexadigits of precision, the result is:

101.0100 1100 1101 * 2^1 = 0x5.4cdp+1
This commit is contained in:
riastradh 2024-05-09 22:38:29 +00:00
parent 598b1af8b9
commit 27df239d91
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
tests/lib/libc/stdio/t_printf.c
View File

@ -1,4 +1,4 @@
/* $NetBSD: t_printf.c,v 1.15 2024/05/09 12:24:24 riastradh Exp $ */
/* $NetBSD: t_printf.c,v 1.16 2024/05/09 22:38:29 riastradh Exp $ */
/*-
* Copyright (c) 2010 The NetBSD Foundation, Inc.
@ -195,7 +195,7 @@ ATF_TC_BODY(snprintf_double_a, tc)
snprintf(buf, sizeof buf, "%.3a", (double)10.6);
ATF_CHECK_MSG((strcmp(buf, "0x1.533p+3") == 0 ||
strcmp(buf, "0x2.a66p+2") == 0 ||
strcmp(buf, "0x5.4ccp+1") == 0 ||
strcmp(buf, "0x5.4cdp+1") == 0 ||
strcmp(buf, "0xa.99ap+0") == 0),
"buf=%s", buf);
@ -220,7 +220,7 @@ ATF_TC_BODY(snprintf_long_double_a, tc)
snprintf(buf, sizeof buf, "%.3La", 10.6L);
ATF_CHECK_MSG((strcmp(buf, "0x1.533p+3") == 0 ||
strcmp(buf, "0x2.a66p+2") == 0 ||
strcmp(buf, "0x5.4ccp+1") == 0 ||
strcmp(buf, "0x5.4cdp+1") == 0 ||
strcmp(buf, "0xa.99ap+0") == 0),
"buf=%s", buf);