NaN inputs whose payload lies below the target precision are rounded to Infinity #13

This is bug 2 of 3 identified by comparing floating point number conversion results across a few open-source implementations. It appears that NAN inputs are converting to INF in some cases. Feel free to cancel if this is expected/intended.

Below is the full AI-generated summary:

A NaN input must give a NaN. CPFloat rounds a NaN's bit pattern as if it were a finite value, so when every set payload bit lies below the target precision, the payload is rounded away and the result is the Infinity pattern:

#include <math.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include "cpfloat_binary64.h"

static double round_to(const char *format, cpfloat_rounding_t mode, double x) {
  optstruct *o = init_optstruct();
  strcpy(o->format, format);
  cpfloat_populate_optstruct_from_format(o);
  o->round = mode;
  double y;
  cpfloat(&y, &x, 1, o);
  free_optstruct(o);
  return y;
}

int main(void) {
  uint64_t bits = 0x7FF0000000000001;  /* a signalling NaN */
  double snan;
  memcpy(&snan, &bits, sizeof snan);
  printf("%g\n", round_to("binary16", CPFLOAT_RND_NE, snan)); /* inf, want nan */
  printf("%g\n", round_to("binary16", CPFLOAT_RND_TZ, snan)); /* inf, want nan */
  printf("%g\n", round_to("E4M3", CPFLOAT_RND_NE, snan));     /* inf, want nan */
  printf("%g\n", round_to("binary16", CPFLOAT_RND_NE, NAN));  /* nan (correct) */
  return 0;
}

cc -std=c11 -Icpfloat/src t.c -lm && ./a.out at 3583976 prints inf inf inf nan.

It happens under RND_NE, RND_TN and RND_TZ. RND_TP keeps these NaNs only because its rounding increment happens to carry into the kept bits. The quiet NaN survives because its quiet bit is the top fraction bit, which every format keeps, except at precision 1 (a custom format), where even 0x7FF8000000000000 becomes Infinity. With a custom exponent range, some NaNs come back as ±0 instead.

Cause: the scalar rounding macros (e.g. RN_TIES_TO_EVEN_SCALAR_OTHER_EXP in src/cpfloat_template.h) check underflow and overflow with < / >= comparisons, which are false for a NaN, so the NaN falls through to the bit-masking step (INTOF(y) + ...) & leadmask, which clears its low payload bits. Passing NaNs through before rounding (e.g. if (isnan(*(y))) *(x) = *(y); else ...), or forcing a quiet NaN, fixes it. Even if signalling NaNs are out of scope, quieting one would be fine; turning it into Infinity is not.

Use the following links to see the expected conversion results:

• sNaN 0x7FF0000000000001 (binary64) → binary16
• sNaN 0x7FF0000000000001 (binary64) → E4M3