fix/cuda-f16-activation-scaling

[kungfudaibi]

Overview

for #26044

This PR addresses the NaN issue reported in #26044 on NVIDIA Volta GPUs (sm_70).

On two Tesla V100 32 GB GPUs, I consistently observed that the failure starts when the input reaches 64 tokens. This
coincides with the backend dispatch boundary where ne11 < MMQ_DP4A_MAX_BATCH_SIZE (64) becomes false and the matrix
multiplication switches from MMQ to the cuBLAS path.

Further investigation localized the first non-finite result to the FFN down projection:

ffn_out-2 = MUL_MAT(blk.2.ffn_down.weight, ffn_swiglu-2)

The F32 input activations (ffn_swiglu-2) are still finite before this operation. In the cuBLAS F16 path, these
activations are converted from F32 to F16 before the faster FP16 GEMM is executed. Some activation values are finite
in F32 but exceed the maximum representable F16 value, so the conversion produces Inf. These values then propagate
through the matrix multiplication and subsequent operations as Inf and NaN.

The proposed fix protects this conversion with dynamic power-of-two scaling. It first computes the maximum absolute
activation value (amax). If amax exceeds the safe F16 range, the activations are scaled down before the F32-to-F16
conversion. The same scaling factor is then passed to cuBLAS through the GEMM alpha parameter, restoring the numerical
scale of the output after the FP16 matrix multiplication.

Using a power-of-two scale avoids introducing additional rounding from an arbitrary scaling factor. This restores the
output magnitude, although it does not recover precision already lost in the F16 conversion.

The workaround is restricted to contiguous F32 inputs used by cuBLAS F16 MUL_MAT operations on Volta (sm_70). It is
not tied specifically to Qwen3, because the underlying issue occurs at the CUDA operator and F32-to-F16 conversion
level.

Compared with forcing the affected matrix multiplication to use F32, this approach retains the fast FP16 execution
path. In my benchmarks:

• Forcing F32 reduced throughput by approximately 21.97% at pp64 and 37.65% at pp512.
• Dynamic activation scaling reduced throughput by approximately 2.76% at pp64 and 2.41% at pp512.

With the scaling fix applied, the original importData reproducer produces a finite 4096-dimensional embedding, and
subsequent canary requests in the same server process also remain finite. Compared with the all-F32 path, the
reproduced embedding had:

• Cosine similarity: 0.99999956
• NMSE: 8.91e-7
• Maximum absolute error: 0.0152

The CUDA MUL_MAT backend test set also completed successfully with 1189/1189 tests passing.

Additional information

I found two related reports in the Qwen3-Embedding-8B community:

• Qwen3-Embedding-8B discussion Fails to load 30B model after quantization #27 (https://huggingface.co/Qwen/Qwen3-Embedding-8B/discussions/27)
• Qwen3-Embedding-8B discussion Add LICENSE #21 (https://huggingface.co/Qwen/Qwen3-Embedding-8B/discussions/21)

Discussion #27 reports that inputs containing import produce all-NaN embeddings when the model is executed with
torch.float16, while switching to torch.bfloat16 produces valid results.

Discussion #21 reports a closely related failure involving token 474 (import) at the beginning of the input. A
maintainer of Hugging Face Text Embeddings Inference later explained that the model was originally trained using BF16
and that some hidden-state values can exceed the maximum representable FP16 value. Using BF16 avoids this overflow
because BF16 has approximately the same exponent range as FP32, despite having fewer mantissa bits.

These reports are consistent with my observations in llama.cpp. The input activation to the affected FFN down
projection remains finite in F32, but some values exceed the FP16 maximum of 65504. In the affected llama.cpp CUDA
path, the activation is converted from F32 to F16 before the cuBLAS GEMM call. The out-of-range values become Inf
during this conversion and subsequently propagate through the matrix multiplication as Inf and NaN.

Volta GPUs such as the Tesla V100 (sm_70) do not provide native BF16 Tensor Core execution. Therefore, the fast
reduced-precision cuBLAS path used in this case relies on FP16 rather than BF16. This makes FP16 activation overflow
observable on V100, although the underlying FP16 range limitation is not exclusive to Volta and has also been reported
in other inference stacks when Qwen3-Embedding-8B is executed in FP16.

This evidence supports the use of activation scaling before the F32-to-F16 conversion: it preserves the existing fast
FP16 GEMM path while preventing finite F32 activation values from overflowing to Inf.

Requirements

• I have read and agree with the contributing guidelines

• AI usage disclosure: YES
  I used Codex to assist me throughout this investigation. This included locating the failure at the operator level,
  exploring the initial FP32 workaround, further identifying the numerical overflow during the F32-to-F16 conversion,
  implementing an experimental dynamic-scaling solution, and running the final tests.

  I also used Codex with GPT-5.6 sol to translate and polish text that I had originally written in Chinese. Writing a
  complete technical explanation in English is difficult for me, but the investigation, observations, conclusions, and
  supporting data described here reflect my own understanding of the issue.

  I have manually reviewed the proposed changes and test results, and I take responsibility for the submitted code and
  technical claims. I can explain the implementation and the reasoning behind it. I hope this investigation and the
  proposed fix can help resolve the issue.