[mmalpe-nv]

Overview

This PR includes the changes for updating the Q8_1 correction sum to use quantized input values which are consistent with the integer dot product and CPU behavior. It addresses the reproduced intermittent Q4_1 MUL_MAT failure on Tesla T4.

Additional information

Failed testcase

On Tesla T4:

MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=1,k=32,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0)

Command run

./build-ci-release/bin/test-backend-ops -j 1 -b CUDA0 \
  -o 'MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=1,k=32,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0)'

Captured failure -

[MUL_MAT] ERR = 0.000536123 > 0.000500000
0/1 tests passed
Backend CUDA0: FAIL

Calculation

A Q4_1 weight is represented as:

A[j] = d_A × q_A[j] + m_A

The F32 input B is internally quantized:

d_B    = max(abs(B)) / 127
q_B[j] = round(B[j] / d_B)
B[j]   ≈ d_B × q_B[j]

Here, d_A and d_B are scales, q_A and q_B are integer codes, and m_A is the weight offset. Zero-input blocks are handled explicitly by the code.

Ignoring additional floating point rounding, the dot product becomes:

output ≈ d_A × d_B × sum(q_A[j] × q_B[j])
       + m_A × correction_sum

The main contribution already uses quantized B. The correction sum differed between CPU and CUDA:

Path	Stored correction sum
Original CUDA	half(sum(original B))
CPU and proposed CUDA	half(d_B × sum(q_B))

Using original inputs in the correction and quantized inputs in the main contribution can disrupt cancellation between the two terms.

Observed Difference

For the captured failing input:

Original CUDA correction = −0.640625
CPU correction           = −0.59765625
Correction difference    = −0.04296875

For the first output row:

Weight offset m_A = −1.005859375

Predicted shift from correction:
m_A × correction difference ≈ +0.04322052

Observed GPU − CPU difference ≈ +0.04405594

The correction mismatch explains the dominant output shift. Additional arithmetic rounding explains the remaining difference.

This was a numerical accuracy discrepancy, not overflow.

Why the Test Failed

The test measures normalized squared disagreement across the outputs:

NMSE = sum((GPU[i] − CPU[i])²) / sum(GPU[i]²)

The tolerance is 5e-4 = 0.0005. It limits output disagreement, not individual output magnitudes.

Before the change:

Sum of squared GPU–CPU differences = 0.0269154607
Sum of squared GPU outputs        = 50.2039178

NMSE ≈ 0.0269154607 / 50.2039178
     ≈ 0.000536123
     > 0.0005 → FAIL

After the change, for the same saved inputs:

NMSE ≈ 0.0000000607016
     < 0.0005 → PASS

Computing the correction from the quantized input codes removes the dominant shift for this sample. Small rounding differences remain.

The tolerance is unchanged; the fix reduces the output disagreement.

Validation

Validation below was performed on Tesla T4 at the investigated revision:

4d756bc72bf00a4aacf410ae15a2d315f3db400d

Metric	Original CUDA	Changed CUDA
CPU–CUDA NMSE	5.36123e-4	6.07016e-8
RMSE against independent calculation	0.0411916	0.0054600
Five identical-input replays	All failed	All passed

The independent calculation uses the decoded Q4_1 weights and original F32 inputs with double precision accumulation. The change reduced RMSE by approximately 7.5× for this sample.

Additional validation on that revision passed:

• 100 fresh-input cases at k=32.
• 100 fresh-input cases at k=256.
• 1891/1891 broader MUL_MAT cases.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[ggml-gh-bot]

Hi @mmalpe-nv, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[ORippler]

See

https://github.com/ggml-org/llama.cpp/actions/runs/37794678453/job/113382326226?pr=28713

for an instance of sporadic CI failure caused by the old behavior. Imo it's not worth it to add a dedicated regression test for this, but I'll defer this to @ggml-org/ci

[gaugarg-nv]

Looks like quantize_mmq_q8_1 has the same issue:

llama.cpp/ggml/src/ggml-cuda/quantize.cu

Line 505 in 889f2ec

sum = xi.x + xi.y + xi.z + xi.w;

Can you fix that as well?

[fff7d1bc]

Hey @mmalpe-nv and @gaugarg-nv since you approved and merged it.

I've been running down new issue I hit after upgrade to latest HEAD of llama.cpp and though binary search arrived at this merge commit. I am able to reproduce this fault each time, below is more details, used llm to make it easier to make sense

I’m seeing a repeatable thinking-off tool-replay regression on ROCm after 8fc0376721e9662d498a5706d7ee8e7574ce99f4.

My setup is a Framework Desktop with Ryzen AI Max+ 395, gfx1151, 128 GB RAM and ROCm 10.1. The model is Unsloth Qwen3.8 Flash-Next UD-Q4_K_XL, using F16 KV cache and a 262144-token shared context pool with four slots.

These tests use a downstream patched build. I have not confirmed reproduction on pristine upstream.

The failing case replays a completed tool call with thinking disabled. The expected response is exactly:

ALDER_937261

The newer build instead returns:

ALDER_937261
</think>

ALDER_937261

Results from five seeds and one additional temperature-zero check:

Build	Correct responses
Parent 740251a920abf92b7744d9c0518ca157db0db579	6/6
Newer head 28b812d6cb9cfe569d368c2774c3886ad4fe10e8	0/6
Same newer build, reverting only this commit’s quantizer calculation	6/6

For the revert comparison, only the physical libggml-hip library changed. Other native binaries and libraries remained byte-identical. Rendered prompts and sampling settings matched between arms.

The six-case comparison used greedy DFlash2 at depth three. The initial short checks also showed the symptom without DFlash2.

For reproduction, use the same model and Jinja template on both revisions, preserve reasoning history, and replay the completed tool turn below. Ensure the template receives enable_thinking=false. The model alias in this example is flash-next.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'JSON'
{
  "model": "flash-next",
  "reasoning_effort": "none",
  "chat_template_kwargs": {"enable_thinking": false},
  "temperature": 0.7,
  "top_p": 0.8,
  "top_k": 20,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repeat_penalty": 1.0,
  "seed": 42,
  "cache_prompt": false,
  "max_tokens": 4096,
  "stream": false,
  "tool_choice": "none",
  "messages": [
    {
      "role": "user",
      "content": "Call lookup_record exactly once with query.key ALDER. After its result reply exactly the value returned, with no explanation."
    },
    {
      "role": "assistant",
      "content": "",
      "tool_calls": [{
        "id": "lookup-1",
        "type": "function",
        "function": {
          "name": "lookup_record",
          "arguments": "{\"query\":{\"key\":\"ALDER\"}}"
        }
      }]
    },
    {
      "role": "tool",
      "tool_call_id": "lookup-1",
      "content": "ALDER_937261"
    }
  ],
  "tools": [{
    "type": "function",
    "function": {
      "name": "lookup_record",
      "description": "Read the exact stored value for a record key.",
      "parameters": {
        "type": "object",
        "additionalProperties": false,
        "required": ["query"],
        "properties": {
          "query": {
            "type": "object",
            "additionalProperties": false,
            "required": ["key"],
            "properties": {"key": {"type": "string"}}
          }
        }
      }
    }
  }]
}
JSON

Repeat with seeds 42, 1234, 2026, 7, 99, then seed 42 with temperature 0.

This is a simplified direct-server replay of the tested fixture, not a separately validated pristine-upstream reproducer. The result isolates a behavior change associated with the correction calculation, but does not establish that the numerical correction itself is wrong.