[LeoBorcherding]

Audit finding IK-2 (P2, confirmed on an L4). Inkling's banded attention on NVIDIA.

Base of this PR. The banded op only exists in the Inkling pin (ggml-org/llama.cpp#25731, pinned in scripts/unsloth/pr-set.json at 534c32182, which is still that PR's head and still the pin on unslothai/llama.cpp master 8dd697cf0), not in the fork's master tree. So this PR targets night-shift-tester/audit-pin-inkling-534c321 (that pin commit) rather than night-shift-tester/audit-base, which keeps the diff to the fix. Landing it means a commit on the Inkling PR branch and a repin, or a fix pin listed after it.

What's wrong

Inkling sets GGML_PREC_F32 on every FLASH_ATTN_EXT_BANDED node (src/models/inkling.cpp:466-468 at 534c32182):

            cur = ggml_flash_attn_ext_banded(ctx0, q_fa, k_fa, v_fa, mask, rel_fa,
                    1.0f/float(head_dim), rel_extent);
            ggml_prec_set_acc(cur, GGML_PREC_F32);

ggml_cuda_flash_attn_ext_banded_use_mma (ggml/src/ggml-cuda/fattn.cu:541) never reads the precision, so with an F16/BF16 KV cache (and F32 caches, which Inkling casts to F16 first) on any Turing or newer card the op goes to BEST_FATTN_KERNEL_MMA_F16 (fattn.cu:607), whose V*softmax accumulator is half2 on NVIDIA (fattn-mma-f16.cuh:1138-1172). The FP32 banded kernel in fattn-banded.cu only ever runs on AMD and pre-Turing. With a nonzero-mean V, once the running sum is large each increment is under half an fp16 ulp and gets dropped while the fp32 row sum keeps growing, so the output is biased toward zero:

$$o = \frac{\sum_j p_j v_j}{\sum_j p_j}, \qquad \delta \text{ is lost when } \delta < \tfrac12,\mathrm{ulp}_{16}\Big(\sum p_j v_j\Big)$$

Fix

ggml_cuda_flash_attn_ext_banded_use_mma returns false when the node asks for F32 or better (prec != GGML_PREC_UNDEFINED && prec <= GGML_PREC_F32, the "at least F32" test ggml.h documents), so the FP32 banded kernel runs. Both call sites (fattn.cu kernel pick and fattn-banded.cu:219) go through this one function, so they stay in agreement. The other option in the audit, an fp32 T_C_VKQ for the banded MMA instance, keeps the fast path but changes the shared MMA tile configs; this is the change that honors the documented ggml_prec_set_acc contract without touching the kernel.

Tradeoff: Inkling always requests F32, so on NVIDIA the banded MMA path is now unused and long-context prefill runs on the simpler FP32 kernel. Speed isn't measured (test-backend-ops perf has no case for this op). If that cost turns out too high, the follow-up is the fp32 T_C_VKQ instance, not dropping the precision request.

Test

test-backend-ops gets test_flash_attn_ext_banded_v_offset: the existing banded case with V drawn from U(900, 1100), at d=128, n_head=8, n_head_kv=1, n_q=512, n_kv=131072, causal, F16 K/V, held to the F32 tolerance (NMSE 2e-6). The existing cases use V in [-0.25, 0.25], where the accumulator stays near zero, and allow 1e-3 for fp16 accumulation.

Testing

Colab L4 (NVIDIA L4, cc 8.9), driver 580.82.07, CUDA 13.0 (nvcc V13.0.88), gcc 13.3.0, cmake 3.31.10, Ubuntu 24.04.4.

• base: 534c32182 (the pin) with only this PR's tests/test-backend-ops.cpp hunk applied, so the new case exists on base
• fix: b58b65dc2 (this branch)

Same command on both:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 8 --target test-backend-ops
./build/bin/test-backend-ops -b CUDA0 -o FLASH_ATTN_EXT_BANDED

	new v=U(900,1100) case	existing 13 banded cases
base 534c32182	FAIL, ERR = 1.130658111 > 0.000002000	pass
fix b58b65dc2	OK	pass (14/14)

Base output:

[FLASH_ATTN_EXT_BANDED] ERR = 1.130658111 > 0.000002000   FLASH_ATTN_EXT_BANDED(d=128,n_head=8,n_head_kv=1,n_q=512,n_kv=131072,rel_extent=1024,mask_kind=1,kv_type=f16,rel_type=f32,strided=0,v=U(900,1100)): FAIL
  13/14 tests passed
  Backend CUDA0: FAIL

Fix output:

  FLASH_ATTN_EXT_BANDED(d=128,n_head=8,n_head_kv=1,n_q=512,n_kv=131072,rel_extent=1024,mask_kind=1,kv_type=f16,rel_type=f32,strided=0,v=U(900,1100)): OK
  14/14 tests passed
  Backend CUDA0: OK

The real-hardware error on base (NMSE 1.13) is far larger than the CPU emulation in the audit predicted (4e-4 to 0.46), so the fp16 accumulator loss is worse in practice than first sized.

Not tested

• Speed of the FP32 banded kernel at long context against the MMA path: no perf case exists for this op.
• T4 (sm_75) itself; L4 (sm_89) takes the same MMA branch. Hopper/Blackwell not run.
• No real Inkling model was run end to end.

[LeoBorcherding]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-19T08:27:01.691983Z	—	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Hooray!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".