[danielhanchen]

Update (2026-10-21): master merged in, #303 at 34a66f4

#303 is now pinned at 34a66f4. e886a61 was refused after the Inkling pin (ggml-org#25731 at e1b89cf) because both edited the same lines in fattn-mma-f16.cuh and test-backend-ops.cpp; 34a66f4 passes the K/V type through the K pointer type instead, so those lines are upstream's again (quantized kernels byte-identical SASS on sm_89 and sm_100). Master is merged in (#312), so the set is master's 23 pins plus #303:

ok   b11786 (aged): 24 pins; ggml-org#25731 additive; unslothai#70 additive; unslothai#298 additive; unslothai#314 additive
ok   b11796 (newest): the same, all 24 pins intact, merge_checks clean

On the composed b11785 tree with ggml-org#25731 at e1b89cf (B200): FLASH_ATTN_EXT 5597/5597, FLASH_ATTN_EXT_BANDED 13/13. The cuda-fa-quant-small-batch feature check passes on a build where the fast path is present; it does not by itself prove the path engaged (that is shown by the nsys kernel census in #303). The sections below describe the original pin and its sims at the time.

Overview

Pins #303 at 34a66f4. With a q4_0 / q8_0 KV cache and 3 to 16 tokens per ubatch (multi-sequence decode, e.g. -np 4 --kv-unified), the CUDA flash-attention dispatcher picks the MMA f16 kernel, which first converts the whole K and V views to f16 for every layer on every step. #303 makes the MMA kernel dequantize q4_0 / q8_0 K/V tiles as it loads them for those batch sizes, so the whole-cache conversion is gone (+8% to +20% tok/s at 4 streams on an L4, numbers in #303). Ampere or newer, same-type K/V, head dims 128 / 256 / 512; single-stream decode, prefill, f16 KV, Turing, Volta and AMD keep the old path.

Not a fix for an upstream break: a new pin. #303's branch is upstream b11642 (base/upstream-2639a3b4e) plus its own two commits, so its GitHub diff is exactly what gets pinned.

The prebuilt is upstream's aged b#### tag plus the pins in scripts/unsloth/pr-set.json, merged in order (git diff3, then additive_merge.py for pure additions only). #303 is appended last. It is independent of #269 (open in #211) and kept out of that PR to keep #211's scope; the sims below cover both landing orders' end state (master + #303, and master + #269 + #303).

scripts/unsloth/pr-set.json          +1  (pin 34a66f42b)
scripts/unsloth/feature-checks.json  +6  (cuda-fa-quant-small-batch: backend-op FLASH_ATTN_EXT, params n_seq=4)

The feature check selects the multi-sequence cases #303 adds to test-backend-ops (4 sequences in one unified cache, f16 / q8_0 / q4_0, 144 cases). They exist only with #303 in the tree, so the check fails with a zero case count if the pin is lost. Like every backend-op check it is deferred on the CPU nightly and runs only under feature_matrix.py --gpu.

Checks

• llama_pinset.py audit with this pr-set: CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303 ok, base base/upstream-2639a3b4e [upstream], +668/-48 in 30 files, GitHub diff == the pin's own change; feature-checks.json coverage ok. The other 6 FIX rows (kimi-k3 : the MoonViT-3d vision tower and full-size loading fixes #70, sampling: index penalties by token id instead of scanning every candidate #95, ggml-cuda: use cudaMemcpyDefault in the ggml_cuda_cpy 2D fast path #157, CUDA: faster MoE prompt processing (compact MMQ tiling, device expert sort, GET_ROWS) #264, llama-mmap: populate the non-lazy ranges when MAP_POPULATE is skipped for lazy tensors #266, model : DeepSeek-V4.1-Flash-1017 full stack pin (all DeepSeek-V4.1 PRs except #292) #298) are the same rows master's own pr-set shows.
• llama_pinset.py sim --tags b11756,b11770 (b11756 = the base a run picks now, b11770 = newest), compared against master's pr-set on the same tags:

                         b11756                                b11770
master                   6 REFUSED, 9 not intact, mc clean     6 REFUSED, 9 not intact, mc clean
master + #303            same 6 REFUSED, 9 not intact, #303 clean    same 6 REFUSED, 9 not intact, #303 clean
master + #269 + #303     same 6, #269 additive, #303 clean     same 6, #269 REFUSED, #303 clean
pin_contract #303        ok: 27 files with new symbols, 426 lines accounted for (both tags, both sets)

#303 adds no refusal and does not raise the not-intact count on either tag, with or without #269. The 6 refusals (ggml-org#25731, #61, #158, #149, #176, #298) are master's, not this pin's. On b11770 #269 is refused in src/llama-context.cpp, tools/server/server-queue.cpp and tools/server/tests/utils.py; #269 merges before #303, so that is #269 against upstream and is for #211, not this PR.

• CUDA on the merged b11756 tree with server: preemption notices, asynchronous parks and exact concurrency (pin branch on b11577) #269 and CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303 both in (tree 7a68ba8cf, sm_100, local B200):

ok   test-backend-ops build (CUDA)
ok   test-backend-ops -o FLASH_ATTN_EXT -b CUDA0: 5600/5600 passed, 0 FAIL
ok   test-backend-ops -o FLASH_ATTN_EXT -p n_seq=4 (the feature check): 144/144 passed
ok   nsys census, hsk=128 nb=4 n_seq=4: flash_attn_ext_f16<..., (ggml_type)8> for q8_0 and <..., (ggml_type)2> for q4_0, no dequantize/convert kernels

So the merged tree still runs #303's in-kernel dequant path, and #269's FA changes (paged FLASH_ATTN_EXT refused in supports_op) do not take these shapes off it.

Not run

• Full llama_pinset.py verify --cuda (full CUDA build, test-chat, MUL_MAT / MUL_MAT_ID / PAD / GET_ROWS, feature_matrix --gpu): not run, only test-backend-ops was built. b11770 was not built for CUDA.
• GPUs other than B200 on the merged tree; CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303's own L4 and B200 results are in CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303. T4 (sm_75) keeps the old path and was not run here.
• Throughput on the merged tree (numbers are CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303's, measured on its own branch).
• CUDA: dequantize q4_0/q8_0 K/V tiles in the FA MMA kernel for small multi-sequence batches (pin branch on b11642) #303 together with server: preemption notices, asynchronous parks and exact concurrency (pin branch on b11577) #269's batch-invariant mode (LLAMA_EXACT_CONCURRENCY) at runtime: read in the code only.
• ROCm, Vulkan, Metal, Windows and ARM legs run only in the nightly.
• The CPU preflight (unsloth-pin-preflight.yml) has not run on this branch yet.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-21T02:31:18.144696Z	7dd9b58	PR opened
🔒 Security Review	✅ Completed 2026-10-21T02:31:35.082720Z	7dd9b58	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7dd9b58e33

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Assert the optimized CUDA path in the feature check

In the GPU feature-matrix run, this probe only checks that FLASH_ATTN_EXT produces correct results for some n_seq=4 cases. If dispatch regresses to the previous whole-cache f16 conversion path, the outputs remain correct and all matched cases still pass, even though the performance feature carried by this pin is no longer active. Add evidence that the q4_0/q8_0 MMA kernel was selected and that conversion kernels were absent, rather than treating numerical equivalence as proof of this optimization.

Useful? React with 👍 / 👎.