cuda: native PTQ1_0 MMQ tile loader (closes the dequantize-to-cuBLAS prefill fallback)

[bri-prism]

Native MMQ support for PTQ1_0 on CUDA. Before this, PTQ1_0 had no entry in the quantized-matmul path, so large-batch matmul fell back to a full dequantize to fp16 followed by cuBLAS, measured at 2.26x slower than PQ2_0 at n=512 even though PTQ1_0 already led on decode.

What

Adds the PTQ1_0 MMQ tile loader, the type dispatch, and the DECL_MMQ_CASE instance so PTQ1_0 takes the same quantized GEMM path as the other low-bit types, plus a packed coalesced dequantize for the remaining conversion sites.

Provenance

Ported from an internal handoff branch whose gates recorded 45/45 PTQ1_0 MUL_MAT on CUDA and an H100 perf pass, but that was against a pre-#152 base. Its Metal and FWHT parts are superseded by the already-landed Metal PTQ1_0 and FWHT work and are excluded; this is the CUDA MMQ half only, rebased onto current prism.

Verification (this PR, on an L40S, sm_89)

• Build: clean, 413 targets, zero errors.
• test-backend-ops test -o MUL_MAT -b CUDA0: PTQ1_0 45 OK, 0 FAIL. (33 further cases report not-supported; those are the type_b=f16 shapes, identical on the Metal backend, unrelated to this change.)
• test-backend-ops test -o MUL_MAT_ID -b CUDA0: PTQ1_0 75 OK, 0 FAIL.

Correctness of the PTQ1_0 CUDA matmul and per-expert matmul is verified on current prism.

What is not in this PR

A fresh end-to-end prefill A/B was not re-measured on this run. The win is the removal of the dequantize-to-cuBLAS fallback by giving PTQ1_0 a real MMQ path; the prior handoff measured that on H100. A maintainer can take the current-tree prefill number, or I can on request. Decode is unchanged (this is the prefill/large-batch path).

[Copilot]

🟡 Changes recommended

PTQ1_0 is currently prevented from using MMQ for batch sizes > 64, which can reintroduce the cuBLAS fallback for large-batch prefill and undermines the PR’s stated goal.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds native CUDA MMQ (quantized matmul) support for GGML_TYPE_PTQ1_0, aiming to avoid the dequantize-to-fp16 + cuBLAS fallback for large-batch matmul by providing a PTQ1_0 MMQ tile loader, dispatch wiring, and a dedicated PTQ1_0 dequantize kernel for remaining conversion paths.

Changes:

• Add PTQ1_0 MMQ tile loading + type dispatch/instantiation so PTQ1_0 can use the quantized GEMM path.
• Add PTQ1_0 MMQ configs for Ampere and update related CUDA selection heuristics (MMVQ/MMQ).
• Add a CUDA-only, coalesced PTQ1_0 dequantize kernel and wire it into the to_{bf16,fp16,fp32} conversion dispatch.

File summaries

File	Description
ggml/src/ggml-cuda/vecdotq.cuh	Adds PTQ1_0 MMQ VDR define used by MMQ util selection.
ggml/src/ggml-cuda/template-instances/mmq-instance-ptq1_0.cu	Adds the PTQ1_0 MMQ template instantiation (CUDA-only).
ggml/src/ggml-cuda/template-instances/generate_cu_files.py	Generates the PTQ1_0 MMQ instance with CUDA-only guards.
ggml/src/ggml-cuda/mmvq.cu	Adds a PTQ1_0-specific MMVQ threshold on NVIDIA Turing+.
ggml/src/ggml-cuda/mmq.cuh	Wires PTQ1_0 into MMQ ds-layout selection, tile sizing, util dispatch, and extern declarations.
ggml/src/ggml-cuda/mmq.cu	Adds PTQ1_0 MMQ type dispatch and a PTQ1_0 MMQ selection rule.
ggml/src/ggml-cuda/mmq-load-tiles.cuh	Implements the PTQ1_0 MMQ tile loader and packed decode path (CUDA-only).
ggml/src/ggml-cuda/mmq-config-ampere.cuh	Adds Ampere MMQ config cases for PTQ1_0.
ggml/src/ggml-cuda/convert.cu	Adds a coalesced PTQ1_0 dequantize kernel and routes conversions to it on CUDA (non-HIP).

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[khosravipasha]

Tested on our L40S (CUDA 12.8, sm_89), PR head d14a927 against prism 7113f64 in separate worktrees.

As written (MMQ cap 64): correct and safe to merge.

• test-backend-ops: PTQ1_0 MUL_MAT 45/45, PTQ1_0 MUL_MAT_ID 75/75, all MUL_MAT types 1283/1283.
• Decode unchanged on all four sizes (mmvq path). pp512 within noise (2B 19112 → 20052, 4B 8759 → 9169, 9B 5397 → 5695, 27B 1635 → 1697).
• KLD identical to the cuBLAS baseline, because at ubatch 512 the new kernel is never selected.
• The two red checks (flake8, pyright) were cancelled after 24 h without ever getting a runner; the python change is a list entry plus a template string. Re-running them now.

Follow-up for a separate PR: raise or remove MMQ_PTQ1_0_MAX_BATCH_SIZE. We also built the PR with the cap lifted (1<<30) so MMQ runs at every batch. That changes two things:

PTQ1_0	cuBLAS path (cap 64, current)	MMQ path (cap lifted)
27B mean KLD / top-1	9.2e-4 / 97.78%	5.6e-4 / 98.39%
9B mean KLD / top-1	7.7e-4 / 98.26%	5.0e-4 / 98.60%

The fp16 dequantize-to-cuBLAS fallback is the source of the extra PTQ1_0 error on CUDA. With MMQ at full prefill the 27B PTQ1_0 clears the top-1 certification gate and lands at the same level as PQ2_0; today it is the only shipped band that fails certification on CUDA.

Cost: MMQ is about 7% slower than cuBLAS at pp512 (27B 1689 → 1578, 9B 5721 → 4786, 4B 9249 → 8361) and faster below roughly ubatch 400 (27B pp128 775 → 1377, pp256 1546 vs 1543 cuBLAS). Decode unchanged either way.

Suggested shape for that PR: cap at 512 or above (or drop the cap) so the default ubatch takes the MMQ path and PTQ1_0 certifies on CUDA, and expose the threshold as an env var for A/B. If the 7% at pp512 matters more than certification for some deployment, that env var is the knob. HIP stays excluded by the !defined(GGML_USE_HIP) guard; bringing it in is a further follow-up.

Merging as is; the cap change can be its own PR.

[bri-prism]

Approving. This fills the gap correctly and the mechanism is the right one for prefill.

The structural problem it fixes is real and was easy to miss: PTQ1_0 had zero references in mmq.cu/mmq.cuh against PQ2_0's 3 and 6, so every large-batch matmul fell through to dequantize plus cuBLAS. That materialises the whole weight matrix to fp16 before a dense GEMM, which is where the 2.26x at n=512 comes from.

The tile loader does the thing that makes this a prefill win specifically. Eight threads cooperatively decode each 128-weight block once into shared memory, so the base-3 unpack amortises across the tile width instead of being repaid per output column. That is also why it cannot help decode, which has no tile to amortise over, and it matches what we saw on TQ1_0 where prefill went 5.6x while decode regressed. The lane split checks out too: lanes 0 to 3 cover qs[0:16] for 80 elements, lanes 4 to 5 cover qs[16:24] for 40, and lane 6 handles the qh tail for 8, totalling 128.

One thing worth stating plainly in the description rather than leaving in the footnotes. This verifies correctness, not the performance claim. 45/45 MUL_MAT and 75/75 MUL_MAT_ID on an L40S establish that the path is right, and the cited prefill number is from a pre-#152 base on different hardware. So the honest claim today is "the dequantize-to-cuBLAS fallback is gone", not "prefill matches PQ2_0". That distinction matters because the question this came from was specifically whether prefill can match, and that is still unmeasured on the current tree.

Two smaller notes, neither blocking:

Lane 7 of the 8 threads_per_block does no decode work, so 12.5% of the decode threads idle. Probably immaterial if the MMA dominates, but free to reclaim if the tile load ever shows up in a profile.

The x_df scale write indexes differently on the Turing MMA and dp4a paths, and the dp4a expression i * (2*MMQ_TILE_NE_K/QI8_0) + i/(QI8_0/2) + ksx is the fiddly kind that is either exactly right or silently overlapping. The 45/45 pass covers it on sm_89, so this is a note for anyone touching it later rather than a request.

Related, and worth landing alongside: the qh tail here uses the same v*3 and & 0x00FF00FF recurrence with __byte_perm(w0, w1, 0x7531) and __vsub4 that test-ptq1_0-packed-dot validates over 100,000 random blocks against a scalar decode. That test currently exists only on a local branch. Landing the two together gives this arithmetic permanent coverage, which matters because the test already in tests/ transcribes the pre-dp4a scalar form and so passes without exercising this path.

[bri-prism]

Correcting my own review above. Copilot's comment is right and my approval was wrong on the central point, so treat this as retracting the approval until the selection rule changes.

MMQ_PTQ1_0_MAX_BATCH_SIZE is 64, and the PTQ1_0 early return in ggml_cuda_should_use_mmq() sits before the general turing_mma_available(cc) return:

if (type == GGML_TYPE_PTQ1_0) {
    return ne11 <= MMQ_PTQ1_0_MAX_BATCH_SIZE;   // 64
}

So MMQ is selected for PTQ1_0 only up to ne11 = 64. At prefill n = 512, which is the exact case this PR exists to fix and the one measured at 2.26x slower than PQ2_0, should_use_mmq returns false and the matmul falls back to dequantize plus cuBLAS. The tile loader is correct and well built, and the selection rule then keeps it from running on the workload that motivated it.

test-backend-ops -o MUL_MAT does not catch this. Those shapes are small enough to take the MMQ path, so 45/45 passing is consistent with large-batch prefill still falling back.

This also reframes something I said in my earlier comment. I described the missing end-to-end prefill A/B as a rigour gap, and asked for the number without expecting it to change the verdict. That was too generous. A fresh prefill measurement on the current tree would have shown no improvement at n = 512, because the path is gated off. The absent measurement was concealing a functional gap rather than merely leaving a real win unquantified. Measuring first would have caught this before review did.

Two ways forward, depending on why the bound exists. If 64 is a deliberate guard because the loader or the Ampere config is not yet correct or not yet faster above that, then say so in the code and in the description, and the PR is a decode-and-small-batch improvement rather than a prefill fix, which means the title and the stated goal need changing. If 64 was carried over from the dp4a threshold by analogy with MMQ_DP4A_MAX_BATCH_SIZE, then it should be dropped so PTQ1_0 falls through to the normal Turing path like the other low-bit types.

Either way the gate should be settled by an end-to-end prefill A/B at n = 512 on the current tree, since that number is what the PR claims and it is currently unmeasured.