IQ kernels: decode each weight part once for every column and entry, bitwise identical by gputier · Pull Request #242 · Niko1221/Strata

gputier · 2026-09-30T13:35:48Z

IQ kernels: decode each weight part once for every column and entry, bitwise identical

Rebased on v0.1.29. One commit: iq_kernels.cu and its header, a new parity test (iq_multi_parity), and additions to iq_parity and native_expert_parity.

What it does

iq_mmvq, native_gu_kernel and native_down_kernel called the format's dot once per column (or group entry), re-reading the weight words and redoing the grid lookups and sign unpacking each time. Each dot is now split into a load (weight side) and an apply (activation side, same integer chain and same float expression), the way native_mmvq.cu's multi-column traits already work. The dense kernel takes 1, 2, 4 or 8 columns per pass (8 at a time beyond 8), the grouped ones 4 entries per pass (about 48 registers; the default verify window fits in one pass).

Every column keeps the same calls, lane stride, summation order and warp_sum, so results are bitwise equal to the old kernels.

Default path is identical to the byte

Measured on an RTX 5090 (sm_120, Windows, MSVC 2022, CUDA 13.4), Release build, v0.1.29 as the base.

iq_multi_parity (new, in ctest, synthetic, no model): old against new bitwise for iq2_xxs, iq2_xs, iq3_xxs, iq4_nl, iq3_s, iq2_s, iq4_xs, iq1_m and q2_0, iq_mmvq with 1 to 8 and 11 columns, and native_expert_grouped with 9 groups and 35 entries for the 24 pairings of 8 gate/up formats with IQ4_NL, Q2_0 and IQ4_XS down. 0 failures.
native_expert_parity on the real IQ3_S shard (layers 0, 1, 2, 3, 20, 47): 0 failures, and for each layer "gpu decode-once vs per-entry kernels: bitwise equal".
iq_parity now also requires every column of a 1 to 8 column native_mmvq call to equal a one-column call bitwise. tools/iq_fixture.py is not in the tree, so I ran it on fixtures built locally from gguf-py (random valid blocks, 64 x 512, dequantized by gguf-py from the pinned llama.cpp): 8 IQ formats and Q3_K pass, dequant error 0, dot within 6e-3 of the float product, and the multi-column check holds. Q2_0 has no gguf-py fixture and reports "missing fixture", exactly as v0.1.29 does.
Full ctest, same machine: v0.1.29 alone 35 of 38 pass, this branch 36 of 39. The same three fail in both, only because the Q2_0 model files are not installed: ple_parity, expert_parity, pool_test.
End to end greedy A/B against v0.1.29 alone, IQ3_S model, --spec 4 --expert-cache 8000 --prefill 2048 --max-new 256, three prompts (32, 2 520 and 23 020 tokens), two passes each: all 256 output tokens are identical in every run, and identical with STRATA_OLD_IQ_MMVQ=1 on the shortest and longest prompt.

Gain

Kernel micro-benchmark (iq_multi_parity --bench, RTX 5090, idle GPU), microseconds per call, old / new:

call	1 column	4 columns	8 columns
IQ2_XXS `iq_mmvq` 8192 x 2560	10.0 / 9.9	18.5 / 13.3	29.9 / 20.5
IQ3_S `iq_mmvq` 8192 x 2560	10.5 / 9.3	21.1 / 16.1	38.9 / 20.5
IQ1_M `iq_mmvq` 8192 x 2560	10.2 / 9.9	20.5 / 15.1	34.8 / 20.5

Grouped, 16 groups of m entries (IQ3_S gate/up with Q2_0 down): m=1 39.9 / 40.1, m=4 76.0 / 54.1, m=8 129.4 / 90.2. Across the six formats benchmarked, m=8 goes from 109 to 129 us down to 82 to 90 us.

End to end, no gain shows on the IQ3_S model with the default --spec 4 window: decode is 142.8 tok/s for v0.1.29 and 142.8 / 143.3 for this branch on the 23K prompt, and 137.5 / 136.9 against 138.8 / 138.1 on the 2.5K one, within run to run noise. Most of the win is at 3 or more columns or entries per call, which the default window rarely reaches on this model.

Switch back

STRATA_OLD_IQ_MMVQ=1 (or iq_set_old_kernels) keeps the per-column kernels.

Not tested

HIP / ROCm: not built, not run.
Two GPUs or more.
Quants other than IQ3_S end to end (the other formats are covered by the synthetic and fixture parity tests only), and a larger --spec window where the multi-column path is hit more often.
Sampled (non-greedy) generation, and architectures other than sm_120.

iq_mmvq, native_gu_kernel and native_down_kernel called the format's dot once per column (or group entry), re-reading the weight words and redoing the grid lookups and sign unpacking each time. Each dot is now split into load (weight side) and apply (activation side, same integer chain and float expression), as native_mmvq.cu's multi-column traits are: the dense kernel takes 1, 2, 4 or 8 columns per pass (8 at a time past 8), the grouped ones 4 entries per pass (~48 registers; the default verify window, --spec 4 with the split off, fits in one pass). Every column keeps the same calls, lane stride, summation order and warp_sum, so results are bitwise equal to the old kernels, which STRATA_OLD_IQ_MMVQ=1 (or iq_set_old_kernels) keeps for A/B timing. Tests: iq_multi_parity (ctest, GPU, synthetic, no model) checks old vs new bitwise for every format, ncols 1..8 and 11, groups of 0..11 entries; --bench times both. iq_parity (real rows from tools/iq_fixture.py) now runs the dot for 1 to 8 columns and requires every column of a multi-column call to equal a one-column call on it bitwise (exact layout, the default). native_expert_parity runs the GPU expert with both the decode-once and the per-entry kernels on real blobs and requires the same bits.

BlueKingMuch · 2026-09-30T23:58:25Z

Confirmed bitwise equal on an RTX 4080 SUPER (sm_89, Windows 11; IQ3_S, 262K; 0.1.30 + this PR): with and without STRATA_OLD_IQ_MMVQ=1 the deterministic requests (8K, no prompt-lookup drafts, no expert swaps) give the same 256 tokens, greedy and sampled, in the same number of rounds. GPU wait per round -0.15 ms there (rounds of about 25 ms), i.e. +0.3 % / +0.6 % tokens/s. (The verify windows here average about 3.4 tokens.)

…_XL), per-role format lists, native_expert_supported at startup, Q8_0 embedding and PLE key - Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata 8029fa9 and #255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y. - Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16 fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types; iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten. - The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before. - native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right after expert_layout_load and refuses an unsupported pair by name before anything is allocated (it used to exit from inside the first verify window). - Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it. - native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged. - Not yet: #242's decode-once traits for the new types (they take the per-entry kernel; see phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

Niko1221 · 2026-10-01T05:14:11Z

Released in 0.1.31: bitwise identical on all four quants (iq_multi_parity, native_expert_parity on real IQ3_S/IQ3_XXS); the Unsloth formats use the per-entry kernel until they get their own traits. Merged under your name (GitHub doesn't show it as merged because it went in by hand). Thank you!

A decode window on this model is a long chain of small kernels, and on a fast card many of them are over before the next one has fetched its weights. These changes cut launches and let the next kernel start early. Every one keeps the arithmetic of the path it replaces (checked with per-layer hashes of the MoE inputs and the final residual over full requests: identical windows), and each has an off switch for A/B runs. - Programmatic dependent launch on sm_90+ (STRATA_DF_PDL, default on): the verify kernels trigger their dependents early, and the dependents prefetch their weight rows into L2 before waiting for the activations. Off on older cards. - Two side streams inside the verify graph (STRATA_DF_BRANCH): the query side (q, the indexer query) runs beside the K/V side and joins before attention. - A Q4_0 KV cache batches too (STRATA_DF_QB_Q4, STRATA_DF_KVAPP): the query rotation is row-wise, so the window's rows go in one launch, and the window's K/V cells are appended in one launch. - PLE: the per-token post-ops of a window in one kernel (STRATA_DF_PLE). - A layer split's hand-off copies (3 x T) in one launch (STRATA_DF_HANDOFF). - native mmvq: a warp per row for large outputs (STRATA_DF_MMVQW): 10-30% less time on 6144..248320-row IQ4_XS on both cards (slower at 640 rows, so it starts at 2048). - Attention loops with their loads batched (STRATA_DF_ATTN), SwiGLU folded into the shared expert's down quantization (STRATA_DF_SHARED). - copy_rows_strided replaces a per-token cudaMemcpy2DAsync for the q/gate split. - An L2 prefetch of the next layer's weights on a side stream, opt-in (STRATA_DF_L2PF_MB=n, STRATA_DF_L2PF_MIN_SM). It did not help on the 5080 + 4060 Ti. Upstream's staged hyper-connection read (Niko1221#315) and decode-once multi-column IQ kernels (Niko1221#242) do what two earlier parts of this change did, so those parts were dropped in favour of upstream's.

…_XL), per-role format lists, native_expert_supported at startup, Q8_0 embedding and PLE key - Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata 8029fa9 and #255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y. - Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16 fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types; iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten. - The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before. - native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right after expert_layout_load and refuses an unsupported pair by name before anything is allocated (it used to exit from inside the first verify window). - Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it. - native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged. - Not yet: #242's decode-once traits for the new types (they take the per-entry kernel; see phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates.

…_XL), per-role format lists, native_expert_supported at startup, Q8_0 embedding and PLE key - Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata 8029fa9 and Niko1221#255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y. - Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16 fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types; iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten. - The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before. - native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right after expert_layout_load and refuses an unsupported pair by name before anything is allocated (it used to exit from inside the first verify window). - Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it. - native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged. - Not yet: Niko1221#242's decode-once traits for the new types (they take the per-entry kernel; see phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates.

…_XL), per-role format lists, native_expert_supported at startup, Q8_0 embedding and PLE key - Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata 8029fa9 and Niko1221#255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y. - Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16 fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types; iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten. - The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before. - native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right after expert_layout_load and refuses an unsupported pair by name before anything is allocated (it used to exit from inside the first verify window). - Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it. - native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged. - Not yet: Niko1221#242's decode-once traits for the new types (they take the per-entry kernel; see phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN

gputier mentioned this pull request Sep 30, 2026

[UP1] Amont : tester les trois lots sur GPU et ouvrir les PR Q2_0 et IQ gputier/StrataGP#58

Closed

BlueKingMuch mentioned this pull request Oct 1, 2026

hc: the multi-token hyper-connection read split finer #315

Closed

Niko1221 closed this Oct 1, 2026

BlueKingMuch mentioned this pull request Oct 1, 2026

experts: smaller and fewer launches for the verify window's PCIe call #363

Closed

lawsirlawsir-png mentioned this pull request Oct 2, 2026

gfx1100 (W7800 48 GB, 30 GB RAM): full-resident experts — decode 76.5 t/s @ ~100K ctx on 0.1.34 (no thinking), + hipBLASLt 100202 table, IQ2_XS vs IQ3_XXS reversed #505

Closed

gputier deleted the pr/iq-grid-decode-0.1.29 branch October 11, 2026 12:38

Navigation Menu

Repository navigation

IQ kernels: decode each weight part once for every column and entry, bitwise identical - #242

IQ kernels: decode each weight part once for every column and entry, bitwise identical#242
gputier wants to merge 1 commit into
Niko1221:mainfrom
gputier:pr/iq-grid-decode-0.1.29

gputier commented Sep 30, 2026

Uh oh!

BlueKingMuch commented Sep 30, 2026

Uh oh!

Niko1221 commented Oct 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

gputier commented Sep 30, 2026