[agorevski]

Title

Opt in to measured sm_75 interleaved dense verify projections

Issue: No closing reference. Related context: #1349, #2007, #1842; this PR does not establish a fix for those reports.

Summary

Allow Turing (sm_75) to reuse interleaved Q8_1 activations for measured IQ4_XS/Q4_K/Q5_K/Q6_K dense projections in 2–4-token verify windows. Require exactly STRATA_MMVQ_IL=1 and exact input/output dimensions; unset, 0, true, unspecified reduction width and unlisted shapes retain the native fallback.

The branch has been rebased onto fetched origin/main at a0116b2782a5b82622358c04a06496485d12b2d6, not master. Original head d0067686ac537ec0f12617e7ce6458d95b913ad1 is preserved by backup/pr1437-d0067686-rebase-20261012-1152. Rebased implementation: 680b7f6846de07255ee5a20a0e852ba080b38bf6; follow-up tests/documentation: 556f82d2e639f2a4292e752bc21dfd8a95b8552b; final paired-emission correction/head: 43c2c361c0f965931380b6467f49602af7e952c3.

What changed

• Integrate the exact-shape sm_75 table into upstream kIlArch and native_mmvq_il_rows_for, extending the lookup with reduction width. Preserve upstream Pascal/Volta/Ampere/shared tables, Q3_K kernels, one-column handling and STRATA_MMVQ_IL_ROWS on the other eligible architectures. Class overrides cannot broaden sm_75's measured exact-shape policy. The sm_75 table entries are unchanged from the original PR.
• Pass the actual n_in at both dense-projection and output-head eligibility call sites in verify.cpp. Keep upstream's n >= 2 / T >= 2 guards so one-column calls do not interleave.
• Retain validated forced-row tuning and thread-local last-launch observation, and adapt parity to the upstream 1–4-column/Q3_K API. Keep MMVQ_IL_ONLY, --bench --emit-table, output-row-class table emission and its one-column entries; retain --sm75-bench for the original 2–4-column paired sweep including pack cost.
• Normalize each emitted row choice against its own paired kernel-only baseline, rather than borrowing the last row choice's reference timing. Pack-inclusive ratios remain separate for the sm_75 sweep.
• Add three CPU-only CTest checks for architecture tables, disabled Q3_K/one-column extensions, and sm_75 isolation from class overrides. Exact-shape rejection includes different reduction width, adjacent output size, unknown width, one column and Q3_K.
• Update docs/OLDER_GPUS.md and docs/MMVQ_IL_TABLE.md with architecture selection, the override boundary, CPU checks and clearly attributed community reports.

No old conflicted file was restored wholesale. No upstream architecture tables, one-column or Q3_K support were removed. No community class table or STRATA_EXP_MODE change is adopted.

Extra Notes

Comment follow-up and independent reports

• ioridev: RTX 2080 Ti report / conflict concern. The framework conflict is resolved locally against current main, retaining its new APIs/features rather than reverting them. Their measurements were a port of the unchanged table onto 0.1.42 (61b3fb5d), with the strict opt-in and actual input widths, not the exact PR head. They did not port this PR's extended harness or last-launch observer. On two RTX 2080 Ti cards (22 GB main / 11 GB peer, 250 W, i9-9820X, CUDA 12.4/GCC 15.2), IQ3_S + MTP, spec 4, INT8 KV, BF16_TC and SELECT_SIMT enabled, they reported bitwise parity using 0.1.42's harness and mean decode 68.3 → 71.3 tok/s (+4.4%, five workloads, four-run medians with reversed arm order). Their ~8K measurements were noisy; baseline greedy replies already varied, so text equality was not their correctness evidence. These are their results, not our new tests or measurements of the rebased head.
• xrip: CMP 50HX report. This was a separate output-row-class table, including different Q3_K and one-column choices, not this PR's exact-shape table. Two CMP 50HX cards (TU102, 2100 MHz, 56 SMs, CUDA 13.3/GCC 14), IQ3_XXS, layer split, spec 4 + MTP, INT8 KV, PCIe fraction 0.17: reported bitwise T=1–4 parity and +7.0% decode, 72.62/72.57 → 77.85/77.46 tok/s, with a fixed 35.7K-token prompt followed by six 512-token questions. The table is not copied here. Their STRATA_EXP_MODE=8 expert-layout results are outside this PR.

The PR discussion comments and reviews, plus GET repos/Niko1221/Strata/pulls/1437/comments, were read. There were no formal reviews or inline review comments returned; the two discussion reports above are addressed in the code/docs and follow-up reply.

Historical author measurements, not a rebased whole-model retest

Original Quadro RTX 8000 tuning retained 46/69 measured shape/window cells: every paired trial in both seven-pair sweeps saved at least 3%, charging the full interleave launch. The retained entries remain unchanged.

Previously recorded controlled whole-model decode medians (three runs per prompt, 128 outputs):

Prompt tokens	baseline (tok/s)	STRATA_MMVQ_IL=1 (tok/s)
537	83.7	88.9
4,057	75.4	80.4
32,057	68.4	72.1

Quadro RTX 8000, 260 W, Xeon W-2295, CUDA 12.4/GCC 13, IQ3_S + MTP, spec 4, fixed 19,000 expert slots, automatic prefill 8,192, INT8 KV, maximum context 262,144 / resident 32,768, PCIe fraction 0.20. Prompt caching and adaptation were disabled; BF16_TC was already enabled in both arms, with BF16 cache, SIMT scorer and CPU prefill sharing off. These are historical results, not fresh server runs after this rebase.

Fresh local build and exact validation outcomes

All commands ran in the independent worktree:
/home/algore/.copilot/session-state/a7c2bcbb-4264-4eb2-879d-c71a90285b0b/files/pr-worktrees/sm75-verify.
The original dirty checkout was not modified; only its requested llama.cpp vendor source was shared read-only.

PATH=/usr/bin:/bin:/home/algore/miniconda3/bin cmake -S . -B build-rebase1437-sm75 \
  -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES=75 \
  -DCMAKE_CUDA_COMPILER=/home/algore/miniconda3/bin/nvcc \
  -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++ \
  -DSTRATA_GGML_DIR=/home/algore/GIT/strata/third_party/llama.cpp
PATH=/usr/bin:/bin:/home/algore/miniconda3/bin cmake --build build-rebase1437-sm75 --target strata mmvq_il_parity -j2
PATH=/usr/bin:/bin:/home/algore/miniconda3/bin ctest --test-dir build-rebase1437-sm75 \
  -R '^mmvq_il_(tables|tables_disable_extensions|sm75_override_isolation)$' --output-on-failure

Results: fresh configure passed, exit 0; both targets built, exit 0, with CUDA 12.4.131 and system GCC/G++ 13.3.0. Final-source rebuild also passed, exit 0. Compiler warnings were emitted, but no build/tool incompatibility was encountered. Focused CPU tests passed 3/3, exit 0.

The host was reported to have an NVML driver/library mismatch (nvidia-smi, NVML 580.178). We did not reboot, replace drivers/libraries or change the user environment. The actual CUDA runtime worked: the first bounded attempt below passed, and only then were full parity and bounded benchmark-interface checks run. This does not assert NVML was repaired.

flock -x /home/algore/.copilot/session-state/a7c2bcbb-4264-4eb2-879d-c71a90285b0b/files/pr-gpu3.lock \
  env -u STRATA_MMVQ_IL_ROWS -u STRATA_MMVQ_IL_T1 -u STRATA_MMVQ_IL_Q3K \
  CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=3 STRATA_MMVQ_IL=1 \
  MMVQ_IL_ONLY='IQ4_XS 2560x6144' ./build-rebase1437-sm75/mmvq_il_parity

Result: passed, exit 0, Quadro RTX 8000 logical 0 / physical 3, bitwise T=1–4; 24 forced interleaved, 6 automatic interleaved / 2 fallback launches.

Final-source full validation command (the exclusive lock covers the entire loop and both smoke commands):

flock -x /home/algore/.copilot/session-state/a7c2bcbb-4264-4eb2-879d-c71a90285b0b/files/pr-gpu3.lock \
  env -u STRATA_MMVQ_IL_ROWS -u STRATA_MMVQ_IL_T1 -u STRATA_MMVQ_IL_Q3K -u MMVQ_IL_ONLY \
  CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=3 bash -c '
    for policy in 1 0 true unset; do
      if [ "$policy" = unset ]; then
        env -u STRATA_MMVQ_IL ./build-rebase1437-sm75/mmvq_il_parity
      else
        env STRATA_MMVQ_IL="$policy" ./build-rebase1437-sm75/mmvq_il_parity
      fi > "build-rebase1437-parity-$policy.log" 2>&1
      result=$?
      cat "build-rebase1437-parity-$policy.log"
      printf "POLICY=%s EXIT=%d\n" "$policy" "$result"
      if [ "$result" -ne 0 ]; then exit "$result"; fi
    done
    STRATA_MMVQ_IL=1 MMVQ_IL_ONLY="Q3_K 2560x6144" \
      ./build-rebase1437-sm75/mmvq_il_parity --bench --emit-table \
      > build-rebase1437-emit-smoke.log 2>&1 &&
    STRATA_MMVQ_IL=1 MMVQ_IL_ONLY="IQ4_XS 2560x6144" \
      ./build-rebase1437-sm75/mmvq_il_parity --sm75-bench \
      > build-rebase1437-sm75-bench-smoke.log 2>&1
  '
git diff --check
git diff --check origin/main...HEAD

Results:

• =1: passed, exit 0, bitwise T=1–4; 720 forced interleaved, 86 automatic interleaved / 154 fallback launches.
• =0, =true, unset: each passed, exit 0, bitwise T=1–4; each 720 forced interleaved, 0 automatic interleaved / 240 fallback launches. Strict opt-in and exact-shape assertions passed.
• --bench --emit-table, filtered Q3_K shape: passed, exit 0, including actual forced Q3_K one-column launches and class/one-column emission; 12 forced interleaved, 0 automatic / 4 fallback.
• --sm75-bench, filtered IQ4_XS shape: passed, exit 0, T=2–4, seven alternating pairs per forced-row choice with pack cost; 9 forced interleaved, 3 automatic / 0 fallback. These bounded smoke checks are not a new full tuning sweep or grounds for replacing the measured table.
• Both diff checks passed, exit 0. A source comparison also confirmed the upstream shared/sm_60/sm_70/sm_86/one-column tables and original sm_75 exact entries are unchanged.

Logs remain in the worktree as build-rebase1437-*.log (untracked build artifacts are ignored). No servers, power-limit changes or other GPUs were used. Other architectures were host-table-tested, not hardware-tested or compiled for their architectures. No new whole-model measurements were taken.

Related issues found through repository issue search

• #1349 — dual RTX 2080 Ti field report: relevant Turing throughput and workload/context sensitivity evidence, but its RAM/peer/prefill/pipeline experiments are not measurements of this table and its questions are not fixed here.
• #2007 — sm_75 T10 long-context regression: relevant Turing performance report; it explicitly reports that disabling MMVQ-IL did not recover the measured prefill regression. This opt-in dense verify table is not a demonstrated regression fix.
• #1842 — Ada long-context MMVQ-IL regression: directly relevant caution about applying another architecture's row-class table to unmeasured hardware. This PR preserves Ada's upstream table/override behavior and does not fix its reported slowdown.

No Resolves/Fixes claim is made.

[ioridev]

Tested on RTX 2080 Ti (sm_75), which may be useful since the table was tuned only on a Quadro RTX 8000. Short version: it helps here too. Whole-model decode is +4.4% on average (up to +7%), and parity is bitwise.

Setup

• 2× RTX 2080 Ti (22 GB modded card as main + 11 GB card as peer tier, --peer-device 1), 250 W power limit, clocks not locked, i9-9820X, driver 595.91, CUDA 12.4, GCC 15.2
• Qwen3.8-Flash-Next IQ3_S + MTP, --spec 4, INT8 KV, max context 131072, --pcie-frac 0, STRATA_BF16_TC=1, STRATA_SELECT_SIMT=1
• The dense shapes and types in this GGUF match the PR's kIlRowsSm75 entries exactly (e.g. attn_output 6144→2560, ffn_*_shexp 2560→640, output Q6_K 2560→248320).

Note: this is a port, not the PR head. The PR conflicts with current main (61b3fb5d, 0.1.42), where the IL table was reworked into per-architecture tables (native_mmvq_il_rows_for, kIlArch, STRATA_MMVQ_IL_ROWS). I ported only the core:

• the kIlRowsSm75 table, unchanged
• the opt-in, enabled only by exactly STRATA_MMVQ_IL=1
• n_in passed to native_mmvq_il_supported from both call sites in verify.cpp

I did not port the parity-harness changes or native_mmvq_il_last_rows(). The diff is at the bottom.

Parity

0.1.42's mmvq_il_parity with STRATA_MMVQ_IL=1 on the 22 GB card: OK (every case, T 2-4, rows 1/2/4 and the table, bitwise). Note that this is 0.1.42's harness, not the PR's extended one.

Kernel timing

From 0.1.42's mmvq_il_parity --bench. The time per call is in µs: multi → IL at the rows the sm_75 table picks for that cell. This bench is a single timed pass, not paired A/B, and it does not include the interleave launch.

shape	T=2	T=3	T=4
IQ4_XS 2560×10240	58.9 → 33.1	76.4 → 35.8	91.4 → 37.6
IQ4_XS 2560×6144	29.7 → 19.9	38.1 → 21.9	49.4 → 22.0
Q4_K 2560×10240	45.3 → 31.1	59.6 → 35.6	73.0 → 38.7
Q5_K 2560×12288	59.3 → 45.8	76.1 → 47.0	91.9 → 60.7
Q5_K 2560×6144	30.8 → 23.3	40.5 → 26.1	47.9 → 31.1
Q6_K 2560×10240	56.9 → 48.1	67.8 → 55.8	83.2 → 54.1
Q6_K 2560×12288	66.3 → 56.8	82.9 → 56.8	101.0 → 61.8

The cells the table leaves out are small or lose on this card too, so the table's exclusions also hold here. For example, Q6_K 2560×512 at T=2 went 6.4 → 7.8 / 8.5 / 17.0 µs at rows 1/2/4.

Whole-model decode

Decode tok/s through the server, temperature 0. Each cell is the median of 4 runs: two passes in opposite order (base before IL, then IL before base), server restarted per variant, 2 alternating rounds each. Runs were 400 tokens for en/ja/code and 300 tokens after a long prompt.

	en	ja	code	after ~8K prompt	after ~30K prompt	mean
0.1.42	70.3	66.1	71.9	62.9	70.5	68.3
+ port, STRATA_MMVQ_IL=1	73.8 (+5.0%)	67.0 (+1.3%)	77.1 (+7.2%)	66.6 (+6.0%)	72.0 (+2.2%)	71.3 (+4.4%)

Caveats:

• The ~8K row is noisy: the same variant ranged 54–74 tok/s.
• The code result is the clearest: all four IL runs (76.5–78.1) were above all four baseline runs (68.5–74.1).
• Text equality could not be used as a check, because on this setup temperature-0 outputs already differ from run to run on the unmodified baseline. Correctness rests on the bitwise kernel parity above.

Port onto 61b3fb5 (0.1.42)

diff --git a/include/strata/kernels/native_mmvq.hpp b/include/strata/kernels/native_mmvq.hpp
index fef71027..f7ff5b67 100644
--- a/include/strata/kernels/native_mmvq.hpp
+++ b/include/strata/kernels/native_mmvq.hpp
@@ -143,7 +143,8 @@ std::size_t native_q8_1_il_bytes(int n_in, int ncols);
 // to x_il (native_q8_1_il_bytes): the same values in the same bytes.
 void native_q8_1_interleave(const void* x_q8_1, void* x_il, int n_in, int ncols, void* stream);
 // True when native_mmvq_il runs its own kernel for this call (IQ4_XS, Q4_K, Q5_K, Q6_K, 2-4 columns, the exact layout).
-bool native_mmvq_il_supported(int ggml_type, int ncols, int n_out);
+// n_in: the reduction width (sm_75's table is by exact shape; 0 = any)
+bool native_mmvq_il_supported(int ggml_type, int ncols, int n_out, int n_in = 0);
 // native_mmvq for 2..4 columns whose copy native_q8_1_interleave wrote, bitwise native_mmvq's output: kernels in which a
 // warp takes 1, 2 or 4 rows and reads the columns from the interleaved copy; native_mmvq (x_q8_1) when not supported.
 void native_mmvq_il(int ggml_type, const void* weights, const void* x_q8_1, const void* x_il, float* y, int n_in,
diff --git a/src/core/verify.cpp b/src/core/verify.cpp
index 3e9ec069..f2a9896d 100644
--- a/src/core/verify.cpp
+++ b/src/core/verify.cpp
@@ -1009,7 +1009,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
         float* xm = mixed_ + tb * N;
         bool il_ready = false;   // xil_ holds xq_'s interleaved copy (reset whenever xq_ is rewritten)
         auto mm = [&](const WeightRef* w, float* out, int n_in, int n_out) {
-            if (g_mmvq_il() && strata::kernels::native_mmvq_il_supported(w->native_type, n, n_out)) {
+            if (g_mmvq_il() && strata::kernels::native_mmvq_il_supported(w->native_type, n, n_out, n_in)) {
                 if (!il_ready) {
                     strata::kernels::native_q8_1_interleave(xq_, xil_, n_in, n, cs);
                     il_ready = true;
@@ -1666,7 +1666,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
         if (head_ != nullptr && head_->loaded()) {
             try {
                 native_quantize_q8_1(head_mixed_, xq_, (int) N, T, cs);
-                if (g_mmvq_il() && strata::kernels::native_mmvq_il_supported(head_->type(), (int) T, (int) n_vocab_)) {
+                if (g_mmvq_il() && strata::kernels::native_mmvq_il_supported(head_->type(), (int) T, (int) n_vocab_, (int) N)) {
                     strata::kernels::native_q8_1_interleave(xq_, xil_, (int) N, (int) T, cs);
                     strata::kernels::native_mmvq_il(head_->type(), head_->weights(), xq_, xil_, head_logits_, (int) N,
                                                     (int) n_vocab_, (int) T, cs);
diff --git a/src/kernels/cuda/native_mmvq.cu b/src/kernels/cuda/native_mmvq.cu
index 6a47e6fa..45e940ce 100644
--- a/src/kernels/cuda/native_mmvq.cu
+++ b/src/kernels/cuda/native_mmvq.cu
@@ -2595,6 +2595,41 @@ namespace {
 // native_mmvq's kernels unchanged
 // Volta (PR 1401): opt-in with STRATA_SM70_TABLE=1 until the author confirms on a V100 with the final code
 bool sm70_opt_in() { const char* v = std::getenv("STRATA_SM70_TABLE"); return v != nullptr && std::atoi(v) != 0; }
+// PR #1437 (ported to 0.1.42): Turing opt-in with exactly STRATA_MMVQ_IL=1.  Quadro RTX 8000, mmvq_il_parity
+// --sm75-bench: two sweeps of seven alternating A/B pairs, cold-weight rotation, T=2/3/4; only cells where every
+// pair in both sweeps saves >=3% (charging the interleave launch).  Exact GGUF dense shapes only.
+struct IlShapeRows { int type, n_in, n_out; uint8_t r[3]; };
+constexpr IlShapeRows kIlRowsSm75[] = {
+    {23, 2560,    640, {0, 0, 2}},
+    {23, 2560,   6144, {2, 4, 2}},
+    {23, 2560,  10240, {2, 2, 2}},
+    {23, 2560,  12288, {2, 4, 2}},
+    {12, 2560,   6144, {2, 2, 2}},
+    {12, 2560,  10240, {2, 2, 2}},
+    {12, 2560,  12288, {4, 2, 2}},
+    {12, 6144,   2560, {4, 2, 2}},
+    {13, 2560,   6144, {2, 2, 2}},
+    {13, 2560,  10240, {2, 4, 4}},
+    {13, 2560,  12288, {4, 2, 4}},
+    {13, 6144,   2560, {0, 4, 4}},
+    {14, 2560,   6144, {0, 2, 4}},
+    {14, 2560,  10240, {2, 2, 4}},
+    {14, 2560,  12288, {2, 4, 4}},
+    {14, 6144,   2560, {0, 2, 2}},
+    {14, 2560, 248320, {2, 4, 2}},
+};
+bool sm75_opt_in() {
+    static const int on = [] { const char* v = std::getenv("STRATA_MMVQ_IL"); return v != nullptr && std::strcmp(v, "1") == 0; }();
+    return on != 0;
+}
+int il_rows_sm75(int type, int ncols, int n_out, int n_in) {
+    for (const IlShapeRows& e : kIlRowsSm75)
+        if (e.type == type && e.n_out == n_out && (n_in == 0 || e.n_in == n_in))
+            return e.r[ncols - 2];
+    return 0;
+}
+// rows a warp for this call on this card (0: native_mmvq)
+int il_dispatch_rows(int type, int ncols, int n_out, int n_in);
 bool il_arch_ok() {
     static int ok[16] = {};   // 0 unknown, 1 yes, -1 no, by device ordinal
     int dev = 0;
@@ -2604,25 +2639,34 @@ bool il_arch_ok() {
         int minor = 0;
         cudaDeviceGetAttribute(&minor, cudaDevAttrComputeCapabilityMinor, dev);
         ok[dev] = (cudaDeviceGetAttribute(&major, cudaDevAttrComputeCapabilityMajor, dev) == cudaSuccess &&
-                   (major >= 8 || (major == 7 && minor == 0 && sm70_opt_in()))) ? 1 : -1;
+                   (major >= 8 || (major == 7 && minor == 0 && sm70_opt_in()) ||
+                    (major == 7 && minor == 5))) ? 1 : -1;
     }
     return ok[dev] > 0;
 }
 }  // namespace
 
-bool native_mmvq_il_supported(int ggml_type, int ncols, int n_out) {
-    if (ncols < 2 || ncols > 4 || !g_multi_exact || !il_arch_ok()) return false;
-    return (g_tune_rows ? g_tune_rows : il_rows(ggml_type, ncols, n_out)) != 0 &&
-           (ggml_type == 23 || ggml_type == 12 || ggml_type == 13 || ggml_type == 14);
+namespace {
+int il_dispatch_rows(int type, int ncols, int n_out, int n_in) {
+    if (ncols < 2 || ncols > 4 || !g_multi_exact || !il_arch_ok() ||
+        (type != 23 && type != 12 && type != 13 && type != 14)) return 0;
+    if (g_tune_rows) return g_tune_rows;
+    if (il_cc() == 75) return sm75_opt_in() ? il_rows_sm75(type, ncols, n_out, n_in) : 0;
+    return il_rows(type, ncols, n_out);
+}
+}  // namespace
+
+bool native_mmvq_il_supported(int ggml_type, int ncols, int n_out, int n_in) {
+    return il_dispatch_rows(ggml_type, ncols, n_out, n_in) != 0;
 }
 
 void native_mmvq_il(int ggml_type, const void* weights, const void* x_q8_1, const void* x_il, float* y, int n_in,
                     int n_out, int ncols, void* stream) {
-    if (!native_mmvq_il_supported(ggml_type, ncols, n_out)) {
+    const int r = il_dispatch_rows(ggml_type, ncols, n_out, n_in);
+    if (!r) {
         native_mmvq(ggml_type, weights, x_q8_1, y, n_in, n_out, ncols, stream);
         return;
     }
-    const int r = g_tune_rows ? g_tune_rows : il_rows(ggml_type, ncols, n_out);
     validate_shape(n_in, ncols, QK);
     validate_pointer(weights);
     validate_pointer(x_il);
@@ -2643,7 +2687,7 @@ std::size_t native_q8_1_il_bytes(int n_in, int ncols) { return native_q8_1_bytes
 void native_q8_1_interleave(const void*, void*, int, int, void*) {
     throw std::invalid_argument("native_q8_1_interleave is CUDA only");
 }
-bool native_mmvq_il_supported(int, int, int) { return false; }
+bool native_mmvq_il_supported(int, int, int, int) { return false; }
 int native_mmvq_il_rows_for(int, int, int, int) { return 0; }
 void native_mmvq_il(int ggml_type, const void* weights, const void* x_q8_1, const void*, float* y, int n_in, int n_out,
                     int ncols, void* stream) {

[xrip]

Another Turing data point, in case it helps this land. I measured a table on a third sm_75 card before I saw this PR, and have closed my duplicate (#2089).

Card: NVIDIA CMP 50HX, a TU102 at RTX 2080 Ti clocks (2100 MHz, 56 SMs). Driver 610.43.03, CUDA 13.3, GCC 14, Linux.

Parity: mmvq_il_parity: OK (every case, T 1-4, rows 1/2/4 and the table, bitwise).

Kernel times, --bench (us per call, plain -> best interleaved):

shape	columns	plain	interleaved
IQ4_XS 2560x10240	4	94.6	35.6
Q4_K 2560x10240	4	88.3	45.3
Q6_K 2560x12288	4	123.6	70.5
Q5_K head 2560x248320	3	1781	1097

Table from --emit-table (row classes by n_out, as in kIlRows_86):

{23, {{0, 4, 2, 2, 2}, {4, 4, 2, 2, 2}, {2, 2, 2, 2, 2}}},   // IQ4_XS
{12, {{0, 4, 4, 4, 4}, {1, 2, 2, 2, 2}, {1, 2, 2, 2, 2}}},   // Q4_K
{13, {{0, 4, 4, 4, 4}, {1, 4, 2, 4, 4}, {1, 4, 2, 2, 2}}},   // Q5_K
{14, {{0, 2, 2, 2, 2}, {0, 4, 4, 4, 4}, {0, 4, 4, 2, 2}}},   // Q6_K
{11, {{1, 2, 2, 4, 4}, {1, 2, 4, 4, 4}, {1, 2, 2, 2, 2}}},   // Q3_K
one column: IQ4_XS {0,2,2,2,2}, Q4_K {0,0,2,2,2}, Q5_K {0,0,1,1,2}, Q6_K {0,0,2,1,1}, Q3_K {0,2,2,2,2}

Decode A/B in the engine:

• Setup: 2x CMP 50HX, --layer-split auto, Qwen3.8-Flash-Next IQ3_XXS, --spec 4 --mtp, --kv int8, --adapt-swaps 0 --pcie-frac 0.17, CPU prompt share off.
• Method: one binary, the arms differ only by the variable. A fixed 35.7K-token prompt, then 6 questions x 512 greedy tokens.
• Result: 72.62 / 72.57 tok/s without the table, 77.85 / 77.46 with it (+7.0 %). The prompt read did not change.

So this is a third Turing card, alongside the Quadro RTX 8000 and the RTX 2080 Ti above, with a gain in the same direction.

On the same card, STRATA_EXP_MODE=8 (Volta's expert layout) was bitwise equal on all 48 layers and gave +2.9 % decode on its own, and +10.3 % together with the table. That is outside this PR, but it may be useful if sm_75 defaults are discussed.

[agorevski]

Thanks, ioridev and xrip. Rebased onto main (a0116b27), integrating exact sm_75 shapes into kIlArch/native_mmvq_il_rows_for while preserving upstream overrides, Q3_K and one-column paths. Both verifier queries pass actual input widths.

CUDA 12.4/GCC 13 build: passed. CPU policy checks: 3/3 passed. Physical GPU 3, exclusively locked: four full parity policies passed bitwise, 720 forced launches each; opt-in selected 86 IL/154 fallback, other policies 0/240. Both benchmark-interface smoke checks passed. CUDA worked despite the reported NVML mismatch; no drivers/power settings changed.

Docs attribute the RTX 2080 Ti port (+4.4%) separately from the CMP 50HX different table (+7%); EXP_MODE remains out of scope. No new whole-model measurements. Updated head 43c2c361. Related issues #1349, #2007 and #1842 are linked as context, not claimed fixes.

[ioridev]

Thanks for the rebase, and for crediting the RTX 2080 Ti numbers separately in the docs!
If it's not too much trouble, I'd appreciate being added as a co-author on the docs commit:

Co-authored-by: Iori Kojima <iori@iori.dev>