[kunal-vaishnavi]

Description

When deterministic compute is enabled, CUDA MatMulNBits now selects a stable fpA-intB tactic instead of using the timing-profiled choice. For small row counts it selects the available CUDA GEMV tactic; otherwise it selects a fixed non-split-K GEMM tactic. This path bypasses the profiled tactic cache. With deterministic compute disabled, the existing profiling behavior is unchanged.

The PR adds tests for cache independence and exact 2-, 4-, and 8-bit weight reconstruction in deterministic mode.

Motivation and Context

Fresh processes running the same quantized model with greedy decoding sometimes selected different MatMulNBits tile and split-K tactics. Those tactics produced slightly different FP16 outputs from identical inputs, eventually changing the ninth generated token from 680 to 264. ORT’s existing deterministic-compute flag did not previously govern this tactic selection.

On SM80, the isolated MatMul output matched bitwise in 12/12 fresh processes after the change. In a downstream full-model validation, 36/36 deterministic runs produced the same token stream and tactic fingerprint; with the flag disabled, both original branches still appeared. Deterministic selection may be slower than a profiled tactic. Validation does not establish identical results across hardware or batch shapes.

Qwen INT4 decode specialization

This PR now also routes the Qwen Flash-Next language-model projection through the existing M=1 symmetric INT4 CUDA GEMV specialization for the exact shape:

• M = 1
• N = 248320
• K = 2560
• block size 32
• no zero points

The dispatch remains shape- and format-gated, so other MatMulNBits workloads retain their existing selection behavior. The specialization reduces single-token draft-head projection overhead used by Qwen speculative decoding.

[github-actions]

You can commit the suggested changes from lintrunner.

[Copilot]

Copilot review overview

🟡 Changes recommended

The selected deterministic GEMM tactic is invalid on SM89 and causes execution failure.

Review effort: Balanced
Findings: 1

Open (1)

• Select an architecture-valid MatMulNBits config for SM89 · New

What changed in this PR

Adds deterministic CUDA MatMulNBits tactic selection independent of profiling state.

Changes:

• Selects stable GEMV or non-split-K GEMM tactics.
• Adds cache-independence and exact reconstruction tests.
• Honors deterministic session configuration during execution.

File	Description
fpA_intB_gemm_profiler.h	Declares deterministic tactic selection.
fpA_intB_gemm_profiler.cc	Implements fixed tactic selection.
matmul_nbits.cc	Uses deterministic tactics when requested.
matmul_nbits_workspace_test.cc	Tests profile-cache independence.
matmul_nbits_fpa_intb_layout_test.cc	Tests deterministic weight reconstruction.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

The new Qwen dispatch lacks regression coverage and leaves router-only documentation inconsistent with its expanded behavior.

Review effort: Balanced
Findings: 1 · 1 · 1

Open (3)

• Select an architecture-valid MatMulNBits config for SM89
• Add regression coverage for Qwen block-size-32 dispatch · New
• Update specialization comments and naming for Qwen support · New

[kunal-vaishnavi]

Synced with latest main (79d3fcf) in merge commit 40c1f97; the merge was conflict-free. Review/CI fixes are in 5b73613, and all five open threads have individual replies and are resolved.

Changes:

• Reject unsupported stage-2 deterministic GEMM tactics on SM89+, preserving older-architecture stage-2 and native SM90 configs. Added architecture-selection coverage and retained profile-cache independence.
• Share the exact single-row int4 dispatch predicate with direct Qwen acceptance/nearby-rejection tests, preventing fallback-only parity from masking a lost dispatch gate.
• Generalize router-only kernel/helper names and comments for GPT-OSS router and Qwen draft-head workloads, preserving the existing environment-variable control and arithmetic.
• Apply both formatting suggestions.

CI diagnosis:

• Linux CUDA/TensorRT, Windows CUDA/TensorRT, and the documentation build failed on the old base because routing_record was undeclared. The main merge incorporates upstream Replace old use of routing_record with routing_snapshot_ in onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc. #32932, which fixes that issue.
• Generated operator/kernel docs match the new target exactly; no schema or registration changes require additional regeneration.
• The old Android minimal-baseline size check exceeded its threshold by 365 bytes. This PR has no CPU/build-system differences from current main, and a recent main baseline passed: https://github.com/microsoft/onnxruntime/actions/runs/37873110425/job/113635421181 . The threshold was not relaxed; new-head CI will verify the refreshed base.

Local validation:

• Compiled/ran the actual edited deterministic-selector function against architecture-sensitive synthetic candidate lists.
• Compiled/ran the actual shape-eligibility test body: 19 runtime assertions and two compile-time assertions passed.
• Full profiler and CUDA-internal test source pass compiler checks with the internal test body enabled.
• FP16/BF16/FP32 CUDA instances compile; clang-format, whitespace checks, and editor diagnostics pass.

No full ORT runtime was rebuilt or installed, and no SM89 hardware execution or fresh-provider gtest execution is claimed. New-head CI is pending with no failures reported at this check; stale-head workflows were not rerun. The user checkout and installed runtime were preserved.

[kunal-vaishnavi]

Synced this PR with main through 327d11a in merge 7c915d3, preserving deterministic selection and the newer shape-aware INT4 decode profiling logic.

Fixed the shared Linux CUDA / Windows CUDA / TensorRT test failure in 39ff12d. Native SM90 tactics intentionally leave the legacy split-K factor and pipeline-stage fields at -1; the regression now checks native tile metadata and those unused-field values, while legacy tactics retain architecture-valid stage and non-split-K assertions (also valid in quick builds). No tactic metadata was changed just to satisfy the test.

The WASM Debug failures were the external-data tests repaired upstream in 2297e77 (#33226), which is now included by the target merge.

All five existing review threads already had individual fix replies and were resolved. Those fixes remain intact; no unresolved threads were present at the final pre-push check.

Validation:

• Reproduced the original CI assertion using the actual selector, actual candidate generator, and original test body. The fixed architecture and shape-eligibility test bodies pass in standalone host gtests in both normal and ORT_QUICK_BUILD configurations.
• Actual profiler, MatMulNBits wrapper, and both regression translation units pass warnings-as-errors syntax checks.
• The complete matmul_4bits.cu translation unit compiles for sm_90.
• H200 direct-source specialization checks: 18 oracle cases across the exact Qwen shape and GPT-OSS group sizes 32/64, FP32/FP16/BF16, with/without bias; 36 bitwise repeat comparisons. CUDA memcheck and synccheck both report zero errors.
• Targeted clang-format and git diff --check pass; worktree is clean.

Full ORT provider, Windows, and WASM suites were not run locally. The push starts fresh CI; no stale old-head jobs were rerun.

[Tianlei Wu (tianleiwu)]

LGTM — deterministic tactic selection across SM75–SM90 and the single-row INT4 GEMV shape gate and tests look clean.