ORT Release 1.27.1 cherry pick 1#29628
Merged
Merged
Conversation
Adds a symmetric weight-only **MoE GEMV fast path** for single-token (batch-1) decode of quantized MoE models such as GPT-OSS-20B, replacing the CUTLASS grouped-GEMM path when the expanded row count is small. The fast path now covers **INT4 and INT8** weights, **per-column and block-wise (group size 32/64/128)** scales, and **FP16 and BF16** activations. On an H200 (SM90) this improves end-to-end GPT-OSS-20B INT4 token-generation throughput by roughly **15% over the CUTLASS grouped-GEMM baseline** and about **8% over the FasterTransformer kernel used in ORT 1.26** across prompt lengths 128/1024/2048, while producing bit-faithful output. At batch-1 decode each token expands to `top_k` rows (4 for GPT-OSS-20B), so the MoE FC1/FC2 GEMMs are extremely skinny. The CUTLASS grouped GEMM is built for throughput at larger M and leaves the decode path memory-bound and underutilized. A dedicated weight-only INT GEMV with per-expert dispatch is a better fit for this regime and closes the gap to (and surpasses) ORT 1.26 for GPT-OSS-20B. This also adds block_size=32 support as requested in #29035. | File | Change | |---|---| | `contrib_ops/cuda/llm/moe_gemm/moe_gemv.h` | Public interface: `is_moe_gemv_supported` dispatch predicate and the symmetric INT launchers `launch_moe_gemv_int_symmetric<T, WeightType>` / `launch_moe_gemv_int_symmetric_interleaved_swiglu<T, WeightType>` (plus the original INT4 per-channel launchers). | | `contrib_ops/cuda/llm/moe_gemm/moe_gemv.cu` | Symmetric INT MoE GEMV (INT4 `uint4b_t` / INT8 `uint8_t`, FP16/BF16). One CTA per expanded row × N-tile; per-expert weight/scale/bias offsets via a direct row-to-expert map (prefix-offset scan fallback). Supports per-column (`group_size <= 0`) and block-wise (`group_size` 64/128) scales. FC1 has an interleaved SwiGLU-fused epilogue; a static top-k one-row finalize specialization is used for FC2 routing. | | File | Change | |---|---| | `contrib_ops/cuda/llm/moe_gemm/moe_kernels.cu` / `.h` | Branch FC1 and FC2 to the GEMV fast path when supported (carrying `group_size` through `QuantParams`); fall back to grouped GEMM otherwise. `ORT_DISABLE_MOE_GEMV=1` forces the grouped-GEMM path for A/B testing. | | `contrib_ops/cuda/llm/moe_gemm/moe_gemm_profiler.cc` / `.h`, `moe_util_kernels.h` | Use M (token count) in the MoE GEMM profiler so tactic selection reflects the decode shape. | | File | Change | |---|---| | `contrib_ops/cuda/moe/moe.cc`, `moe_quantization.cc` (+ `.h`) | Treat `swiglu_fusion == 0` with no separate FC3 as interleaved fusion (`== 1`). The published GPT-OSS-20B model (and any model exported by ORT < 1.27) hard-coded the interleaved layout and emits no `swiglu_fusion` attribute, so it defaults to 0; those weights are pre-fused into FC1 and must be treated as fusion mode 1. | | File | Change | |---|---| | `test/python/transformers/profile_qmoe_gemv.py` / `.sh` | Standalone QMoE GEMV profiling harness (NVTX-ranged, GEMV-vs-grouped-GEMM kernel comparison). | | `test/python/transformers/test_qmoe_cuda.py`, `test_moe_cuda.py` | QMoE GEMV decode-latency/parity coverage across INT4/INT8, per-column/block-wise, FP16/BF16, plus import tidy-ups. | | `docs/contrib_ops/cuda/qmoe_gemv_experiments.md` | Full experiment log: kernel-level sweeps, dispatch-gate tuning, block-wise and BF16 enablement, INT8 per-column root-cause + fix, and the GenAI end-to-end throughput comparison (CUTLASS baseline / GEMV final / FT baseline). | | `docs/contrib_ops/cuda/moe_qmoe.md` | QMoE doc updates describing the GEMV fast path and its dispatch gate. | Token-generation throughput (tps, higher is better): | Prompt length | CUTLASS baseline (gemm) | GEMV (final) | FT (ORT 1.26) | GEMV vs cutlass | GEMV vs FT | |---|---|---|---|---|---| | 128 | 248.9 | 288.0 | 265.2 | +15.7% | +8.6% | | 1024 | 237.8 | 272.2 | 252.9 | +14.5% | +7.6% | | 2048 | 231.3 | 265.0 | 245.6 | +14.6% | +7.9% | Benchmark-loop latency in milliseconds, lower is better. `Enabled` is the default GEMV build; `Fallback` sets `ORT_DISABLE_MOE_GEMV=1` (grouped GEMM). Every case reported `has_invalid_output=false`. | Case | Quant | DType | Enabled ms | Fallback ms | Speedup | |---|---|---|---|---|---| | `int8_per_column_m1_top2_1024x4096_e8` | INT8 per-column | FP16 | 0.0566 | 0.0816 | 1.44x | | `int8_per_column_m1_top2_1024x4096_e8` | INT8 per-column | BF16 | 0.0578 | 0.0862 | 1.49x | | `gpt_oss_20b_m1_top4_int8_2880x2880_e32` | INT8 per-column | FP16 | 0.0785 | 0.0947 | 1.21x | | `gpt_oss_20b_m1_top4_int8_2880x2880_e32` | INT8 per-column | BF16 | 0.0785 | 0.0989 | 1.26x | Block-wise INT4 (`block_size=64`, `1024x4096`, e8) routes to GEMV for both FP16 and BF16 with FC1/FC2 kernel times within noise across dtypes (FP16 4.53/6.95 us, BF16 4.62/7.01 us). See `qmoe_gemv_experiments.md` for the full sweep. - GEMV-enabled and `ORT_DISABLE_MOE_GEMV=1` (grouped GEMM) produce identical, correct output for the GPT-OSS-20B sanity prompt; Nsight traces confirm `moe_gemv_kernel` (FC2) and `moe_gemv_interleaved_swiglu_kernel` (FC1) run for the decode shapes. - INT4/INT8 SwiGLU parity cases pass for FP16 and BF16 (max absolute difference ~1e-3 against the reference). Regression: `pytest -k "TestSwigluQMoE or TestQMoEIntPrePackSmoke"` → `34 passed, 4 skipped`. - Test on GPT-OSS-20b with block_size=32, and generated results looks good. - Build: `bash .env/cuda_130.sh --build` (CUDA 13.0, `CMAKE_CUDA_ARCHITECTURES=89;90`). Use `--clean_moe` after dispatch-code edits to avoid stale `moe_kernels.cu.o`. - Kernel profiling: `onnxruntime/test/python/transformers/profile_qmoe_gemv.sh`. - Parity/latency: `pytest onnxruntime/test/python/transformers/test_qmoe_cuda.py`. - End-to-end: GenAI `benchmark_e2e.py` on GPT-OSS-20B INT4, batch 1, prompt lengths 128/1024/2048. Run on an idle GPU — shared-GPU contention corrupts the decode-latency measurement. - A/B check: compare default vs `ORT_DISABLE_MOE_GEMV=1` for both throughput and output parity.
) ### Description `QMoECPU::ComputeCommon` runs an expert loop over the routed experts. Two issues hurt multi-threaded execution: 1. **Nested intra-op parallelism livelocked the pool.** When the expert loop fanned out over the shared session pool (`tp`), the per-expert body issued *further* work on the same pool (token copy, dequant blocks, activation, `MlasGemm` / `DirectQ4Gemm` / `TryRunLutGemm` / `DequantizeBlock`). This nested parallelism on ORT's Eigen pool intermittently livelocked (workers spinning at 100% CPU, never completing). 2. **Decode (batch=seq=1) was poorly parallelized.** The expert thread count was capped by `num_experts` (not *active* experts), and an unconditional work tier cap serialized the active experts. For decode each per-expert GEMM is effectively a GEMV (`M == 1`) that MLAS does not thread internally, so the cores were left idle. This change makes the expert loop parallelize across the **active** experts and keeps parallelism single-level so the livelock fix is preserved: ```cpp // Only experts that actually received tokens do work. int num_active_experts = 0; for (const auto& tokens : expert_token_map) { if (!tokens.empty()) ++num_active_experts; } // One expert (batch) per thread; cap by active experts so a single busy expert // does not look "parallel". int num_expert_threads = std::max(1, std::min(num_active_experts, max_expert_threads)); // Single-level parallelism preserves the no-livelock guarantee: // - expert loop multi-threaded -> inner ops run serially (inner_tp == nullptr) // - single active expert -> loop is serial, inner GEMM gets the full pool (inner_tp == tp) concurrency::ThreadPool* inner_tp = (num_expert_threads > 1) ? nullptr : tp; ``` Changes: - Count active experts and cap `num_expert_threads` by `num_active_experts`. - Derive `inner_tp` from the *active* fan-out, so a single-active-expert batch keeps full inner MLAS/GEMM parallelism (addresses review feedback). - Remove the work tier cap that serialized active experts. This keeps parallelism single-level (no nested pool use), so the original livelock fix is intact while decode and imbalanced routing run on all available cores. ### Motivation and Context A random `Run()` on a quantized MoE model (`com.microsoft.QMoE`) intermittently pinned one core at 100% inside `QMoECPU::Compute` and never returned under multi-threaded intra-op execution; `intra_op_num_threads = 1` avoided it at ~3× latency. The signature — non-deterministic, multi-threaded-only, identical stacks (main thread in the QMoE expert lambda, pool threads in `WorkerLoop`) — points to nested parallelism on the shared thread pool. Restricting QMoE to single-level parallelism removes the hang. Capping by *active* experts (rather than `num_experts`) and dropping the tier cap then restores — and substantially improves — throughput for decode and imbalanced routing. ### Experiment Results Latency in ms/inference (fp32, 4-bit, block size 32, default intra-op threads, AMD EPYC 7763 32 logical / 16 physical cores). **Decode (batch = seq = 1)** | Model (experts / top-k) | Before | After | Speedup | | --- | ---: | ---: | ---: | | gpt_oss_20b (e32 / top4) | 359 | 88 | ~4.1× | | qwen3 (e256 / top8) | 91 | 16 | ~5.6× | | gemma (e128 / top8) | 173 | 27 | ~6.4× | **Mid-range sequence lengths (gpt_oss)** | seq | Before | After | Speedup | | ---: | ---: | ---: | ---: | | 8 | 1776 | 146 | ~12× | | 16 | 1096 | 176 | ~6.2× | | 32 | 1165 | 188 | ~6.2× | | 64 | 705 | 190 | ~3.7× | **Prefill (neutral)** | seq | Before | After | | ---: | ---: | ---: | | 128 | 164 | 166 | | 512 | 192 | 194 | ### Validation - Parity tests: `test_qmoe_cpu.py` — 80 passed, 1 skipped. - Lint: clean (`lintrunner`). - Livelock stress: 8 threads × 300 iterations × 5 shapes = 2400 concurrent runs, no hang, no errors. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tlwu <tlwu@example.com>
<!-- Describe your changes. --> This PR fixes a convolution performance regression affecting some OCR models with large-kernel convolutions when the KleidiAI SME IGEMM convolution path is selected. The change has 2 parts: 1. updates to the KleidiAI IGEMM LHS packing to pack rows in bounded chunks instead of packing the full LHS buffer up front, which reduces memory usage and improves cache locality for large convolutions, 2. a new route selection function `ArmKleidiAI::SelectConvRoute` that decides between `Igemm`, `GemmFallback` and `None` based on convolution parameters and a workload size-based heuristic. The function `CheckCapabilitiesSme` runs `SelectConvRoute` and only returns true if the selected route is `Igemm`. The patch also adds a standard GEMM fallback to the `ConvRoute` possibilities, and runs `MlasGemm` if said fallback is selected. If the function selects `None`, then the convolution falls back to `MlasSgemmOperation`. <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fixes #27633. --------- Signed-off-by: Qxiang Xu <Qixiang.Xu@arm.com> Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com> Signed-off-by: Martin Klacer <martin.klacer@arm.com> Co-authored-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com> Co-authored-by: Damien Dooley <damien.dooley@arm.com>
…29574) ### Description Remove the blanket version check in CustomOpKernel that prevented custom ops compiled against a newer ORT from loading on an older runtime. Instead, cap the version passed to GetApi() at ORT_API_VERSION. This aligns custom ops with the EP plugin ABI pattern where forward compatibility is supported via runtime version detection. Individual newer functions in OrtCustomOp (CreateKernelV2, InferOutputShapeFn, GetMayInplace, etc.) are already gated by per-function version checks throughout custom_ops.cc, making the blanket reject both redundant and harmful. The EP is responsible for not calling API functions newer than the runtime version -- the same contract as the EP plugin interface. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
tianleiwu
requested review from
apsonawane and
hariharans29
and removed request for
apsonawane
July 8, 2026 23:06
tianleiwu
changed the title
hariharans29
previously approved these changes
Jul 8, 2026
apsonawane
previously approved these changes
Jul 8, 2026
### Description Use new GitHub CI identity for azcopy. ### Motivation and Context GitHub CI pools have been assigned a new identity.
### Description <!-- Describe your changes. --> Update most Apple pipelines to use Xcode 26. Keep a build on Xcode 16 for coverage of the older toolchain. Other fixes: - Patch XNNPACK to fix an iphonesimulator x86_64 build error with the newer iphonesimulator SDK. - Fix issue where object files were not copied from the right location for the Mac Catalyst static framework build. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Upgrade to a newer Xcode version. Xcode 16 is no longer available on some build images. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Newer Homebrew versions refuse to load formulae from untrusted third-party taps. This adds `brew trust wix/brew` after tapping to allow the subsequent `brew install applesimutils` to succeed in React Native CI. **Changes:** - `.github/workflows/react_native.yml` - `tools/ci_build/github/azure-pipelines/templates/react-native-ci.yml` Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…yml. (#29575) ### Description <!-- Describe your changes. --> Don't echo command when setting VSO variable in mac-cpu-packing-jobs.yml. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> If the command is echoed again with `set -x`, the VSO variable may end up with a trailing `'`, which is invalid. Whether this actually happens is intermittent and probably dependent on the output ordering.
### Summary Two small, independent CI fixes that unblock currently-failing required pipelines. The NPM packaging pipeline's web e2e consuming test broke when a floating `vite` range pulled the just-released 7.3.x line, and the Python DML pipeline started failing on a MeanVarianceNormalization precision mismatch. Neither change affects runtime code. ### Key Changes | Pipeline | File | Change | Why | |---|---|---|---| | NPM packaging (`web-ci` → e2e) | `js/web/test/e2e/package.json` | Cap `vite` from `^7.1.12` to `>=7.1.12 <7.3.0` | The e2e runner uses a non-deterministic `npm install`, so `^7.1.12` floated onto vite 7.3.6 (the tarball the install aborted on). vite 7.3.0 also bumped esbuild `^0.25.0 → ^0.27.0`, pulling platform binaries not reliably mirrored in the internal feed. Capping below 7.3.0 keeps the bundler smoke test on the known-good line. | | Python DML | `onnxruntime/test/testdata/onnx_backend_test_series_filters.jsonc` | Exclude `^test_mvn_cpu` from the DML EP backend test list | `test_mvn` (MeanVarianceNormalization) fails on DML with 27/27 mismatched elements (max rel diff ~25) — a precision issue, not a functional regression. Filtering it matches how other DML precision/known-issue cases are already handled in this list. | ### Testing Notes - **NPM packaging**: re-run the web CI e2e step (`npm run test:e2e -- --browser=Chrome_default`); `npm install` in `build/js/e2e` now resolves vite to a 7.1/7.2 release instead of 7.3.6, so the install no longer aborts (and the Windows `npm warn cleanup ... EPERM` rollback noise disappears). - **Python DML**: re-run the DML python backend test job; `test_mvn_*` is now skipped for the DML EP alongside the existing excluded cases.
tianleiwu
marked this pull request as draft
July 9, 2026 00:29
tianleiwu
marked this pull request as ready for review
July 9, 2026 04:12
tianleiwu
enabled auto-merge (squash)
July 9, 2026 04:30
hariharans29
approved these changes
Jul 9, 2026
apsonawane
approved these changes
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This cherry-picks the following commits for the release:
Also fixed version missed by version update script.