You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add dense Qwen PagedAttention export for ORT GenAI - #753
Opt in the existing --features paged-attention flow for text-only dense Qwen2, Qwen2.5, and Qwen3 models.
Export separate K/V-packed-token PagedAttention for ORT GenAI CUDA Engine dynamic batching.
Preserve the dense MLA path and reject unsupported graph, configuration, and execution-provider combinations. Native onnx-genai rejects packed page graphs before saving model artifacts; model-specific eligibility lives in the task rather than the reusable component.
Validation
183 related non-integration tests passed. All 18 synthetic CUDA tests passed on A100, comparing full logits and K/V caches against dynamic GQA for Qwen2 and Qwen3 at head dimensions 16, 64, and 128, including decode and a 256-token page crossing. Ruff check/format and git diff --check passed.
The broader non-integration suite had 10,596 passed, 69 skipped, and 2 failures: one test requires missing onnxruntime-genai distribution metadata, while a GGUF shard-identity test did not raise its expected error (also reproduced in isolation). Neither GGUF test implementation nor its source was changed here; this run is not claimed green.
Targeted real-checkpoint standard non-paged Qwen2.5/Qwen3 validation passed: L4 run 36941440582 (2 passed) and L5 run 36941440653 (2 passed, 2 non-applicable skips), both at earlier head f7b8088.
Limits/waivers
Dense paged BF16 is deliberately disabled due to independent BF16 parity evidence. No real-checkpoint paged or ORT GenAI Engine full-generation validation has been completed. Qwen3.8 hybrid is out of scope.
The required aggregate L4/L5 checks in PR run 36932993026 were cancelled after approximately 60 minutes with broad run_all=true because of shared infrastructure edits. Identical Bamba/Bloom/Arcee failures and cancellation were present on the exact base main commit c2fde938 in scheduled L4 run 36813810487 and L5 run 36817813688. The aggregate checks are not claimed green, and required checks were not changed or removed.
Add an opt-in dense PagedAttention export for Qwen2, Qwen2.5, and Qwen3 using the separate K/V packed ABI. Integrate CUDA Engine scheduling while preserving the existing MLA path and its validations.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Dense Qwen paged exports showed A100 h64/h128 full-logit discrepancies versus dynamic GQA and an independent CPU FP32 reference. Gate BF16 through the builder/task/direct GenAI config while preserving MLA BF16 support.
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preflight native runtime compatibility before saving outputs
src/mobius/__main__.py:556
Only the ORT GenAI runtime is preflighted before pkg.save(). With --runtime onnx-genai --features paged-attention, the new native-writer check raises later at write_onnx_genai_config, after the ONNX model and weights have already been written, leaving a partial output directory without runtime metadata. Run the packed-page incompatibility preflight here for the native runtime as well, before saving.
Both CUDA parity suites instantiate the default head_dim=16, but the feature accepts every multiple of 16 and the targeted Qwen checkpoints use representative 64/128-wide heads. PagedAttention dispatch and page-tile behavior change at those widths, so these tests can pass on a non-production kernel path while the advertised Qwen export fails or diverges. Add at least FP16 head_dim 64 and 128 prefill-plus-decode parity coverage (including the page-boundary case) before enabling the full range.
Move Qwen dispatch policy out of the reusable component
src/mobius/components/_paged_attention.py:17
This puts model-family dispatch policy inside the reusable component layer: the new component is now the only component that branches on config.model_type, whereas the analogous MLA component determines eligibility from operator geometry (_paged_mla.py:152-170). Move the Qwen allowlist and Qwen-specific eligibility checks to the model/task or integration policy layer, leaving this component responsible only for the generic SEPARATE PagedAttention contract.
Reject unsupported native runtime exports before saving models, keep Qwen eligibility in the task layer, and exercise representative FP16 head widths across CUDA prefill, decode, and page boundaries.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--features paged-attentionflow for text-only dense Qwen2, Qwen2.5, and Qwen3 models.Validation
183 related non-integration tests passed. All 18 synthetic CUDA tests passed on A100, comparing full logits and K/V caches against dynamic GQA for Qwen2 and Qwen3 at head dimensions 16, 64, and 128, including decode and a 256-token page crossing. Ruff check/format and
git diff --checkpassed.The broader non-integration suite had 10,596 passed, 69 skipped, and 2 failures: one test requires missing
onnxruntime-genaidistribution metadata, while a GGUF shard-identity test did not raise its expected error (also reproduced in isolation). Neither GGUF test implementation nor its source was changed here; this run is not claimed green.Targeted real-checkpoint standard non-paged Qwen2.5/Qwen3 validation passed: L4 run 36941440582 (2 passed) and L5 run 36941440653 (2 passed, 2 non-applicable skips), both at earlier head
f7b8088.Limits/waivers
Dense paged BF16 is deliberately disabled due to independent BF16 parity evidence. No real-checkpoint paged or ORT GenAI Engine full-generation validation has been completed. Qwen3.8 hybrid is out of scope.
The required aggregate L4/L5 checks in PR run 36932993026 were cancelled after approximately 60 minutes with broad
run_all=truebecause of shared infrastructure edits. Identical Bamba/Bloom/Arcee failures and cancellation were present on the exact basemaincommitc2fde938in scheduled L4 run 36813810487 and L5 run 36817813688. The aggregate checks are not claimed green, and required checks were not changed or removed.