Repository navigation
Add experimental INT3 QMoE checkpoint export - #764
Open
titaiwangms wants to merge 2 commits into
Open
titaiwangms wants to merge 2 commits into
titaiwangms wants to merge 2 commits into
Conversation
Gate raw Olive INT3 fused expert export behind an experimental flag. Validate packing, scales and source profiles, and cover public local-checkpoint export and external-data round trips. Runtime schema and kernel qualification remain pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: titaiwang <titaiwang@microsoft.com>
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The documented programmatic opt-in relies on a private, test-oriented API rather than a supported public export.
2 open findings
What changed in this PR
Adds an opt-in experimental exporter for Olive INT3 QMoE checkpoints while explicitly deferring runtime support.
Changes:
- Adds INT3 layout validation, packing, and QMoE attributes.
- Introduces an experimental feature flag and documentation.
- Adds comprehensive offline export and round-trip tests.
| File | Description |
|---|---|
README.md |
Documents experimental usage and limitations. |
src/mobius/_flags.py |
Adds the INT3 export feature flag. |
src/mobius/_weight_utils.py |
Validates and preprocesses INT3 checkpoint data. |
src/mobius/components/_moe.py |
Emits draft INT3 QMoE graphs and metadata. |
src/mobius/models/moe.py |
Passes activation dtype into preprocessing. |
src/mobius/models/int3_qmoe_test.py |
Tests packing, validation, builds, and persistence. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| ```python | ||
| from mobius import build | ||
| from mobius._flags import override_flags |
Comment on lines
+290
to
+291
| "INT3 QMoE export requires experimental_int3_qmoe_export=True; " | ||
| "the proposed INT3 schema and kernels are not released runtime support." |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a default-off, experimental exporter for Olive fused INT3 MoE checkpoints.
This is an export-only draft, not a claim that released ORT can execute INT3
QMoE models. It follows the proposed raw-weight direction in
microsoft/onnxruntime#32657 and consumes the native Torch INT3 checkpoint format
introduced by microsoft/Olive#2716.
Changes
MOBIUS_EXPERIMENTAL_INT3_QMOE_EXPORT=1oroverride_flags(experimental_int3_qmoe_export=True).(3,4),(3,3), and(4,3)in the generic MoE adapter,including Qwen3-MoE, with exact Olive expert overrides and an INT4 model-wide
fallback.
zero point 4. Interleave FC1 gate/up bytes and scales during preprocessing;
preserve FC2's K-last layout without requantization.
activation/scale agreement, finite nonnegative scales, and optional explicit
INT3 zero points. Preserve existing INT4 zero-point semantics.
metadata, without assigning an unapproved schema version.
local-checkpoint public-build coverage, and external-data save/reload tests.
Validation
Run from the isolated draft worktree:
PYTHONPATH="$PWD/src" HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \ python -m pytest src/mobius/models/int3_qmoe_test.py \ src/mobius/models/moe_test.py src/mobius/_weight_utils_test.py \ src/mobius/components/_moe_test.py -q --tb=shortPassed: 312 tests, no skips, including 57 new INT3 tests.
Passed.
Passed.
Draft boundaries
quality, or performance qualification was run.
MatMulNBits, quantized embeddings,(3,8)/(8,3), INT2/INT3 mixedpairs, and other checkpoint layouts are not enabled.
runtime qualification remain follow-up gates. This does not complete the
end-to-end plan's P5 acceptance gate.