Skip to content

feat(moe): support gsq q2_0 routed experts - #660

Draft
wenlive wants to merge 1 commit into
FlashML-org:mainfrom
wenlive:feat/gsq-q2-routed-experts
Draft

wenlive wants to merge 1 commit into
FlashML-org:mainfrom
wenlive:feat/gsq-q2-routed-experts

Conversation

@wenlive

@wenlive wenlive commented Oct 11, 2026

Copy link
Copy Markdown

GSQ Q2_0 routed experts cannot currently bind to the MoE path. This adds an expert-only GGUF source for Qwen3.8-Flash-Next while preserving the HF trunk, shared-expert and PLE quantization schemes.

Related to #549. This does not implement complete GSQ GGUF model loading or IQ3_XXS, so it does not close that issue.

  • Native 64-weight / 18-byte blocks, bounded complete-expert reads, and directory/geometry/receipt checks before bank allocation.
  • Existing offload cache with a BF16/FP32 Triton vector kernel; initially TP=1, SiLU and pure GPU offload.
  • Real miniature GGUF tests, independent Torch numerical references, slot remapping and CUDA graph replay; source contract and usage.

Draft for maintainer feedback on the external-source contract and reader placement. Maintainer design discussion, Linux testing, GPU testing with main's latest dependencies and full-model generation on this exact feature remain pending. No end-to-end performance claims.

Tested on Windows 11, Ryzen 9 9900X, 64 GB RAM, RTX 5090 D v2 24 GB, driver 610.88, Python 3.12.13:

$env:PYTHONPATH = "$PWD\python"
python -B -m pytest tests/moe/test_q2_0.py tests/moe/test_q2_source.py -q
  • Torch 2.11.0+cu130 / triton-windows 3.6.0.post26: 16 passed.
  • Torch 2.14.1+cpu: 11 passed, 5 CUDA cases skipped.
  • Complete source read: all 48 x 512 experts / 33,973,862,400 packed bytes from ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF revision ed59f92082b1e93c0e96d60a8b11aab089b52f09; no inference or full bank allocation.

Commands and raw validation logs.

AI-assisted. These tests were executed by the assistant on my local hardware; a separate manual human rerun is not claimed.

@jason-fxz

Copy link
Copy Markdown
Collaborator

Thanks for the careful work here. #690 loads the ISTA-DASLab Qwen3.8-Flash-Next-GSQ-RCO IQ3_XXS GGUF directly, Q2_0 experts included, on llama.cpp's ggml-cuda kernels, and it runs end to end with fused and offload. It does not use the HF trunk plus GGUF experts approach of this PR.

#690 is the new GGUF infrastructure. Since it loads the whole GGUF file directly, we are not planning to merge a separate GGUF expert path, and this PR can be closed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants