Repository navigation
Conversation
Collaborator
|
Thanks for the careful work here. #690 loads the ISTA-DASLab Qwen3.8-Flash-Next-GSQ-RCO IQ3_XXS GGUF directly, Q2_0 experts included, on llama.cpp's ggml-cuda kernels, and it runs end to end with #690 is the new GGUF infrastructure. Since it loads the whole GGUF file directly, we are not planning to merge a separate GGUF expert path, and this PR can be closed. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GSQ Q2_0 routed experts cannot currently bind to the MoE path. This adds an expert-only GGUF source for Qwen3.8-Flash-Next while preserving the HF trunk, shared-expert and PLE quantization schemes.
Related to #549. This does not implement complete GSQ GGUF model loading or IQ3_XXS, so it does not close that issue.
Draft for maintainer feedback on the external-source contract and reader placement. Maintainer design discussion, Linux testing, GPU testing with main's latest dependencies and full-model generation on this exact feature remain pending. No end-to-end performance claims.
Tested on Windows 11, Ryzen 9 9900X, 64 GB RAM, RTX 5090 D v2 24 GB, driver 610.88, Python 3.12.13:
ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUFrevisioned59f92082b1e93c0e96d60a8b11aab089b52f09; no inference or full bank allocation.Commands and raw validation logs.
AI-assisted. These tests were executed by the assistant on my local hardware; a separate manual human rerun is not claimed.