[Feature Request] Support compressed-tensors per-channel FP8 checkpoints for Qwen3.5 MoE

Summary

I would like to request support for loading compressed-tensors FP8 checkpoints that use per-channel weight scaling, specifically for models such as:

ornith-ai/Ornith-1.5-35B-A3B-FP8

The model architecture itself appears to already be supported by FreeToken through the existing Qwen3_5MoeForConditionalGeneration / qwen3_5_moe implementation.

The missing part seems to be support for this checkpoint's FP8 serialization and quantization scheme.

Model

Hugging Face model:

ornith-ai/Ornith-1.5-35B-A3B-FP8

Architecture:

"architectures": [
  "Qwen3_5MoeForConditionalGeneration"
],
"model_type": "qwen3_5_moe"

The checkpoint uses compressed-tensors with FP8 weights:

"quant_method": "compressed-tensors",
"format": "float-quantized"

The relevant quantization scheme is approximately:

FP8 weights
static per-channel weight scaling
dynamic per-token activation scaling
selected modules excluded from quantization

This differs from the FP8 formats currently handled by the Qwen3.5 MoE loader, such as block FP8 and ModelOpt-style FP8.

Current behavior

FreeToken already has support for the Qwen3.5 MoE model architecture, so no new model implementation should be required.

However, the current quantization detection and weight-loading paths do not appear to recognize compressed-tensors per-channel FP8 checkpoints.

As a result, this checkpoint cannot be loaded using its intended FP8 representation.

Requested support

It would be useful to add a dedicated loading path for compressed-tensors FP8 checkpoints with per-channel weight scales.

A possible initial scope could be:

Detect compressed-tensors FP8 checkpoints with float-quantized format.
Support FP8 tensors accompanied by per-channel weight_scale tensors.
Correctly handle scale broadcasting according to the stored tensor shape rather than assuming per-tensor or block scaling.
Support Qwen3.5 MoE expert weights, including stacked expert banks.
Respect the checkpoint's quantization ignore/exclusion rules.
Support weight and scale tensors located in different safetensors shards.

Possible incremental implementation

A first implementation does not necessarily need native per-channel FP8 MoE execution.

For correctness-first support, FreeToken could initially load:

FP8 weight + per-channel scale
              |
              v
        dequantize to BF16
              |
              v
     existing BF16 operators /
        MoE expert backend

This would allow the checkpoint to run without requiring a new FP8 MoE kernel immediately.

Native storage and execution could then be added separately:

FP8 expert weights
+ per-channel scales
+ dynamic activation scaling
        |
        v
native FP8 MoE expert backend

That would preserve the memory savings of the original checkpoint.

Likely affected areas

The main changes appear to be around the existing Qwen3.5 MoE quantization and weight-loading code, for example:

python/freetoken/models/qwen3_5_moe/config.py
python/freetoken/models/qwen3_5_moe/weight.py

The existing model architecture, attention implementation, and registry should not need major changes.

Suggested tests

Possible regression coverage:

Detect compressed-tensors per-channel FP8 config correctly.
Load a synthetic FP8 weight with a 1D per-output-channel scale.
Verify dequantized weights against a reference implementation.
Verify weight and scale lookup across different safetensors shards.
Verify stacked MoE expert loading.
Compare logits against a Hugging Face / reference implementation on a small deterministic input.

Motivation

FreeToken already supports the underlying Qwen3.5 MoE architecture, and supporting this additional FP8 checkpoint format would make models such as Ornith-1.5-35B-A3B-FP8 usable without requiring a separate architecture implementation.

It may also benefit other Qwen3.5 MoE checkpoints distributed using the same compressed-tensors per-channel FP8 format.

目前freetoken 可以使用 Ornith-1.5-35B 么?

目前freetoken 可以使用 Ornith-1.5-35B 么?

It works on Ornith-1.5-35B-A3B, but not Ornith-1.5-35B-A3B-FP8

We added support for this checkpoint class and we tested it on hardware. We found three separate problems. Each problem hides the next one, so we give all three here.

Problem 1: the detector does not accept `strategy: "channel"`

_attn_quant in models/qwen3_5_moe/config.py returns "none" for a per-channel FP8 group.

On main (af71ba4) this function has no compressed-tensors branch. PR #390 adds the branch, but the branch tests strategy == "tensor". A per-channel checkpoint still gets "none".

We checked this with unsloth/Qwen3.6-35B-A3B-NVFP4-Fast:

tree	attn_quant	lm_head_quant
main `af71ba4`	`none`	`none`
#390 head	`none`	`none`

This value is a routing key. iter_weights calls _iter_weights_attn_fp8 only when the value is "fp8_pertensor". With a wrong value, the loader uses a different path. That path cannot read this checkpoint. We saw a KeyError for the fused shared_expert.gate_up_proj.

The correction is one token: strategy in ("tensor", "channel").

Problem 2: the loader cannot read a per-channel weight scale

_per_row_scale in models/qwen3_5_moe/weight.py does scale.reshape(1). This is correct for a single scalar. A per-channel scale holds one scalar for each output row, and it arrives as [rows, 1]. The reshape then fails:

RuntimeError: shape '[1]' is invalid for input of size 248320

Fp8PerTensorLinear.weight_scale is already per output row, so the correction is small. Reshape to [-1], then use the element count to select the action. One element broadcasts. A count of rows passes through. Any other count must raise an error.

The error is the important part. A wrong count that broadcasts applies the scale of row 0 to every output row. The model then loads, and it gives fluent but wrong output.

Problem 3: an FP8 `lm_head` cannot load

The FP8 branch in _iter_weights_attn_fp8 emits every FP8 weight in native form, and this includes lm_head. But the model builds the bf16 ParallelLMHead for every head that is not NVFP4. That class has no weight_scale buffer, so the load stops:

Unexpected keys in state_dict: ['lm_head.weight_scale']

No existing head class can use a head that is already FP8 on disk. Nvfp4LMHead is FP4. The FP8 heads in glm_moe_dsa and glm5_next are subclasses of the bf16 ParallelLMHead, and they quantize at load time.

A native FP8 head is 0.474 GiB for a [248320, 2048] matrix. The bf16 form is 0.947 GiB. On a 16 GiB card this difference is more than decode traffic. The bf16 head left 118 MiB free, and the GDN chunked-prefill workspace then caused an out-of-memory failure at 8192 prompt tokens.

Results

Hardware: one RTX 4080 SUPER, 16 GiB, sm_89, PCIe 4.0 x16, TP=1.

The build also carries an unrelated FP8 KV cache change, so these throughput numbers do not come from a clean tree.

The checkpoint loads.
Decode: 120.05 tokens/s at 1200 tokens of context.
Prefill: 3311 tokens/s at 8192 tokens, 8 of 8 requests.
BFCL parallel tool calls: 95.00.

About `ornith-ai/Ornith-1.5-35B-A3B-FP8`

We did not test this checkpoint. Its shape differs from ours. We read its config:

format: "float-quantized", not "mixed-precision"
one FP8 group with strategy: "channel"
no NVFP4 group
lm_head in ignore

Warning

Corrected. This paragraph originally claimed the checkpoint needs only the fixes for problems 1 and 2. That was wrong -- I had read the config but never run the detectors against it. It needs strictly more than that. See the correction further down this thread.

Offer

We have the three changes on separate branches, each with tests in the style of tests/models/test_glm5_next_config.py.

Tell us how you want the work split, and we will open the pull requests. The channel token may belong in #390 instead of a separate change. That is your decision.

Branches

The branches are below, in case you want to read or cherry-pick the commits. Each branch holds one commit for each concern. The history is linear, so each commit cherry-picks on its own.

#	Commit	Base	Fixes	Branch
1	`165931c`	`main` `af71ba4`	problem 2	`upstream/ct-fp8-per-channel-scale`
2	`a8df84d`	commit 1	problem 3	`upstream/fp8-lm-head`
3	`f4e3270`	#390 head	problem 1	`upstream/ct-fp8-channel-strategy`
4	`bc27359`	commits 1-3	the packed shared expert	`upstream/ct-fp8-packed-shared-expert`

Notes on the bases:

Commits 1 and 2 apply to main. They do not need fix(models): better support for mixed-precision compressed-tensors NVFP4 #390.
Commit 3 needs the compressed-tensors branch that fix(models): better support for mixed-precision compressed-tensors NVFP4 #390 adds, so it sits on the fix(models): better support for mixed-precision compressed-tensors NVFP4 #390 head.
Branch 4 is the full stack. It is the tree we measured.

Commits 1 and 2 apply to main, but they are not reachable there. On main the only checkpoints that reach _iter_weights_attn_fp8 are ModelOpt checkpoints, and those use a per-tensor scale and an NVFP4 head. So on main these two commits are hardening, and they become necessary when a per-channel checkpoint can route. We merged commits 1 and 2 into the #390 head with no conflict.

Commit 2 also skips an lm_head that the checkpoint lists in ignore. compressed-tensors gives ignore precedence over a group's targets, and such a head is bf16 on disk.

Each commit adds tests in the style of tests/models/test_glm5_next_config.py. The new files are tests/models/test_qwen3_5_moe_config.py, test_qwen3_5_moe_weight.py and test_qwen3_5_moe_fp8_lm_head.py. There were no test_qwen3_5_moe_* files before, so nothing is replaced.

We are happy to open the pull requests in any order you prefer, or to combine them. We can also leave the branches here if you would rather apply the changes yourself.

We added support for this checkpoint class and we tested it on hardware. We found three separate problems. Each problem hides the next one, so we give all three here.

Problem 1: the detector does not accept strategy: "channel"

_attn_quant in models/qwen3_5_moe/config.py returns "none" for a per-channel FP8 group.

On main (af71ba4) this function has no compressed-tensors branch. PR #390 adds the branch, but the branch tests strategy == "tensor". A per-channel checkpoint still gets "none".

We checked this with unsloth/Qwen3.6-35B-A3B-NVFP4-Fast:

tree attn_quant lm_head_quant
main af71ba4 none none
#390 head none none
This value is a routing key. iter_weights calls _iter_weights_attn_fp8 only when the value is "fp8_pertensor". With a wrong value, the loader uses a different path. That path cannot read this checkpoint. We saw a KeyError for the fused shared_expert.gate_up_proj.

The correction is one token: strategy in ("tensor", "channel").

Problem 2: the loader cannot read a per-channel weight scale

_per_row_scale in models/qwen3_5_moe/weight.py does scale.reshape(1). This is correct for a single scalar. A per-channel scale holds one scalar for each output row, and it arrives as [rows, 1]. The reshape then fails:
RuntimeError: shape '[1]' is invalid for input of size 248320
Fp8PerTensorLinear.weight_scale is already per output row, so the correction is small. Reshape to [-1], then use the element count to select the action. One element broadcasts. A count of rows passes through. Any other count must raise an error.

The error is the important part. A wrong count that broadcasts applies the scale of row 0 to every output row. The model then loads, and it gives fluent but wrong output.

Problem 3: an FP8 lm_head cannot load

The FP8 branch in _iter_weights_attn_fp8 emits every FP8 weight in native form, and this includes lm_head. But the model builds the bf16 ParallelLMHead for every head that is not NVFP4. That class has no weight_scale buffer, so the load stops:
Unexpected keys in state_dict: ['lm_head.weight_scale']
No existing head class can use a head that is already FP8 on disk. Nvfp4LMHead is FP4. The FP8 heads in glm_moe_dsa and glm5_next are subclasses of the bf16 ParallelLMHead, and they quantize at load time.

A native FP8 head is 0.474 GiB for a [248320, 2048] matrix. The bf16 form is 0.947 GiB. On a 16 GiB card this difference is more than decode traffic. The bf16 head left 118 MiB free, and the GDN chunked-prefill workspace then caused an out-of-memory failure at 8192 prompt tokens.

Results

Hardware: one RTX 4080 SUPER, 16 GiB, sm_89, PCIe 4.0 x16, TP=1.

The build also carries an unrelated FP8 KV cache change, so these throughput numbers do not come from a clean tree.

The checkpoint loads.

Decode: 120.05 tokens/s at 1200 tokens of context.

Prefill: 3311 tokens/s at 8192 tokens, 8 of 8 requests.

BFCL parallel tool calls: 95.00.

About ornith-ai/Ornith-1.5-35B-A3B-FP8

We did not test this checkpoint. Its shape differs from ours. We read its config:

format: "float-quantized", not "mixed-precision"

one FP8 group with strategy: "channel"

no NVFP4 group

lm_head in ignore

This checkpoint needs the corrections for problem 1 and problem 2. It does not need the correction for problem 3.

Offer

We have the three changes on separate branches, each with tests in the style of tests/models/test_glm5_next_config.py.

Tell us how you want the work split, and we will open the pull requests. The channel token may belong in #390 instead of a separate change. That is your decision.

Branches

The branches are below, in case you want to read or cherry-pick the commits. Each branch holds one commit for each concern. The history is linear, so each commit cherry-picks on its own.

Commit Base Fixes Branch

1 165931c main af71ba4 problem 2 upstream/ct-fp8-per-channel-scale
2 929f4ce commit 1 problem 3 upstream/fp8-lm-head
3 f4e3270 #390 head problem 1 upstream/ct-fp8-channel-strategy
4 fa8c878 commits 1-3 the packed shared expert upstream/ct-fp8-packed-shared-expert
Notes on the bases:

Commits 1 and 2 apply to main. They do not need fix(models): better support for mixed-precision compressed-tensors NVFP4 #390.

Commit 3 needs the compressed-tensors branch that fix(models): better support for mixed-precision compressed-tensors NVFP4 #390 adds, so it sits on the fix(models): better support for mixed-precision compressed-tensors NVFP4 #390 head.

Branch 4 is the full stack. It is the tree we measured.

Commits 1 and 2 apply to main, but they are not reachable there. On main the only checkpoints that reach _iter_weights_attn_fp8 are ModelOpt checkpoints, and those use a per-tensor scale and an NVFP4 head. So on main these two commits are hardening, and they become necessary when a per-channel checkpoint can route. We merged commits 1 and 2 into the #390 head with no conflict.

Each commit adds tests in the style of tests/models/test_glm5_next_config.py. The new files are tests/models/test_qwen3_5_moe_config.py, test_qwen3_5_moe_weight.py and test_qwen3_5_moe_fp8_lm_head.py. There were no test_qwen3_5_moe_* files before, so nothing is replaced.

We are happy to open the pull requests in any order you prefer, or to combine them. We can also leave the branches here if you would rather apply the changes yourself.

Thanks for investigating this and for providing the branches. I’m only the issue author and not a FreeToken maintainer, so I’m not in a position to decide how the changes should be split or merged.

@yongjer

@yongjer that's completely fair -- and honestly it was the wrong question to put to you. Thanks for saying so plainly rather than letting it sit.

Turned out two of the four commits don't overlap with #390 or with @Romeriz's parallel work, so I've opened those on their own instead of waiting for someone to adjudicate a split:

fix(qwen3_5_moe): accept a per-channel fp8 weight_scale in the dense loader #415 -- the per-channel weight_scale fix (problem 2 above)
feat(qwen3_5_moe): keep a natively-fp8 lm_head native instead of failing to load #416 -- the FP8 lm_head (problem 3)

Holding the other two back: the strategy: "channel" token belongs in #390 rather than in a PR of mine, and the packed-shared-expert commit depends on how the MoE routing question between #390 and Romeriz resolves.

Correction -- your checkpoint needs more than I said

I need to walk back something from my first comment. I claimed ornith-ai/Ornith-1.5-35B-A3B-FP8 would need only the per-channel scale fix plus accepting strategy: "channel". I'd read the config but never actually run the detectors against it. When I did, it came back all none -- even with every one of my four commits applied:

attn='none'  lm_head='none'  expert='none'  fp8_block='none'  dense_mlp='none'

The blocker isn't the strategy at all. It's targets:

"config_group_0": { "targets": ["Linear"], "weights": {
    "num_bits": 8, "type": "float", "strategy": "channel", "group_size": null } }

"Linear" is a class-name target -- in compressed-tensors it matches every nn.Linear in the model, minus ignore. But _attn_quant (both on main and in #390) decides by looking for the literal substring .self_attn or linear_attn in each target string. A class-name target contains neither, so the group is skipped and the checkpoint never routes, regardless of "channel". Making "channel" work was necessary but nowhere near sufficient, and I'm sorry for the confident-sounding wrong answer.

There's a second, larger consequence I missed. With targets: ["Linear"] and only 7 entries in ignore, your experts are per-channel FP8 as well. FreeToken's expert offload banks are keyed by format -- currently bf16, fp8_block, q4_0, awq, ct_int4, ct_int4_asym, nvfp4, mxfp4, ds_fp4 -- and there's no per-channel FP8 entry. So supporting this checkpoint means a new expert bank format, not a small loader patch. That's a different and much bigger piece of work than anything in #415/#416.

I also can't test it here even if it were supported: per-channel FP8 experts come to roughly 30 GiB of pinned host RAM for 40 x 256 experts, and this box has 31 GB total.

So to be straight with you: #415 and #416 do not close this issue. They're both still needed for a per-channel FP8 dense side, and #415 in particular is a prerequisite for any of it, but the class-name-target resolution and the expert bank format are the parts that actually stand between FreeToken and your model. I didn't want to leave my earlier estimate sitting there looking authoritative.

For what it's worth, primitive-ai/Ornith-1.5-35B-A3B-mixed-NVFP4-FP8 does resolve correctly today (attn='fp8_pertensor', expert='nvfp4') because its targets are real module regexes and its experts are NVFP4. I'm pulling that one now to benchmark, since it's a clean test of #390 on hardware. Different checkpoint from yours, though -- I won't pretend otherwise this time.

Navigation Menu

Repository navigation

[Feature Request] Support compressed-tensors per-channel FP8 checkpoints for Qwen3.5 MoE #252

Summary

Model

Current behavior

Requested support

Possible incremental implementation

Likely affected areas

Suggested tests

Motivation

Problem 1: the detector does not accept `strategy: "channel"`

Problem 2: the loader cannot read a per-channel weight scale

Problem 3: an FP8 `lm_head` cannot load

Results

About `ornith-ai/Ornith-1.5-35B-A3B-FP8`

Offer

Branches

Problem 1: the detector does not accept `strategy: "channel"`

Problem 2: the loader cannot read a per-channel weight scale

Problem 3: an FP8 `lm_head` cannot load

Results

About `ornith-ai/Ornith-1.5-35B-A3B-FP8`

Offer

Branches

Commit Base Fixes Branch

Correction -- your checkpoint needs more than I said

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Feature Request] Support compressed-tensors per-channel FP8 checkpoints for Qwen3.5 MoE #252

Description

Summary

Model

Current behavior

Requested support

Possible incremental implementation

Likely affected areas

Suggested tests

Motivation

Activity

twinkle311 commented on Sep 3, 2026

yongjer commented on Sep 6, 2026

salekseev commented on Sep 8, 2026

Problem 1: the detector does not accept strategy: "channel"

Problem 2: the loader cannot read a per-channel weight scale

Problem 3: an FP8 lm_head cannot load

Results

About ornith-ai/Ornith-1.5-35B-A3B-FP8

Offer

Branches

yongjer commented on Sep 8, 2026

Problem 1: the detector does not accept strategy: "channel"

Problem 2: the loader cannot read a per-channel weight scale

Problem 3: an FP8 lm_head cannot load

Results

About ornith-ai/Ornith-1.5-35B-A3B-FP8

Offer

Branches

Commit Base Fixes Branch

salekseev commented on Sep 8, 2026

Correction -- your checkpoint needs more than I said

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Problem 1: the detector does not accept `strategy: "channel"`

Problem 3: an FP8 `lm_head` cannot load

About `ornith-ai/Ornith-1.5-35B-A3B-FP8`

Problem 1: the detector does not accept `strategy: "channel"`

Problem 3: an FP8 `lm_head` cannot load

About `ornith-ai/Ornith-1.5-35B-A3B-FP8`