[jason-fxz]

Fixes #747 (thanks @SUKUNA-AI for the analysis).

Qwen3.5 / Qwen3.6 / Qwen3.8-27B (the qwen3_5_moe family, dense variants included) store zero-centred RMSNorm weights: the multiplier is 1 + w, which transformers computes as output * (1.0 + weight.float()), as do vLLM (GemmaRMSNorm) and SGLang (gemma_rmsnorm / gemma_fused_add_rmsnorm). FreeToken folded the 1 into the bf16 weight at load time, so it multiplied by bf16(1 + w). This PR keeps w and adds the 1 in fp32 inside the norm kernel, for the input / post-attention / q / k / final norms. The GDN gated norm (linear_attn.norm) is a plain w * x norm and is unchanged.

Changes

• models/qwen3_5_moe/weight.py: the safetensors loader yields the norm weights as stored (drops the + 1.0 fold).
• models/qwen3_5_moe/gguf.py: llama.cpp's converter stores these norms as F32 w + 1; the GGUF loader now subtracts the 1 in fp32, as the qwen4exp GGUF loader already does. Without this the GGUF path would multiply by 2 + w. The GGUF path had the same rounding before: its F32 1 + w was cast to the bf16 norm parameter.
• models/qwen3_5_moe/model.py, attention.py: the norms are the existing GemmaPlusOneRMSNormFused / GemmaPlusOneRMSNorm (flashinfer gemma_rmsnorm / gemma_fused_add_rmsnorm, Triton fallback), the classes Qwen3.8-Flash-Next, Muse-Glimmer and MiniMax-M3 use.
• docs/models.md, docs/ftw-hotfix.md: old FTWs of this family must be reconverted.

Breaking: an FTW of Qwen3.5 / Qwen3.6 / Qwen3.8-27B converted by an older build stores the norms with the 1 already added. It loads without an error but gives wrong output (the multiplier becomes 2 + w); reconvert it with ft checkpoint. A converted FTW cannot be told apart by its values (the zero-centred weights of one checkpoint have per-tensor means from -0.27 to 1.63), so it is not detected.

Results

Weights (every input / post-attention / q / k / final norm tensor): share of multipliers where bf16(1 + w) != 1 + w, and the relative error of the folded value:

checkpoint	multipliers inexact	rel. error mean / max
Qwen3.6-35B-A3B (bf16 and NVFP4)	77.4%	1.38e-3 / 3.89e-3
Qwen3.6-27B (bf16 and NVFP4)	88.9%	1.52e-3 / 3.89e-3
Qwen3.8-27B-NVFP4	88.7%	1.51e-3 / 3.89e-3
Qwen3.6-35B-A3B GGUF (F32 1 + w cast to bf16)	77.1%	1.38e-3 / 3.89e-3

After the fix the stored value is w itself: the bf16 checkpoint value, and on GGUF (1 + w) - 1 in fp32, which equals the bf16 checkpoint w bitwise (checked on Qwen3.6-35B-A3B's output / attention / q norms).

Norm op against an fp32 reference of x * rsqrt(mean(x^2) + eps) * (1 + w) (bf16 inputs, H = 2048 and 5120, per-head q norm D = 256): the new norms give the correctly rounded bf16 result on every element (plain, fused add, Triton fallback, per-head); the folded weight missed on about 25% of the elements, by up to 3.1e-2. Same speed as the sgl_kernel norms they replace (CUDA graph): decode fused add 1.35 vs 1.37 us, 8192 x 2048 prefill fused add 23.8 vs 23.9 us, q norm over 8192 rows x 24 heads 130 vs 128 us.

Model, Qwen3.6-27B bf16, last-position logits of 14 prompts (64 - 480 tokens) against transformers 5.18 bf16, RTX PRO 6000:

	KL(transformers || FreeToken) mean	median	max	top-1 agreement
main (0db5023)	1.76e-4	4.6e-6	6.8e-4	14 / 14
this PR	1.59e-4	3.8e-6	1.17e-3	14 / 14

The PR is closer to transformers on 10 of the 14 prompts. The gain is small: KL(main || PR) is 2.6e-4, about the distance of either to transformers, so other bf16 differences in the stack dominate. No PPL or eval was run.

Serving (greedy, 96 tokens): Qwen3.5-0.8B GGUF, Qwen3.6-35B-A3B NVFP4 (GPU offload) and Qwen3.6-35B-A3B GGUF (IQ3_XXS) give coherent text with the same opening as main.

Tests

pytest tests/models/test_qwen3_5_moe_weight.py tests/models/test_quant_config.py tests/models/test_models_registry.py tests/checkpoint tests/models/qwen4_exp tests/layers on the RTX PRO 6000: 307 passed, 150 skipped, 5 failed; the 5 (tests/models/test_quant_config.py, from the local HF cache) also fail on main. The loader test now expects the norm weights as stored.