[Feature] int4 routed experts: AWQ -> Q4_1 and compressed-tensors pack-quantized -> Q4_0 #396

Summary

FreeToken currently has no path for int4 routed experts: the quant formats in the tree are
bf16, fp8_block, fp8_pertensor, NVFP4 (modelopt and compressed-tensors) and GGUF. There is
no issue or PR for AWQ, compressed-tensors pack-quantized int4, or W4A16 int4 that I could
find.

That rules out a large share of the community 4-bit checkpoints for Qwen3.6-35B-A3B, and for
a 16 GB card the int4 MoE checkpoints are the ones that fit. I have both formats working
locally against af71ba432 and would be glad to upstream them if you want them -- opening
this first to check the approach rather than dropping an unsolicited PR.

The idea: no new kernel

Both formats are affine int4 at group_size 32, which is exactly the shape of a GGUF 32-wide
block, and the borrowed ggml MoE GEMV already dispatches both block types. So this is a
load-time repack plus the table entries a bank format needs -- no CUDA changes at all:

The compressed-tensors case is the nicer one and is exact, for two reasons that are easy to
miss:

• pack_to_int32 offsets before packing (value.to(torch.int32) + offset,
  offset = 1 << (num_bits-1)), so the stored nibble is already Q4_0's offset-binary code.
  The codes copy through with no arithmetic -- not even a XOR.
• bf16 scales widen into fp16 losslessly (8 mantissa bits into 11) inside fp16's exponent
  range, so d is exact too.

It is also free structurally: weight_packed is [N, K//8], already N-major like the expert
banks, and the packing is sequential. (AWQ is K-major with an 8-column interleave
(0,4,1,5,2,6,3,7) per word, so it needs a transpose and a de-interleave.)

Because Q4_0 is already a native bank format, the compressed-tensors path needs no new
_BANK_SCHEMAS entry, no dequant.py entry and no _expert_gemm dispatch branch -- only a
repack module, a config detector and the bank builder. The AWQ path additionally registers
q4_1.

Validation

Packing order is the part that cannot be reasoned into correctness, so both are checked
against the third-party library that wrote the checkpoint, with controls.

compressed-tensors -> Q4_0, oracle compressed_tensors.unpack_from_int32, 80 real expert
projections / 2,621,440 blocks of Avesed/Qwen3.6-35B-A3B-INT4-W4A16:

code mismatches vs compressed-tensors     : 0
max |repack - fp64 truth|, fp16-normal    : 0.000e+00
max |repack - fp64 truth|, fp16-subnormal : 0.000e+00
max |repack - fp64 truth|, flushed blocks : 1.909e-35
control, lo/hi nibbles swapped            : 80/80 detected as wrong
control, two's-complement (^8) codes      : 80/80 detected as wrong

AWQ -> Q4_1, oracle autoawq's unpack_awq / reverse_awq_order, 60 real projections of
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit: integer unpack bit-exact, reconstruction error equal
to autoawq's own fp16 rounding (2.14e-04), the residual being the single fp16 store of m.

Cross-check between them: the two checkpoints quantize the same base weights independently,
and dequantized expert projections correlate 0.982-0.992, while a deliberately-wrong code
reading scores −0.53.

There is one lossy case in the symmetric path, and I would want it reviewed: Q4_0 mandates an
fp16 scale, and a bf16 scale below fp16's smallest subnormal (~5.96e-8) flushes to zero,
zeroing that group of 32 weights. On this checkpoint that is 0.752% of ~1.0e9 blocks
(reported by the loader at load time), concentrated in layers 0-1, with a median flushed
scale of 8.2e-37 -- a dead region of the base model, not quantizer error. Clamping up to the
smallest representable value would invent nonzero weights, so zeroing looks like the correct
choice, but it deserves a maintainer's opinion.

Measured

RTX 4080 SUPER 16 GB, Qwen3.6-35B-A3B (40 layers, 256 experts, H=2048, I=512),
--moe-backend offload --moe-cache-auto --max-running-requests 1, engine af71ba432,
evalscope perf (1200-token prompt, 256 out, concurrency 1):

The slot ratio (1.119x) tracks the bytes-per-block ratio (20/18 = 1.111x), which is where the
decode gain comes from on a partly-resident expert cache. Prefill measured slightly slower
for Q4_0 (1428.8 vs 1497.9 prompt tok/s), which I have not explained and which is only n=8.

BFCL v4 parallel (200 entries, FC mode): 0.935 AWQ vs 0.910 CT int4, McNemar exact
two-sided p = 0.302 -- not separable, and note #307 means the harness is not reproducible with
thinking on, so I would not read a quality difference into that either way.

What a PR would touch

Per the extension contract in moe/expert_banks.py, plus the qwen3_5_moe override:

• models/ct_q4_0.py, models/awq_q4_1.py (new, repack only)
• models/qwen3_5_moe/config.py -- two detectors; disjoint from
  detect_compressed_tensors_nvfp4 by the weights' type field ("int" vs "float")
• models/qwen3_5_moe/weight.py -- two bank builders + branches in
  setup_offload_expert_banks
• moe/offload_cache.py -- _BANK_SCHEMAS["q4_1"], and _BANK_BYTES_PER_EXPERT entries so
  the pin-quota sizing works for both
• models/gguf/dequant.py -- GGML_Q4_1 metadata; moe/fused_q4_0.py -- format->ggml-type map
• layers/moe.py -- dispatch q4_1 alongside q4_0

Questions before I write it up properly:

1. Do you want both formats, or only compressed-tensors (the exact one)?
2. Naming: I used expert_quant tags "awq" and "ct_int4". Preference?
3. The Q4_0 banks are byte-identical to the native GGUF Q4_0 banks, so the CPU MoE executor
   reads them via _WFMT_IDS["q4_0"] and cpu/hybrid decode works. I left it enabled but
   untested -- would you rather it be gated until measured?
4. Both builders are TP=1 only. Acceptable for a first cut?