Repository navigation
MTP for qwen4_exp (Qwen3.8-Flash-Next): is the draft/verify path planned beyond DSV4? #421
Description
The roadmap (#79) lists speculative decoding including MTP, and #69 lays out an exact DSpark draft/verify implementation for DeepSeek-V4. Both are DSV4-scoped, so I could not tell from either whether qwen4_exp is in scope.
Qwen3.8-Flash-Next ships its own MTP head — mtp.layers.0.mlp.experts.*, 128x128 block-FP8, ~3.1k tensors — and models/qwen4_exp/weight.py currently drops it outright:
Dropped: mtp.* (speculative head, including its stacked mtp.layers.0.mlp.experts.*) and model.visual.* (served text-only)
Two questions:
- Is MTP for
qwen4_expplanned, and roughly where does it sit relative to the DSV4 work in [2/3] feat(dsv4): implement exact DSpark speculative decoding #69? - Will the draft/verify machinery from [2/3] feat(dsv4): implement exact DSpark speculative decoding #69 be generalised into a shared abstraction, or is DSpark expected to stay DSV4-specific with a separate implementation per architecture?
Asking because the payoff looks larger on the offload path than on a resident one. On an RTX 3090 (PCIe 3.0 x16, 104 GB host RAM) serving RadixArk/Qwen3.8-Flash-Next-NVFP4, decode runs at ~19 tok/s while prefill reaches ~910 tok/s on a 39k prompt — the gap is expert transfer per decode step, which is exactly what verifying gamma tokens in one pass would amortise.
Happy to test a branch on that setup if it helps.
Activity
gdevenyi commentedon Sep 11, 2026
Adding a second data point for the "payoff looks larger on the offload path" argument, from the same model on different hardware — and confirming the dropped-weights part concretely.
The head is present and is a third of the checkpoint. On
nvidia/Qwen3.8-Flash-Next-NVFP4:The shard the PLE table lives in is even named
model-fp8-mtp-ple.safetensors.models/qwen4_exp/weight.pydrops every one of them, so this is weight we pay to download and store and never load.Why the offload path is the right place to argue this. We serve the same model on 2 × RTX 6000 Ada at TP=2, offload MoE, fp8 KV — a much bigger machine than your 3090, and the shape of the bottleneck is identical. Measured this week:
Single-stream is 105 tok/s while eight concurrent streams reach 355 — 3.4× the throughput for 8× the work. That gap is the tell: a decode step is dominated by pulling non-resident experts over PCIe, and the cost is per step, not per token. Batching amortises it, which is why concurrency scales so well and why a single stream cannot.
Speculative decoding amortises exactly the same cost without needing concurrent users, because verifying γ tokens in one pass is one expert pull instead of γ. On a resident-weights deployment MTP buys you arithmetic; here it would buy the transfer, which is the part that actually costs.
I ran the obvious upper-bound check on whether more residency alone fixes this, and it does not. Upstream #408 (nvfp4 KV) frees 5.25 GiB/rank and lifts residency 71.7% → 88.2%, cutting expert fetch traffic ~58%. Decode throughput did not move at all — 105.5 against 105.6 — because the dequant cost on sm_89 eats the saving. So on this architecture the transfer cost is not something you can buy your way out of with a smaller KV cache; amortising it across tokens looks like the remaining lever.
Offer: we have the checkpoint resident with its MTP tensors on disk, a TP=2 box, and a harness that does settled throughput plus GSM8K-300. If the draft/verify work does get generalised past DSV4, I can test a branch at TP=2 and report tok/s and accuracy against the numbers above. Two cautions from our own measuring, if it helps whoever implements it: benchmarks on this box read 2–3% low inside ten minutes of start while the expert cache fills, and GSM8K at the default 768-token cap measures truncation as much as reasoning — 9 of our 10 truncated answers became correct at 1536.
🤖 Generated with Claude Code
https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt
vstandos commentedon Sep 11, 2026
Third data point, from an RTX 5090 box on the Windows-native path (FreeToken Desktop beta, engine
0.1.2+g141c31a8d), same model family —Qwen3.8-Flash-Next-NVFP4,--moe-backend auto --max-running-requests 4 --memory-ratio 0.85.Hardware profile. 1×RTX 5090, PCIe verified x16/x16 under load (nvidia-smi sampled at 94–99% util across 8 reads, width 16 every time), 189 GB host RAM.
ft bench bwprofile (regenerated after a memory re-tune) reports ceilingspcie_linear_h2d 56.06 / d2h 56.81 GB/swith CPU stream read ≈ PCIe (ratio 0.94–1.04), and the auto policy correctly picks offload for every dtype — hybrid buys nothing when PCIe ≈ CPU bandwidth. So this box is the pure-PCIe-transfer regime @3250.Measured (client-side, stream-timed, no server metrics):
--moe-prefill-hit-d2dnow enabled (post-bounce A/B still pending)One negative result that speaks to the transfer story:
--ple-backend pinned(the 47.7 GiB n-gram table into page-locked host RAM) produced no decode or prefill gain on this box, while costing a much slower engine start and ~48 GB of locked RAM — the disk backend already rides the page cache well. Consistent with @gdevenyi 's residency experiment: on this architecture you cannot buy your way out of the per-step expert transfer; the only lever left is amortizing it across tokens.Speculative verify of γ tokens in one expert pull is exactly the shape of win this config is starving for — decode is at the published ceiling while prefill runs 20× faster two millimeters away in the same chip.
Offer: this box is Windows-native (not WSL, not Linux), engine+model fully reproducible (NVFP4 checkpoint on NVMe, 189 GB RAM, single 5090). Happy to run a branch when a qwen4_exp draft/verify path exists and report tok/s + GSM8K against the numbers above. Two cautions from our measuring: the first minutes after engine start read a few % low while the expert cache fills, and a pinned-then-dropped PLE experiment can silently skew comparisons — regenerate
ft bench bwafter any host-memory change (we re-tuned RAM mid-week and the auto strategy flipped dtype tables).lukascechovic commentedon Sep 15, 2026
Fourth data point, and a different one: I built it, and it works. There's a
qwen4_expdraft head running on a downstream branch — https://github.com/lukascechovic/FreeToken/tree/rocm-gfx1201-mtp — on 2 x Radeon AI PRO R9700 (gfx1201, ROCm 7.2.3), at TP=2 and TP=1. Themtp.*tensorsweight.pydrops get rebuilt into a ~1.4 GB expert bank by a script on the branch, so nothing binary is published.TP=2: +12.2% decode, and that is a floor. Two measured quantities: the cost of one speculating step is 1.618 plain steps, which puts break-even acceptance at 0.618; and a census of the live row's own acceptance over a couple of hours of ordinary use — 6 requests, 21,303 checked draft steps — reads 0.6363 to 0.9363, pooled by token 0.8153. That gives +12.2%, and my real traffic sits at or above that pooled figure, so the top of the range (~+20%) is the realistic end rather than the optimistic one.
I want to be precise about what kind of number that is: it's step-cost x acceptance, both measured on matched arms, not an A/B of served traffic. The one direct decode A/B I have was run on a prose corpus at acceptance 0.61 and reads −1%, which is exactly what the same arithmetic predicts there. The model and the measurement agree; what moves is the acceptance rate, and a prose benchmark corpus understates what real traffic does.
The caution, which is aimed at the single-GPU boxes in this thread. The draft head's expert bank has to live in host RAM. At TP=2 each rank pins half of it. At TP=1 one process pins the whole thing, and on a 122 GB host that displaced my PLE table off pinned memory onto the disk backend — about 27% of decode — which ate more than speculation gave back. So before sizing the win on a one-GPU offload box, check what the bank displaces. @vstandos, given
--ple-backend pinnedbought you nothing on the 5090, this may simply not apply to you — but it's the failure mode I'd look for first.Two things I don't claim: anything above batch size 1, since my row admits one request at a time — which means the per-step amortisation argument in this thread is untested exactly where it should be strongest. And bit-identical output: one differing generation in 602 against the head off, which I traced to bf16 rounding rather than a defect.