[timkhronos]

Overview

Adds the NextN draft head for GLM-5.3-Flash

Builds on #27773, which does convert the Nextn tensors, but leaves them unused.

Additional information

Enable --spec-type draft-mtp for glm5-next.

Also implements index_share_for_mtp_iteration which is true in the GLM 5.3 Flash config. The draft head computes its indexer selection once, then reuses it across each call, instead of rebuilding.

Measured draft acceptance rate with the layer quantized to Q4 (I will also test with Q8):

draft acceptance = 0.74485 ( 3474 accepted /  4664 generated), mean len =  4.13
acc per pos = (0.924, 0.813, 0.600, 0.441, 0.348)

On my heavily CPU offloaded setup, with draft max 3 I am getting roughly 15-30% speedup with MTP enabled.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, AI was used in an assistive capacity.

[borakilo]

Spent a few hours benchmarking this branch on a 4-GPU box and figured the numbers might be useful. Thanks for the MTP work — the long-context decode fix in d2fc716 made a big difference.

Benchmarks

4× 48GB Ampere, -sm layer -fa on, UD-IQ4_XS:

depth	pp512	tg128
0	505	25.0
32k	311	22.3
128k	146	19.6

Cold prefill on an actual 45k prompt comes out around 492 t/s. The -d numbers are marginal cost at full depth, so they read much lower than what you see prefilling a growing context.

For comparison, the same benchmark on #27773 and on Unsloth's glm5next/upstream gives 13.8–14.0 t/s at 32k versus your 22.3. Prefill is identical across all three, so d2fc716 looks decode-specific.

MTP came out slower for me, which surprised me. With --spec-type draft-mtp: 19.0 t/s at --spec-draft-n-max 1 (77.6% acceptance) and 16.4 at 3 (58.5%), against 25.0 with no speculation. Everything was in VRAM, so I assume the draft overhead only pays for itself when each target pass is expensive enough — which would match your CPU-offload setup.

One other thing worth knowing: --override-kv glm5next.attention.indexer.top_k=int:1024 gets about +7.5% decode with no measurable prefill change. Fits the idea that prefill is dominated by the indexer scoring pass rather than attention over the selected keys.

Investigation and write-up done with Claude Opus 5.

[fairydreaming]

@timkhronos just a heads up that after merging Hy4-preview support global tensor map now contains HC tensors (same that you wanted to add), so you can add your maping entries there, also looks like there are some other conflicts to solve (in the MTP PR).

[AIalliAI]

Cross-ref: #27773 / timkhronos#11 (k-pool softmax gridDim.y fix at 262k context).

CC @timkhronos

[TheBlueMatt]

Needs rebase now that glm5-next landed.

[amaze28]

Crash report: abort in draft-mtp path on long prompts (~18K tokens) — ROCm/gfx906, 5× MI50

Thanks for this PR — short-form results are excellent on my hardware (detailed below). Sharing a reproducible crash in the hope it helps.

Environment: llama.cpp built from this PR head (HIP backend, -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx906, ROCm 7.2.1 toolchain), 5× AMD Instinct MI50 32GB, tensor-split 1,1,1,1,1. Model: unsloth/GLM-5.3-Flash-GGUF UD-IQ3_XXS (contains blk.45.nextn.* tensors). Server flags: -ngl 999 -fa on -ctk q8_0 -ctv q8_0 --spec-type draft-mtp --spec-draft-n-max 2.

What works: short prompts + 256-token generations are stable and fast. Measured vs. 11.7 t/s undrafted baseline:

• n-max=1: 13.50 t/s (acceptance 0.809)
• n-max=2: 14.93 t/s (acceptance 0.710, mean len 2.42)
• n-max=3: 13.73 t/s (acceptance 0.581, mean len 2.74)

What crashes: any request with a long prompt (~18K tokens in our repro) aborts the server during decode, at both -c 196608 and -c 245760. Same model/flags without --spec-type draft-mtp handles the identical 18K-token prompt fine (needle retrieval correct, server healthy after), so the trigger is specific to the MTP draft path.

Stack trace (tail):

libllama.so.0(llama_context::decode(llama_batch const&)+0x1db0)
libllama.so.0(llama_decode+0xf)
libllama-common.so.0(common_speculative_impl_draft_mtp::process(llama_batch const&)+0x454)
libllama-common.so.0(common_speculative_process(common_speculative*, llama_batch const&)+0x38)
libllama-server-impl.so(server_context_impl::decode(...)...)
Aborted (core dumped)

Repro: build PR head for HIP/gfx906 → llama-server with the flags above → send a chat completion whose prompt is ≥ ~16-18K tokens. Happy to run patched builds or provide fuller logs/core details if useful — this is a standing 5×MI50 test rig.