[gdevenyi]

Stacked on #71/#72/#73 (branch base 5080deploy @ 6599710). Expert prefetch for the offloaded-MoE decode, behind FREETOKEN_MOE_PREFETCH=off|measure|on (default off), plus two debug detectors that were needed to make it lossless.

What. While layer L-1 computes, predict layer L's routed experts and copy them over the otherwise-idle PCIe link. A decode step leaves the link idle about half the time, ~225 µs per layer in nsys.

• Predictor stream: runs layer L's router on layer L-1's router input (training-free lookahead), with no cache reads or writes. It is forked at the start of layer L-1's MoE block.
• Compute stream: right after ensure(L-1), prefetch_ensure(L) installs the first budget non-resident candidates into low-priority slots: held with usage 0, keyed between empty slots and every real resident, so a wrong guess is the next victim. It writes its own copy plan.
• Highest-priority copy stream: copies that plan with the slim copy kernel (5080deploy: opt-in slim expert copy kernel for copies beside compute (FREETOKEN_MOE_SLIM_COPY, off) #73) after the demand copy(L-1).
• Joins: the compute stream waits for the prefetch copy before copy_missing(L). A demand hit on a prefetched slot clears its low-priority mark.
• Settings:
  • FREETOKEN_MOE_PREFETCH_BUDGET: default 3 before GDN layers and 4 before attention layers; 4 is the measured best.
  • FREETOKEN_MOE_PREFETCH_K: candidates, default 16.
  • measure: counts useful predictions without copying.

Single writer. Every write to the slot maps (slot_for_id, id_of_slot, usage, eviction mirrors) happens on the compute stream in program order. Only reads and byte copies cross streams. An earlier version ran prefetch_ensure on the predictor stream behind an event join. On the real model it corrupted greedy output in rare runs:

• 1/60 cold prompts at budget 4
• up to 6/60 under the verifier's timing

A CPU model of both kernels proved the recorded bad state is reachable only if the two map writers overlap. nsys showed no violation in 400k samples, so the join failed only in some rare execution (first decode after a prefill). Rather than depend on that edge, the maps now have one writer.

Measured. RTX 5080, Qwen3.8-Flash-Next-NVFP4, 1650 slots, rule eviction + copy overlap (#71/#72), bs=1, interleaved ABAB, decode tok/s at 32K / 128K:

Arm	32K	128K	vs off	demand misses/token
off	43.36	42.67	—	206
on, default budget	51.32	50.45	+18.3%	123
on, budget 4	53.02	52.12	+22.2%	111

Measure mode on the real model: useful predictions 1.73–1.84 per layer at precision ~0.55, covering ~42% of demand misses.

Losslessness evidence on the real model (greedy, 256-token outputs, cold nonce-prefixed prompts, compared prompt by prompt with a prefetch-off reference that is identical across three server restarts):

• default budget: 0/60 mismatches
• budget 4: 0/60
• budget 4 + FREETOKEN_MOE_PREFETCH_VERIFY=meta: 0/120, 0 verifier records. This is the config that amplified the old bug to 2–6/60.
• budget 4 + FREETOKEN_MOE_SLOT_AUDIT=64: 0/120; 444 full-cache audits byte-checked every held slot, 0 bad.
• Greedy fingerprints are identical to production on 3 fixed prompts.

Debug tools (default off, zero cost when off):

• FREETOKEN_MOE_PREFETCH_VERIFY=meta|1 checks every routed slot's maps (meta), plus its bytes before and after each expert GEMM (1).
• FREETOKEN_MOE_SLOT_AUDIT=N records the last writers of every slot and byte-checks the whole cache every N decode steps.
• Both have fault-injection tests (missing join, corrupt bytes, stale map, write during GEMM, skipped copy).

Tests: 75+ new tests, including:

• bit-identity of on vs off over 200 steps (eager and graph, bs 1 and 2, rule and lru, overlap on and off, bf16 and nvfp4)
• delay-kernel races on every stream, garbage predictors and prefills in between
• low-priority band arithmetic
• the single-writer structural test
• eviction of a slot whose prefetch copy is in flight, with a negative control

Full not-slow suite: only the 4 known environment failures.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

[gdevenyi]

Deployed on 5080deploy (8ba20f6) with FREETOKEN_MOE_PREFETCH=on FREETOKEN_MOE_PREFETCH_BUDGET=4, on top of rule eviction and copy overlap. Validated on the production launcher:

• Greedy output: the fingerprints on 3 fixed prompts are identical to the pre-prefetch production output.
• Quality: quality.py scores 14/15 (code2 is the known model miss); needle 32K and 128K both pass.
• 255K prompt with 3 concurrent side calls: decode runs at 50.8 tok/s. It was 43.0 with rule + overlap and 38.4 with the original LRU. Side calls take about 1.7 s.
• Peak VRAM: 15,716 MiB against about 15,882 usable, i.e. +148 MiB over rule + overlap. That covers the predictor stream's cuBLAS workspace, the plan buffers and the extra graph branches, and leaves about 166 MiB of headroom.
• 15-minute agentic soak with side calls: PASS. It ran 57 main turns (50 without prefetch), 41 side calls and 593 tool calls, with a worst token gap of 14.3 s (a long-turn prefill).

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt