[gdevenyi]

Under FREETOKEN_MOE_COPY_OVERLAP=1 (default off), a qwen4_exp GPU decode layer (TP=1) runs its shared expert and shared-expert gate on a dedicated side stream. They fork before the router and join before the combine. The router, fused top-k, ensure, miss copy and routed GEMM stay back to back on the compute stream. Flag off launches exactly the old sequence.

Why it has this shape: the first version moved the miss copy to the side stream, under the shared expert. An nsys profile of the real decode graph showed that copy starting about 9 µs after its ensure, queued behind the shared expert's launches, which ate half the overlap: +1.3–1.6% measured, against +3.6% modelled.

• Inverting the streams keeps the copy right behind its ensure.
• The side branch has about 100 µs of slack, so it absorbs the fork latency and the SM contention. Beside the 1024-thread copy CTAs, act_and_mul stretches 12× there, harmlessly.
• The side stream is created with cudaStreamCreateWithFlags and wrapped in an ExternalStream. A pooled torch.cuda.Stream() can come back as the compute or capture stream, which silently serializes the side work. This happened in the full test run, and test_decode_side_stream_never_aliases_a_pooled_stream covers it.

Measured on the RTX 5080 (Qwen3.8-Flash-Next-NVFP4, 1650 slots, bs=1), interleaved, 2 rounds each, decode tok/s at 32K / 128K:

Arm	32K	128K
lru	40.14	39.67
lru + first design (copy on the side stream)	40.79	40.31 (+1.6%)
rule eviction	42.45	41.80
rule + this design	43.35	42.66 (+2.1% over rule; +7.8% over lru)

The lru and first-design rows come from the first A/B session. The rule rows come from a second session, in which rule reproduced exactly (42.45 / 41.80).

Greedy outputs are token-identical to production on 3 fixed prompts, and quality.py scores 14/15.

Tests:

• Flag on vs off is bitwise identical (outputs, slot map, slot contents) for bf16 and NVFP4, bs 1 and 2, eager and CUDA graph. The test also checks the copy stays on the compute stream and the shared expert runs on the side stream.
• decode_side_applies declines for prefill, CPU/hybrid decode and TP>1.
• The side stream survives a runtime rebuild.
• The side stream never aliases a pooled stream.

benchmarks/bench_moe_copy_overlap.py is a synthetic 4-layer bench. It measures a saving of ~10 µs per layer, a median copy start slip of 2.4 µs, and a shared expert that ends inside the copy in 100% of layers.

VRAM (same code, a 255K prompt plus 3 side calls, sampled every 100 ms):

Config	Idle	255K peak
lru	14,592 MiB	15,544 MiB
rule	14,592 MiB	15,544 MiB
rule + overlap	14,616 MiB	15,568 MiB

The overlap costs a fixed +24 MiB: the side branch's graph nodes and its stream's pool. That is about 9 expert slots, and 314 MiB of the ~15,882 usable remains free.

Deployed on 5080deploy (with --moe-cache-policy rule):

• A 15-minute agentic soak passed (50 turns, 42 side calls, worst token gap 14.5 s).
• A 255K prompt with 3 concurrent side calls decodes at 43.0 tok/s with 1.7 s side-call latency.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt