[sergqwer]

Replaces #1050, which GitHub closed when main's history was rewritten. The closing note says this was not a rejection. It is the same commit, cherry-picked onto the new main (v0.1.40.1).

---

A chunk below stream_all_min() (1,024 tokens) streams every routed expert outside VRAM over PCIe. An agent's tool result, a test's output and a short follow-up are all such chunks. Meanwhile the decode CPU pool sits idle, and the arena holds the same experts in RAM.

This adds an opt-in, STRATA_PREFILL_CPU_SHARE=auto|x. Unset (the default), no expert moves and the output is byte-identical to v0.1.40.1.

What it does

• The CPU's experts. The pool takes the non-resident experts routed by at most MAXT of the chunk's tokens, fewest first. It runs them on a thread while the GPU streams and runs the rest.
• Their rows. The CPU's unweighted rows go to Dm's tail, as a peer GPU's do (the same mechanism as --peer-device), and the combine weights them as any other row.
• auto: the share is measured. Each layer times the CPU thread per expert, and the GPU's expert work per streamed expert. The GPU side uses CUDA events around its expert work, read after the next layer's routing sync, because the host reaches the combine long before the GPU does; timing the host instead gave a wrong share. The next layers hand the CPU g / (c + g) of the streamed experts, the point where both sides end together. A small CPU therefore takes a small share and does not make the prompt slower. x fixes a share.
• Activations. Q2_0 layers take the pool's Q2_0 kernels' activations (ActQ, as decode's), and i-quant layers take the native vec_dot type.
• Scope. Serve without batch slots, and generate; one GPU.

Measured

On another machine, by @did-technomancer (#1050): RX 6800 16 GB (gfx1030) + Ryzen 7 5700X3D (AVX2, no AVX-512), Windows 11, ROCm, IQ3_XXS, 128K context. Short prompts (~850 tokens) read 37-45% faster, and long prompts and decode are unchanged. Their quality suite gave 22/22 with the switch on.

engine	short prompts, tok/s	prompt 8K / 30K, tok/s
v0.1.39 + this PR, off	159-166	594 / 543
v0.1.39 + this PR, auto	233-241	593 / 544
v0.1.40 + this PR, off	176-186	739 / 761
v0.1.40 + this PR, auto	241-255	735 / 758

On 0.1.40, here (IQ2_XS, one chunk, ms, off / auto): 600 tokens 487 / 463, 489 / 457, 491 / 485 (share 0.86-0.87); 250 tokens 386 / 357, 386 / 352, 379 / 356 (share 0.90).

On 0.1.39, here. RTX 5090 + Ryzen 9 9950X3D, generate, one chunk, ms (off / auto):

model	600 tokens	250 tokens
UD-Q4_K_XL (every expert in RAM)	1,548 / 1,369, 1,528 / 1,296 (share 0.65)
Q2_0	472 / 445, 491 / 448 (0.90)
IQ2_XS	493 / 481, 502 / 460, 527 / 529 (0.83-0.86)	424 / 355, 402 / 382, 379 / 367
IQ2_XS, 2 AVX2 pool workers (emulated)	512 / 487 (0.49)

• Where it helps most. The gain grows with the experts' size. UD-Q4_K_XL's larger experts make a small chunk PCIe-bound. Q2_0's and IQ2_XS's ~1.2 MB experts mostly do not, so they move less.
• Accuracy. The CPU's rows are not bit-identical to the GPU's (another activation format), but they are as accurate.
  • IQ2_XS, first-token KL to STRATA_PREFILL_MMQ=0: 0.0053 off, 0.0080 auto.
  • UD-Q4_K_XL, KL between off and auto: 0.006.
  • On a fork with larger NVFP4 experts, layer 0's expert rows were 1.086% from the FP16 path on the CPU against 1.087% for the GPU's MMQ rows.
• Identity. Off, the logits and the 32 greedy tokens equal v0.1.40 on IQ2_XS after 600 and 32K tokens (same expert cache). 0.1.40.1 changed only the server and setup, and the engine's sources here are the same bytes as on that branch.
• Not changed. stream_all_min() stays 1,024. On the fork with NVFP4's 2.76 MB experts the routed-only walk paid up to 4,096 tokens, but on Q2_0 it lost against streaming every expert, so it stays out of this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

[jase100k]

Measured on RX 7900 GRE + Ryzen 7 5700X3D: short prompts get slower, not faster

Ryzen 7 5700X3D (the same CPU as the #1050 measurement) but a much faster GPU: RX 7900 GRE (gfx1100), NixOS, ROCm 10.2 nightly, Qwen3.8-Flash-Next Coder IQ1_M, generate single GPU (so the pool is wired), --prefill auto. n=3 per prefill arm; run-to-run spread within an arm was <0.5 tok/s.

Short prompt (849 tokens, one chunk below stream_all_min()), prefill tok/s:

share	tok/s (3 runs)	vs off	experts on CPU
off (default)	320.1 / 319.7 / 320.4	—	0
0.10	313.3 / 313.5 / 313.4	−2.1%	783
0.20	309.6 / 308.6 / 308.4	−3.5%	1556–1558
auto	305.8 / 305.8 / 305.8	−4.5%	2487–2491 (share 0.46–0.51)

Every fixed share and the measured one is slower than off, monotonically in the share — so this is not just auto picking a bad point; the CPU path itself loses on this box. The sign is the opposite of the +37–45% in #1050.

Engagement / diagnosis. The flag engaged as designed (on the CPU 2487 (share 0.47) of 7789 streamed experts). One number stands out: with auto, host expert time rose 832.6 → 1060.8 ms even though streamed experts fell 7789 → 5315 — the CPU expert thread appears to contend with the page-locked streaming path on the host rather than overlap it. Also, auto chose ~0.47 (near CPU/GPU parity), while the timing model's own logic (g / (c + g)) should have predicted a much smaller share on a GPU this much faster than the CPU; the per-layer CUDA-event timing seems to overstate the CPU side's competitiveness once that contention exists.

Unchanged, as claimed:

• Long prompt (3000 tokens): 634.1 off vs 632.6 auto — gate correctly idle (share 0.00).
• Decode (512 tokens): unchanged. Caveat on methodology: this build wanders between output groups run-to-run at fixed settings (the documented adaptive-tier / IQ multi-token rounding issue, Native CPU expert results depend on verifier width (plain/MTP divergence) #152 — DETAILS "Reproducible greedy output"), and decode speed tracks the output group (39.4–46.0 tok/s) in both arms; the flag's arms show the same mix of groups.
• Greedy first token identical off vs on (279).

Why the sign flip: #1050's machine is an RX 6800 (gfx1030) — a far slower GPU, so g per streamed expert is large and CPU offload has room to pay. Here the GPU finishes expert work quickly and the CPU (same 5700X3D) becomes the straggler and adds host contention. Suggests the win is GPU-speed-dependent, and/or the share measurement should account for contention with the streaming path.

Scope note: with layer_split / multi-GPU serving, set_cpu_pool is never called (!multi_gpu gate), so none of this applies to 2-GPU configs — measured in single-GPU generate mode.

[sergqwer]

@jase100k thank you - your host-time numbers explained it. Per streamed expert the host went from 107 us (off) to 200 us (auto, 0.47): with the CPU busy, the streaming side kept 0.53 of the experts at 1.87x the cost each (0.99), so it saved nothing and the CPU's work came on top. auto balanced c and g measured with the share on and never compared a layer without it.

#1379 makes auto time adjacent layers with and without the share (CUDA events from the routing sync to the combine) and keep sharing only while that is faster. Here it keeps ~90% of the gain where sharing wins and caps the loss where it does not (one core + one worker: up to +11.5% before, +2.0..+2.3% after - the cost of the first few measured layers). I could reproduce the rise of the host's time per expert (+86% with an emulated host-side copy) but not a loss of your size: this CPU still wins. If you can run your 849-token prompt with #1379 and STRATA_PREFILL_CPU_SHARE=auto (STRATA_DBG_CPU_GATE=1 prints each decision), that is the measurement that settles it.

[sergqwer]

Landed in v0.1.40.2 as 978d355 (opt-in, with 667f2ec and 4322e24 on top) - thank you. Closing this one.

The follow-up from jase100k's RX 7900 GRE numbers is #1379: auto times adjacent layers with and without the share and keeps sharing only while it is faster.

[Niko1221]

Thanks @sergqwer, and @did-technomancer for the RX 6800 numbers. This is in 0.1.41 with your authorship kept on the commit, and it is on by default on NVIDIA now: one GPU, no --batch slots, chunks below 1,024 tokens. STRATA_PREFILL_CPU_SHARE=0 turns it off and restores the exact 0.1.40.x answers; layer splits, batch slots and AMD are as before. The engine prints one line at start saying the share is on.

Final numbers, interleaved whole-engine runs, 8 rounds, medians, prompt time against the share off (8 of 8 pairs faster on each): 512 tokens -28 to -34% and 1,000 tokens -20 to -26% on an RTX 3060, a Tesla P100 and an RTX 5070; 2K and longer unchanged. The answers can differ slightly: first-token KL against off averages 0.004 (max 0.025).

Two things we changed on top: the share's thread no longer calls the expert source itself (its blob() counts reads without a lock, so it raced with the prefill thread's own calls), and STRATA_DBG_NAN no longer counts the CPU's rows of the GU/H buffers (the GPU never writes them). A 50-prompt mixed-length stress run with the share on had no errors, no NaN reports and no broken answers. #1379's per-layer timing (share only while it is faster) is in too.