[bsvinay]

Resubmitted from #936, which GitHub closed when main's history was rewritten. Same 4 commits, cherry-picked onto the new main (no conflicts). The review discussion and test results from others are in #936.

On 2x RX 7900 XTX the layer split decoded slower than one card. The cause is the order inside every verify window: stage 1 finishes, the host syncs its stream, then stages and launches stage 2's graph. Under HIP both the sync and the graph launch are expensive, and they sit on the critical path of every window while both cards are idle.

With STRATA_SPLIT_OVERLAP=1 (off by default; stages on different devices only):

• each hand-off buffer gets a ready word (mapped pinned memory);
• the writing stage's graph ends with handoff_publish: the payload with volatile stores (RDNA keeps plain stores to mapped memory in L2, same reason as doorbell_ring_kernel), a system fence, and the last block raises the flag;
• the reading stage's graph starts with a wait on that flag;
• Verifier::run launches the next stage's window on a helper thread while it serves its own layers, and syncs the earlier stage only once the chain is done;
• release_gpu_waits also raises the hand-in flag, and an early return after the early launch releases the next stage (Windows: the verify-window stall wedges the GPU into an unrecoverable "device lost" state (manual power cycle required), with no TDR logged #267).

Changes since #936 (from the review there):

• the GPU-side hand-off wait is bounded: after STRATA_SPLIT_WAIT_MS (default 30000, 0 = no limit) it sets an error word and lets the graph finish; the host then fails that window with a clear error instead of using a stale hand-off;
• a test hook, STRATA_TEST_HANDOFF_DROP=N, leaves the flag down in the Nth window (the publish reads it only when the hook is set);
• the early launch keeps the Fresh prompts over --short-read decode !!!! (token 0) when the GPU holds 100% of the experts (48 GB card) #871 graph variant the next stage picked;
• the overlap switches itself off with --pipeline-windows (Layer split: pipelined verify windows for one conversation (--pipeline-windows, opt-in) #859), which overlaps the stages its own way.

Measured on 2x RX 7900 XTX (gfx1100, ROCm 10.0.0, Linux), Ryzen 9 9900X, 52 GB RAM, Swift 1.5 IQ3_XXS, 128K context, 8-bit KV, split 0-25 / 26-47, 512-token greedy answers:

	serial (default)	STRATA_SPLIT_OVERLAP=1
decode, code prompt	53 tok/s	69.6 tok/s
decode, prose prompt	45 tok/s	57.8 tok/s

• Greedy output byte-identical with the flag at 0 and 1.
• Fault test with the hook: with STRATA_SPLIT_WAIT_MS=2000 the request fails after ~2 s with the hand-off error (HTTP 400) and both GPUs are idle; with the default, the Windows: the verify-window stall wedges the GPU into an unrecoverable "device lost" state (manual power cycle required), with no TDR logged #267 host timeout fires at 20 s, the GPUs are idle and the server restarts the engine (503); with the overlap off the hook does nothing.
• From layer split: overlap the stages' hand-off (STRATA_SPLIT_OVERLAP=1, op… #936: +1.6% decode on 4x V100 (@justxiami), no change on 4x / 2x RTX 5080 (@blange48) and on RDNA4 (@aswin-dot-R, byte-identical output). So it mostly helps RDNA3 - that is why it stays opt-in.

[fioruccione]

Another data point, from NVIDIA with two identical Ada cards: no change, and nothing broke.

Rig: 2x RTX 4060 Ti 16 GB (sm_89), PCIe 4.0 x8 each, layer split auto (CUDA0 layers 0-22, CUDA1 23-47), Threadripper PRO 3975WX (AVX2), 8x DDR4-2666. Unsloth UD-IQ4_XS, 262K context, --kv int8 --kv-resident 32768 --spec 4 --mtp, experts partly on the CPU (~3 CPU experts per layer-window). Linux, CUDA 12.6.

Build and method: this PR (f6196d6, on main e8ca9af), one binary for both arms, built the way setup builds (sm_89). Arms switched only by STRATA_SPLIT_OVERLAP=0|1, interleaved A B A B with a fresh server start each, STRATA_DECODE_TIMING=1 in both. benchmark.py from the RTX 5090 report at 4K and 32K (3 runs, greedy, 256 tokens), plus 4 Italian chat prompts (greedy, 256 tokens).

The engine log of the B arms confirms the hand-off ran overlapped (overlapped: STRATA_SPLIT_OVERLAP=1).

	overlap off	overlap on
ms per verify window	42.4	42.5
GPU-reach wait, ms per window	13.2	13.7
decode tok/s, 4K	56.2	56.0
decode tok/s, 32K	57.3	56.6
decode tok/s, Italian prompts	53.0	54.7

Every difference is within the run-to-run spread of the same arm, so the overlap makes no measurable difference here. That matches the 2x and 4x RTX 5080 reports: on these NVIDIA cards the host sync and graph launch between stages are already cheap compared with the window. No errors or stalls in any arm. I did not check byte-identical output: greedy runs already differ from run to run on this rig (GPU/CPU rounding), so a token compare would not tell the two arms apart.

Measured on my machine, with an AI coding assistant helping on the scripts and the write-up.

[bsvinay]

Rebased onto 0.1.44.1 (296b59d). Only the first commit changed: upstream's
HostGraphLaunch in verify.cpp's launch block now sits inside the if (!pre)
branch (held in a std::optional; a prelaunched window never creates it and falls
back to cudaStreamQuery / no-op). The other three commits are unchanged.

What it changes and leaves alone: STRATA_SPLIT_OVERLAP=1 is still opt-in and the
default path is untouched. A prelaunched next stage uses a plain cudaGraphLaunch,
so STRATA_VERIFY_ASYNC_LAUNCH=1 (non-HIP, opt-in) applies to the first stage only.

Measured on 2x RX 7900 XTX (gfx1100, ROCm 7.15, Linux), Ryzen 9 9900X, Swift 1.5
IQ3_XXS, with --pcie-frac 0 --adapt-every 1; 3 alternating rounds, each the median
of 5 runs of a 512-token greedy answer:

• code decode 61.1 -> 76.7 tok/s (+25%)
• prose decode 50.2 -> 61.6 tok/s (+23%)
• prefill 4K / 16K / 32K unchanged (about 1,445 / 2,220 / 2,790 tok/s, within 1%)
• quiz set 47/51 with and without it.
  One card: the rebased build equals the previous one (decode 95.8 / 76.5 vs
  95.5 / 77.2, prefill within 0.2%, quiz 47/51 on both).

Not tested: CUDA (the HIP build only; the CUDA branch of the launch block was merged
by hand and has not been compiled), SYCL, NVIDIA hardware.

[qiuzan2001]

The default path looks untouched (no env var, no hand-off flags). One build issue, from reading (no Mac here): Verifier::record_window in src/core/verify.cpp now calls wait_handoff (line 964) and handoff_publish (line 1757), defined only in src/kernels/cuda/verify_kernels.cu. The Mac engine compiles src/core/verify.cpp (metal/CMakeLists.txt:94) and resolves such kernel entry points through metal/src/engine/stubs.cpp, which stubs the neighbouring wait_flag_ge but has no wait_handoff / handoff_publish, so the Mac link should fail on these two symbols. Adding stubs (or regenerating with metal/tools/gen_stubs.py) would fix it.

[Niko1221]

Thanks, and thanks to the people who tested it on other cards. Ported with your authorship; one conflict was ours to resolve (0.1.44 moved the graph launch into a HostGraphLaunch object, so the early-launched window keeps no launcher). On 2x RTX A4000 the layer split with the overlap on is neutral (x1.001 over 5 pairs) and the answers are bit-identical (4/4 slots, exact settings); the default path is unchanged (identity gate). We have no RDNA3 two-card box, so the gain you measured on 2x 7900 XTX is not reproduced here; it stays opt-in and the docs name it for RDNA3.

[bsvinay]

Thanks both. Ran the interleaved comparison Niko asked for in #936, on v0.1.45 (main, 477ac13), 2× RX 7900 XTX (gfx1100, ROCm 10.1), Swift 1.5 IQ3_XXS, layer split auto.

Method: 10 rounds, four arms, run order rotated every round so each arm is first/second/third/last equally often, 200 tokens, 3 runs per cell, median per round, then median across rounds.

arm	code tok/s	vs serial	prose tok/s	vs serial
S serial (no flags)	65.1	1.00	54.7	1.00
O STRATA_SPLIT_OVERLAP=1	80.9	×1.24	67.4	×1.23
P --pipeline-windows 2	105.8	×1.63	86.1	×1.57
M P + STRATA_MTP_STAGE0=1	91.8	×1.41	69.2	×1.27

• Overlap reproduces my earlier gain under interleaving (×1.24 / ×1.23, range 80.6–81.0 over 10 rounds).
• Pipelined windows give a larger gain than overlap on this pair: P/O is ×1.31 (code) and ×1.28 (prose), and P is ahead of O in all 10 rounds.
• STRATA_MTP_STAGE0=1, which the engine's note recommends, was slower than plain P here and noisier (prose 60.6–72.8).
• P is noisier on prose than O (77.5–91.0), but its worst round still beats O's best.
• Cost check: prefill with P vs serial is equal within noise (4K: 1528/1510 vs 1514/1515; 32K: 2834/2836 vs 2815/2796 tok/s), and the 51-question quiz scores 47/51 on both (two borderline trap answers swapped, so outputs are not identical).
• For reference, the same model on one card (v0.1.45, same prompts, 8 runs, not interleaved with the above) does 102.9 / 81.5 tok/s, so on this model the pipelined split is about equal to a single card; the layer split mainly pays off for models that don't fit on one card.

So --pipeline-windows 2 covers this case and does better on this pair. I'm happy for this PR to be closed as covered. Suggest the MULTI_GPU docs name --pipeline-windows 2 for gfx1100 two-card splits (it is on by default only for gfx12), with these numbers.

Mac/Metal link failure (@qiuzan2001): a small separate PR is coming with stubs for wait_handoff and handoff_publish, may need help with testing this.

[bsvinay]

@qiuzan2001 , can you please try #3002 for the stubs. Thanks

[Niko1221]

Thanks for the pull request.

Your change is already in main for 0.2.0-beta, with your credit, so there is nothing left for you to do here. We're closing it because we're resetting the tracker for 0.2.0-beta: there are more than 1,300 open issues and pull requests, and we're closing everything that isn't being worked on right now.

If part of your work did not make it in and you want it considered, open a new pull request on top of main that follows CONTRIBUTING.md, or comment /reopen after fixing the title area prefix and the template.