prefill: let Stager threads sleep instead of yield-spinning (Linux + --mmap-experts: prompts ~10x faster on DGX Spark, ~1 core instead of ~20) by uncle-daddy-jp · Pull Request #1057 · Niko1221/Strata

uncle-daddy-jp · 2026-10-06T06:59:58Z

Fixes the prefill slowdown and CPU burn described in the linked issue: the Stager threads waited by spinning (a yield loop on issued, and cudaEventSynchronize on events created without cudaEventBlockingSync, which spins under the default scheduler). With --mmap-experts there are 32 of them; on Linux they inherit the host thread's single-core pin and starve it (prompts read ~10x slower on a DGX Spark), and unpinned they take every core for the whole prompt.

This change makes both waits sleep:

the ring wait uses C++20 std::atomic::wait, with notify_all in issued_one() and finish();
dma_done events are created with cudaEventDisableTiming | cudaEventBlockingSync;
the HIP shim gets cudaEventBlockingSync -> hipEventBlockingSync (not tested on HIP).

No affinity changes. Measured on a DGX Spark (GB10, CUDA 13.0, Ubuntu 24.04) with IQ3_S, all experts in the GPU cache, prompts of 7,031 / 26,383 tokens:

build	8K read tok/s	30K read tok/s	process CPU while reading	decode tok/s
before (`44df8fd` / same code as main)	91	88	~100 (all on core 0)	42-48
stagers unpinned instead	1,068-1,183	1,344-1,397	1,640-1,880	40-42
this PR	1,222-1,229	1,388-1,390	102	41-43

Output tokens are unchanged (greedy, same prompts). git apply --check passes on main (1735d64) and on PR #409's branch.

Measured with the help of Claude (Anthropic).

🤖 Generated with Claude Code

…variant 5) recurrence kernels, bitwise T = 1, 7, 8, 9, 17, 100, 2048, 2051 (full and partial chunks), y, its FP16 bits and the final state: 0 values differ. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJJwUv9g3cD2ztX6KWS8rk

… with hipBLASLt solution 1176/1177 k order (StaggerU included), bitwise Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJJwUv9g3cD2ztX6KWS8rk

… and sum finite too 0.1.39 clamped two q8_1 activation quantizers (Niko1221#606: an fp16 sum or scale past 65504 rounds to inf, the dot products then read inf * 0 = NaN, and a NaN in the residual ends as one token repeated). Two more store the block unclamped: - native_swiglu_quantize_q8_1_kernel (native_mmvq.cu), the shared expert's fused SwiGLU + q8_1, on by default (STRATA_FUSED_SWIGLU_Q81) in decode and the multi-token pass; - native_gu_fused_kernel (iq_kernels.cu), the routed experts' fused SwiGLU + q8_1 (gfx906 builds, STRATA_EXP_MODE=8). Both now go through q8_1_finite / q8_1_quant / q8_1_ds as their siblings do: every finite block is stored bit for bit as before. iq_multi_parity's Niko1221#606 check covers the shared expert's one (gate = 32, up = x / 32: its activations are x exactly). rwkeyes found the first one (Niko1221#606). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

… moe_combine, LDS alias + occupancy) Bitwise-identical to f4e96d5 (generated ids identical on UD-IQ4_XS, UD-Q4_K_XL, IQ3_S, IQ3_XXS at 8K/4K/64K; harness fused-output hash unchanged): - table16: IQ4_NL / IQ4_XS decode through the direct-selector v_perm form (X5; exhaustive 2^32 check). -DSTRATA_W_NO_T16 restores the old form. - moe_combine: four columns per thread with float4 loads, same fma order (X5); falls back to the scalar kernel if Dm / shared / bo are not 16-byte aligned. -DSTRATA_W_NO_COMB. - native_w11_kernel: the H tile aliased onto the weight buffers (gate/up LDS 46.5 -> 30.2 KB), amdgpu_waves_per_eu(8), 4 blocks per WGP (X1), only for the types that do not spill under the 192-VGPR cap (gate/up IQ2_*, IQ3_*, IQ4_XS; down Q2_0, Q8_0, IQ4_NL). -DSTRATA_W_NO_OCC. - X3 on1 (skip the second 16-row n tile of the down kernel) NOT included: no gain alone (-0.1%), and with the occupancy cap it makes IQ4_NL down spill 62 VGPRs and the whole bundle 2.3% slower. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJJwUv9g3cD2ztX6KWS8rk

…): a host OOM skips parking Co-authored-by: zengyishou <zengyishou@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…er cards --vram-reserve-mib is every card's reserve, but the card that drives the monitors needs more headroom than one that drives none: one value has to fit the display card, and the other cards then keep that headroom for nothing. --vram-reserve-later-mib N is the reserve of every card after the first (default: --vram-reserve-mib's value), in the split search and in each later card's cache sizing. RTX 4060 Ti (no display, layers 0-19) + RTX 5080 (display, 20-47), Swift 1.5 IQ3_XXS at 160K: --vram-reserve-mib 1000 gives the 4060 Ti's cache 4551 slots; --vram-reserve-mib 300 --vram-reserve-later-mib 1800 gives it 4867, and the 5080 keeps 1.3 GB free with everything loaded.

--resident-experts (the low-RAM mode's resident variant) ran as plain --mmap-experts as soon as a layer split was set. It now keeps its RAM copy on a split: - the experts every stage's cache holds are left out of the RAM copy, not only CUDA0's; - when the rest does not fit whole (Niko1221#467), the copy keeps the hottest by the whole expert profile. Each stage only keeps its own layers' share of the profile, and ranked by CUDA0's share alone the copy held layers 0..K-1's experts and read every later layer's from the files (4.0 GiB in RAM instead of 21.2 on the PC below); - an adaptive swap copies the evicted expert back into RAM from the cache of the card that owns its layer, on that card's stream. host_res holds slot numbers of whichever cache owns the layer: reading a later stage's slot out of CUDA0's cache put another expert's bytes in RAM, and after the first swaps the answers turned to garbage. (Latent until now: the guard kept the mode off a split.) - the decode profile prints every stage's GPU profile, not only CUDA0's. Remote expert caches (--expert-cache-remote) are still refused with it. Swift 1.5 IQ3_XXS at 160K, q4_0 KV, RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47), i9-14900KF, 32 GB of RAM, Windows 11, four greedy prompts, decode tok/s (first round / fourth): - 5080 alone, --resident-experts: 25 / 29 - split, --mmap-experts (what --resident-experts became): 32 / 64 - split, --resident-experts (22 GiB of experts locked in RAM): 71 / 69 With --pcie-frac 0 --adapt-every 0 the split's greedy output is the same as --mmap-experts' (4/4 prompts). With adaptive swaps on, 33 varied requests at temperature 0.6 (thinking on and off, five languages, code, each checked for degenerate output) exchanged 81,783 experts with the two caches and gave no degenerate answer.

) With an engine that keeps --resident-experts on a layer split, setup no longer steers a low-RAM PC with two cards to one of them: - at install, the low-RAM mode on several GPUs takes the resident variant by the same rule as on one card (what no card holds fits the RAM with LOW_RAM_HEADROOM_GB beside it), without the one-card question; - at a start, a resident config on several GPUs keeps --resident-experts (split_mmap leaves it alone); - a resident config started on a PC with two cards is offered both, "y" by default, as any other config. It is gated on MIN_ENGINE >= RESIDENT_SPLIT_ENGINE (0.1.40 here, a guess at the release that would carry the engine change), so until then setup behaves as before and the golden configs are unchanged. Two tests run the new paths with MIN_ENGINE patched; tools/test_setup_*.py pass (260 tests).

…; later stages say which stage Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…non-blocking streams read the table Every upload of the expert residency table (host_res -> d_res, and each stage's) was a plain cudaMemcpy: a pageable copy on the legacy stream, which may return before its DMA has landed, while the kernels that read the table run on streams created cudaStreamNonBlocking, which do not wait for the legacy stream. A kernel enqueued right after could read a table half old, half new: an expert computed by both the GPU and the CPU, or by neither. The ~100 KB copy lands in microseconds, so this rarely fires, but nothing ordered it. The uploads now go through res_put(), which waits for the copy on the current device's legacy stream only. Same values, same output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

…221#550) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…M; a -1 raises a plan error flag (Niko1221#871) A prompt loan, a VRAM shrink or an adaptive swap marks experts -1 in the residency table. The zero-doorbell graph planned from the device table and returned silently on a -1, so the window ran a stale plan (!!!! at 100% residency). refresh_ar() checks the host table before each window and runs the doorbell graph (its own captured variant) while any expert is out; resident_plan also writes a mapped error flag the host checks after the window. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ill::init (Niko1221#828 test switch) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…1221#879 abort latency) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…s and an incomplete checkpoint (Niko1221#752) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…bundle) onto 0.1.39 Merges f-halo-int (36fa455 + 56 commits: the gfx1151 target, WMMA prompt attention / GEMM / scorer, GDN recurrence, fused gfx11 prompt experts, decode switches, drafting) with f-halo-int-S (STRATA_HCD_EXACT, 16bc95a) and f-halo-int-W (table16, float4 combine, LDS alias + occupancy, d23d4e2). Conflicts resolved against 0.1.39's own all-resident verify graph, forked shared-expert stream, fused swiglu q8_1, PLE batch, GDN commit split and peer prompt path; the gfx1151-only code stays gated (build fixes and the arch defaults follow in separate commits). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…NT is 0.1.39's all-resident graph 0.1.39 already skips the host round trip when every expert is in VRAM (all_resident_, on by default, Niko1221#871's refresh_ar), so the port keeps that graph and adds only what halo built on top of it: the LFUSE gated combine. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…LE_HIP (it includes hip_runtime.h; the CUDA build broke) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… zeros on gfx1103 / gfx1152), cached per device WmmaTraits::mma's #else returned its accumulator (zeros) for a gfx11 part outside the __gfx1100__..__gfx1151__ list; it traps now, and the runtime gates of the WMMA GEMM, the prompt attention, the block scorer and the HC up-projection take exactly gfx1100 / 1101 / 1102 / 1150 / 1151 (strata/kernels/gfx_arch.hpp) so a gfx1103 or gfx1152 falls back to hipBLASLt instead of launching code that is not there. The gate is cached per device (a process-wide static answered for the first GPU only; hipGetDeviceProperties on every call was the review's other complaint). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…iko1221#409 review), not in a process-wide static Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…1151 only (arch_defaults), the rounding-level ones stay opt-in 16 switches whose output was byte-identical to the switch off on Aurora (GDN head / PP / CONVL2 / NOY / SPLIT, QFUSE, CVEC_FUSE, HCD_EXACT, PF_PAD, Q8 / Q6 packed, MMVF_ROWS, ATTN_LANECELL, EXPERT_V2, TSUM, LFUSE, PLE_BATCH) are set at start-up when device 0 is gfx1151 and the user has not set them; STRATA_GFX1151_DEFAULTS=0 turns the table off. The GDN head kernels are AMD-only now (HIP builds; CUDA keeps its key-head kernel), and verify.cpp's namespace-scope env reads became lazy so the table reaches them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…, the DONE line's newline escapes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…o tuning table, or a solution other than 1176 / 1177), and hip_prefill_hcd_exact_parity checks the kernel against hipBLASLt The kernel reproduces one hipBLASLt solution's summation order; the test compares both on the same inputs for T from 64 to 16384 (bitwise) and SKIPs without STRATA_HIPBLASLT_TUNING. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

PleTable had one bool per format (q5_0, fp8) and an if-chain in decode(), open() and format(). Now one list, ple_formats(), holds one entry per table type: its GGUF type name, the bytes of a 160-value row, whether it carries a scale and its dequantizer. open(), format(), the row size, the error string and the row buffers all read it, so the next formats (Q8_0, Q5_1, Q4_0, BF16) are one entry each. No behaviour change: IQ4_NL, Q5_0 and FP8 decode with the same functions, and the "not IQ4_NL, Q5_0 or FP8 (I8)" error text is built from the list and reads the same. ple_reader_test --selftest now writes a PLE-only GGUF for every format in the list and checks both readers (direct and mapped) against the format's own dequantizer, for single rows and a 16-row token gather. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Ordinary GGUFs ship per_layer_token_embd.weight in other formats than IQ4_NL: Unsloth's UD-Q6_K_XL and Swift-1.5 Q4_K_L as Q8_0 (170-byte rows, ~54 GB), a Q5_K_M finetune as Q5_1 (120-byte rows). The engine refused both at start. - dequantize_q5_1 in strata/artifact/dequant.hpp (ggml's dequantize_row_q5_1: unsigned 5-bit codes times d plus the block minimum m); Q8_0 rows use the existing dequantize_q8_0 - two entries in the PLE format list (the table-driven commit before this one): Q5_1 and Q8_0, on both readers; PLE_ROW_BYTES_MAX grows to 170 - tests: dequant_q5_1_test (synthetic blocks against ggml, no model, no GPU), dequant_bf16_test knows Q5_1, ple_q5_parity takes Q5_0, Q5_1 or Q8_0 tables and checks both the mapped and the direct reader Ported onto the format table by the maintainer from anon761's PR Niko1221#651 (the --ple-io ram fault-in is the next commit). Quality of the Q8_0 table: ksanislo measured 0.53% error against the BF16 table, FP8 2.66%, IQ4_NL 7.59%. Co-authored-by: ksanislo <ksanislo@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

PleTable::open compared rows x row_bytes against the file after the 64-bit product could already have wrapped, and mapped the tensor before checking it fits. Now: refuse a row count that overflows rows x row_bytes and a data section that starts past the end of the file, check offset and size without an addition that can wrap (offset > have, need > have - offset), require a single-tensor shard to start at offset 0 and fill the file, and map the table only after all of that. close() also resets the FP8 scale and the byte counter. ple_reader_test --selftest writes tables that claim one row too many, one too few and a wrapping row count, for every format, and checks both readers refuse each with a message and stay closed. Checks taken from CC-David-CC's Niko1221#865 (its opt-in Q8_0 table is Niko1221#651's; the direct reader reads Q8_0, so the experimental switch and the mapped-only limit are not needed). Co-authored-by: CC-David-CC <CC-David-CC@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Niko1221#651) mlock alone brings a cold file in one page at a time from one thread, ~0.47 GiB/s (about two minutes for a 54 GB Q8_0 table); 16 threads reach ~2 GiB/s on anon761's machine. The pages are then resident and mlock only pins them. When mlock fails, the pages stay faulted in (the message says they may be reclaimed) instead of being touched again one by one. ple_reader_test gets --make-table PATH FORMAT (a big synthetic table, with --rows) and --ram-time PATH (open mapped and locked, print the seconds) to time it cold. Ported by the maintainer from anon761's PR Niko1221#651. Co-authored-by: ksanislo <ksanislo@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

A plain llama-quantize Q4_0 file of this model (bartowski's Qwen3.8-Flash-Next-Q4_0 and Swift-1.5 Q4_0: gate/up Q4_0, down Q4_0 on 42 layers and Q4_1 on 6, a Q4_0 PLE table and token embedding) was refused at start: no GPU kernels for Q4_0/Q4_1 experts, and the PLE reader took only IQ4_NL, Q5_0 and FP8. Follows Niko1221#222 (Q4 to Q6 quantizations). - Q4_0: Fmt<2> with the codes centred before dp4a (q - 8 as signed bytes), the exact integer form ggml-cpu's q4_0 x q8_0 dot uses, rather than llama.cpp's CUDA chain (uncentred codes minus 8 x the q8_1 float sum): on a real Q4_0 expert the CUDA chain measured 1.4e-2 relative error against the float product, the centred form 5.3e-3. - Q4_1: Fmt<3>, the Q5_1 chain without the fifth bit and the same min-term choice. - dq_q4_0 / dq_q4_1 for the prompt path's dequant fallback; row bytes, is_iq and the GU/D/MMVQ format lists. - PLE: Q4_0 rows (90 B, as IQ4_NL) through the same reader: one entry in the PLE format list (ple_format_test and ple_reader_test cover it). - docs/UNSLOTH_Q4.md: a Q4_0 section with the measured run. native_expert_parity --synthetic (now in the CMake matrix): q4_0/q4_0 CPU 1.29e-2, GPU 1.29e-2, CPU-GPU 1.1e-7; q4_0/q4_1 CPU-GPU 3.2e-4; GPU dequant bit-exact against to_float. The existing pairs are unchanged. On bartowski's file (revision 928589f), layers 0 and 5 (Q4_1 down) and 20 and 47: CPU and GPU 1.1-1.3e-2, CPU-GPU 1.1e-7 (Q4_0 down) and 5e-4 to 1.1e-3 (Q4_1 down). Served on one RTX 3060 (all experts in RAM, 131K context): 24,461-token prompt 787 tok/s read, 24.0 tok/s out; 67,934 tokens 737 / 26.4, the hidden fact found. Ported onto the format table by the maintainer (the PLE part of the PR is the Q4_0 table entry). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The full checkpoint stores the table as BF16 (320-byte rows, 102.4 GB), the source of record; the engine gets it as IQ4_NL (about 8% off per row) or FP8. A bfloat16 is the top half of a float32, so widening is a shift: exact for every value, no rounding and no scale. One entry in the PLE format list (type BF16, self-describing: no scale, no metadata) and bf16_dequant_row; the direct reader takes the 320-byte row like any other width, and PLE_ROW_BYTES_MAX is 320. ple_reader_test checks the widening is exact over the patterns a lossy path breaks on (subnormals, +-inf, both NaNs); the generic every-format test covers a synthetic BF16 GGUF through both readers. Not in this commit: the PR's rename of tools/ple_fp8_pack.py (nothing needs it) and any setup option. Whether a table format other than IQ4_NL is worth offering is a quality question (see the measurement in the 0.1.40 notes). Ported by the maintainer from constantindjonkam's PR Niko1221#586. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… wheel's library/gfx1151/) instead of crashing Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJJwUv9g3cD2ztX6KWS8rk

…e-VRAM readout after the release vmm_test failed on the RTX 5070 / driver 610.47: the pageable host-to-device copy was still in flight when a.unmap() pulled the chunks out from under it, an illegal memory access that stuck to every later call (the last two checks, the 8-chunk free-VRAM gain and cudaGetLastError, failed as a result). It was not a VRAM leak: released memory comes back at once (18 MiB gained for 9 chunks of 2 MiB, polled 10 s). generate.cpp already synchronizes before it unmaps. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJJwUv9g3cD2ztX6KWS8rk

…n Linux rejects its std::array locals) dma_batch_parity.cu failed to compile on Linux/WSL (nvcc 13.0 with g++): 'template argument 2 is invalid' at the std::array<T, cap> locals of copy_expert_blobs, whatever the form of the size (int, size_t, a literal). The engine objects and the Windows build were not affected; the Linux tests build was. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJJwUv9g3cD2ztX6KWS8rk

The prefill Stager waited by spinning: a yield loop on `issued`, and cudaEventSynchronize on events created without cudaEventBlockingSync (which spins under the default scheduler). With --mmap-experts there are 32 stagers. On Linux they inherit the host thread's single-core pin and starve it: on a DGX Spark (GB10) prompts were read ~10x slower (IQ3_S 8K 91 -> 1,222 tok/s, 30K 88 -> 1,388 tok/s with this change). Unpinned instead, the 32 spinners took every core for the whole prompt (process CPU ~1,800%); with the waits sleeping it is ~100%. - ring wait: std::atomic::wait, with notify_all in issued_one() and finish() - dma_done events: cudaEventDisableTiming | cudaEventBlockingSync - HIP shim: cudaEventBlockingSync -> hipEventBlockingSync (untested) Decode speed and output tokens are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Niko1221 · 2026-10-06T10:06:04Z

Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when main was force-pushed (that is the "base branch was force-pushed" event above). It is not a rejection.

GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new main, cherry-pick your own commits onto it (git fetch upstream && git switch -c my-fix upstream/main && git cherry-pick <your commits>), and open a new PR. Or simply leave it: when we take your change for a release we port it onto the new main ourselves and keep you as its author.

Sorry for the noise, and thank you for the contribution.

uncle-daddy-jp · 2026-10-06T10:54:01Z

Thanks for the heads-up. Re-opened on the new main as #1101.

The prefill Stager waited by spinning: a yield loop on `issued`, and cudaEventSynchronize on events created without cudaEventBlockingSync (which spins under the default scheduler). With --mmap-experts there are 32 stagers. On Linux they inherit the host thread's single-core pin and starve it: on a DGX Spark (GB10) prompts were read ~10x slower (IQ3_S 8K 91 -> 1,222 tok/s, 30K 88 -> 1,388 tok/s with this change). Unpinned instead, the 32 spinners took every core for the whole prompt (process CPU ~1,800%); with the waits sleeping it is ~100%. - ring wait: std::atomic::wait, with notify_all in issued_one() and finish() - dma_done events: cudaEventDisableTiming | cudaEventBlockingSync - HIP shim: cudaEventBlockingSync -> hipEventBlockingSync (untested) Decode speed and output tokens are unchanged. (PR #1057)

… only; Windows keeps the yield spin (5070 8K Q2_0 prompt -1.2% with them). STRATA_STAGER_SLEEP=1/0 forces either

Niko1221 · 2026-10-07T12:31:56Z

Thank you, uncle-daddy-jp. Your Stager wait change is in 0.1.40.2 with your authorship kept, as the default on Linux only. Windows keeps the old yield-spin, because the sleeping wait measured 1.2% slower there (RTX 5070, 8K Q2_0 prompt). STRATA_STAGER_SLEEP=1/0 forces either. On an RTX 3060 under Linux prompt speed was unchanged (290.05 vs 289.9 tok/s, 10 interleaved pairs); the gain you saw on GB10 comes from CPU-starved Stagers, and a desktop with spare cores does not show it. The PR was closed by the history rewrite.

Upstream now carries our picks of Niko1221#851 (AVX2 Q8_K quantizer), Niko1221#789 (expert uploads beside the shared expert), Niko1221#813 (--ple-io ram on Windows), Niko1221#1185 (window graphs free older batch layouts), Niko1221#1201/Niko1221#1118 (doorbell per window), Niko1221#1101/Niko1221#1057 (Stager sleep: now the default on Linux only, STRATA_STAGER_SLEEP=1 forces it) and Niko1221#1063 (the shared draft head's type); their versions are kept. Conflicts resolved: - prefill: upstream's own CPU share (STRATA_PREFILL_CPU_SHARE, opt-in) sits beside CPU assist. Its setter is set_cpu_share_pool; when it is asked for, it replaces CPU assist for the chunks it covers, otherwise CPU assist runs as before (on by default). The debug NaN check skips both kinds of CPU rows. - conversation cache: a pool-borrowed entry and upstream's pinned shared prefix (pin=N) are both kept from eviction; the borrow (Niko1221#1164) and pin=N's sibling path do not park at once. - generate: pool batch groups launch only up to their last active slot (Niko1221#1249's G.S), also on the workers; --adapt-async's poll between verdicts runs beside the pool's verified-path-first loop. - server: upstream's chunked bodies (Niko1221#893) and the pool's read-once body are one helper; do_POST's BadBody wrapper around the pool's routed-request check. - native dense: the pool's served_bytes beside upstream's weight_bytes_for (Niko1221#1238). - expert source: the commit-limit retry (Niko1221#1064) beside upstream's paced page-locking (Niko1221#1250). - setup: no ready-made engine (StrataPool compiles its own); REPO names StrataPool for the SHA-256 lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157fjixQD2nRxvCvtXq66DD

The v0.1.40.2 Stager sleep waits (Niko1221#1057) call cudaEventCreateWithFlags with cudaEventBlockingSync. include/strata/platform/hip_compat/strata_hip.h is the header the upstream STRATA_HIP_GFX906 text build uses, and it had no mapping, so the ROCm 10 clang 23 build stopped with "cudaEventBlockingSync was not declared in this scope". HIP names the flag hipEventBlockingSync. The older fork compat header (include/strata/hip_compat/) already mapped it. Verified on T5810: the v0.1.40.2 text engine build reaches this header after the merge.

Built the merged tree on T5810 in the separate worktree /home/chris/dev/strata-merge-check (2026-10-07 17:56-18:06 UTC). The deployed build directories and the 8082 service were not touched; both GPUs were idle but hold the running model. The first text build stopped on src/prefill/prefill.cpp:399: cudaEventBlockingSync was not declared. Upstream's Stager sleep waits (Niko1221#1057) need that flag, and the platform HIP compat header had no mapping; a7eeee3 adds it. The rebuild linked with no errors, and the vision encoder built 275/275. New artifacts, not deployed: - build-text-rocm10-v402/strata 055a9fed38fac6d42c6786a4d52fd52287cac0f25e2ad4751580267b6e001830 - build-vision-rocm10-v402/bin/strata-vision 28c4895f3913d9f0f2a5f65e590d8f10767a14b3c20b5885c339a9dffceb87c0 The engine reports engine=0.1.40.2, links 20 ROCm 10 libraries and no ROCm 7.2.4 ones. Tests in the merged tree: tools setup suite 397 tests, 2 failures, 1 skip; both failures reproduce at upstream e8ca9af alone (which also has 4 errors this fork's mock already fixes). tests/test_gfx906*.py: 44 tests, OK, 5 skipped, after this commit updates the version pin to source_version 0.1.40.2 and MIN_ENGINE (0, 1, 40, 2). No model generation, pipeline parity, parking, image encoding or tok/s measurement was run: the running service holds the VRAM. Deployment of v0.1.40.2 needs an owner-approved restart.

265 commits: the 0.1.40.2 batch (F4 verify windows, the Linux file tier Niko1221#1194, the stager wait Niko1221#1057, opt-in CPU share Niko1221#1282, Intel Arc) and the 0.1.40.3 hotfix (Intel Arc install path and A750 first answer, Windows AMD cache floor Niko1221#1376, bundled HIP runtime Niko1221#461, Docker config links Niko1221#1244, the Niko1221#1357 MTP router guard, the tokenizer hardening Niko1221#1385). main keeps 33af099 (AMD PCIe link speed/width and GPU name detection); its 34 added telemetry lines are all present after the merge.

The 0.1.40.3 docs measure sleeping stager waits as 5-6% slower (and 1.2% faster when spinning on a desktop card), so Niko1221#1057 is a CPU-load feature, not a speed feature. The measured numbers are unchanged; only the explanation is removed.

The prefill Stager waited by spinning: a yield loop on `issued`, and cudaEventSynchronize on events created without cudaEventBlockingSync (which spins under the default scheduler). With --mmap-experts there are 32 stagers. On Linux they inherit the host thread's single-core pin and starve it: on a DGX Spark (GB10) prompts were read ~10x slower (IQ3_S 8K 91 -> 1,222 tok/s, 30K 88 -> 1,388 tok/s with this change). Unpinned instead, the 32 spinners took every core for the whole prompt (process CPU ~1,800%); with the waits sleeping it is ~100%. - ring wait: std::atomic::wait, with notify_all in issued_one() and finish() - dma_done events: cudaEventDisableTiming | cudaEventBlockingSync - HIP shim: cudaEventBlockingSync -> hipEventBlockingSync (untested) Decode speed and output tokens are unchanged. (PR Niko1221#1057)

…on Linux only; Windows keeps the yield spin (5070 8K Q2_0 prompt -1.2% with them). STRATA_STAGER_SLEEP=1/0 forces either

…wait (Niko1221#1488) The Stager waits from Niko1221#1057 use std::atomic<int>::wait/notify_all, a C++20 library feature libstdc++ only ships from GCC 11, so a GCC 10 build (Ubuntu 20.04 / Debian 11 newest packaged compiler) fails at the three call sites even though CMAKE_CXX_STANDARD 20 is satisfied. Gate the sleeping wait on the feature-test macro: toolchains without it fall back to the pre-Niko1221#1057 yield spin, and the notify_all() calls are redundant on the spin path (spinners poll, they never block in wait()). Behavior on GCC 11+ / Clang is unchanged.

Stager のスレッドは、自分のリングのバッファの前の DMA が積まれるのを yield で回って待っていた。 --mmap-experts では 32 個の Stager が回り、upstream では host のスレッドを飢えさせてプロンプトが 10 倍遅くなった（Niko1221#1057）。 std::atomic::wait で眠り、issued_one と finish が notify_all で起こす。STRATA_STAGER_SLEEP=0 で今までどおり回る。 B70 と RTX 4070 のプロンプト（4,945 トークン）の速さは変わらない（1,139 / 1,138、790 / 791 トークン/秒）。

…t so a GCC 10 build spins as before #1057 (#1488)

…t so a GCC 10 build spins as before Niko1221#1057 (Niko1221#1488)

claude and others added 30 commits October 4, 2026 20:26

prefill: STRATA_HCD_EXACT=1 - the HC down projection by a WMMA kernel…

16bc95a

… with hipBLASLt solution 1176/1177 k order (StaggerU included), bitwise Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJJwUv9g3cD2ztX6KWS8rk

cache: preserve checkpoints when split allocation fails

64d0136

serve: park_current's try covers the checkpoint split too (Niko1221#752…

5f5452b

…): a host OOM skips parking Co-authored-by: zengyishou <zengyishou@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

serve: a split's stage-0 GPU profile line keeps its text (Niko1221#848)…

cd62e5e

…; later stages say which stage Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

generate: res_put syncs stream 0 (HIP has no cudaStreamLegacy) (Niko1…

2d82703

…221#550) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

prefill: STRATA_TEST_PAGEABLE=1 forces the pageable fallbacks of Pref…

7892d9a

…ill::init (Niko1221#828 test switch) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

serve: a Stop during the prompt read goes to the engine at once (Niko…

1deb66c

…1221#879 abort latency) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

test: the checkpoint split's allocation-failure test covers 1-3 stage…

c469ab1

…s and an incomplete checkpoint (Niko1221#752) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

build: wmma_gemm.cu is added to strata_prefill only under STRATA_ENAB…

3a75727

…LE_HIP (it includes hip_runtime.h; the CUDA build broke) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

core: device_free_bytes caches the unified-memory check per device (N…

119002b

…iko1221#409 review), not in a process-wide static Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

merge fixes: the stray #endif of the two fused-expert dev_info blocks…

34a806c

…, the DONE line's newline escapes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Niko1221 and others added 4 commits October 6, 2026 02:03

fix(hip): package_windows copies rocBLAS arch subfolders (the gfx1151…

3674aa5

… wheel's library/gfx1151/) instead of crashing Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QJJwUv9g3cD2ztX6KWS8rk

uncle-daddy-jp mentioned this pull request Oct 6, 2026

aarch64 / NVIDIA DGX Spark (GB10): build, run and set up with unified memory #409

Closed

Niko1221 closed this Oct 6, 2026

Niko1221 force-pushed the main branch from 1735d64 to 82f46a8 Compare October 6, 2026 09:48

uncle-daddy-jp mentioned this pull request Oct 6, 2026

prefill: let Stager threads sleep instead of yield-spinning (Linux + --mmap-experts: prompts ~10x faster on DGX Spark, ~1 core instead of ~20) #1101

Closed

uncle-daddy-jp mentioned this pull request Oct 6, 2026

Batch slots: the engine stalls ("no progress for 60 s … reading the prompt (batched)") when a long prompt is read while another slot decodes; STRATA_BATCH_DECODE_SHARE=0 avoids it #1118

Closed

Niko1221 added a commit that referenced this pull request Oct 7, 2026

prefill: the Stager's sleeping waits (#1057) are the default on Linux…

2a8cded

… only; Windows keeps the yield spin (5070 8K Q2_0 prompt -1.2% with them). STRATA_STAGER_SLEEP=1/0 forces either

Niko1221 mentioned this pull request Oct 7, 2026

Prefill: Stager threads yield-spin while waiting. On Linux with --mmap-experts (DGX Spark / GB10), prompts are read ~10x slower; elsewhere ~20 cores are burned during prefill #1056

Closed

Dmitry-B mentioned this pull request Oct 7, 2026

bench: community report - RTX 4090, Flash-Next IQ3_XXS at 204800, engine 0.1.40.2 #1377

Closed

Dmitry-B mentioned this pull request Oct 7, 2026

bench: community report - RTX 4090, Flash-Next IQ3_XXS at 204800, engine 0.1.40.3 #1467

Closed

0xrjman pushed a commit to 0xrjman/Strata that referenced this pull request Oct 7, 2026

prefill: the Stager's sleeping waits (Niko1221#1057) are the default …

d33b985

…on Linux only; Windows keeps the yield spin (5070 8K Q2_0 prompt -1.2% with them). STRATA_STAGER_SLEEP=1/0 forces either

zhouxihong1 mentioned this pull request Oct 8, 2026

prefill.cpp: a GCC 10 build fails (std::atomic::wait is a GCC 11 library feature, with no guard and no documented minimum) #1488

Closed

nekomario28 mentioned this pull request Oct 8, 2026

prefill: guard atomic::wait/notify_all so GCC 10 builds again (#1488) #1532

Closed

2 of 4 tasks

Niko1221 added a commit that referenced this pull request Oct 10, 2026

prefill: guard std::atomic::wait/notify_all with __cpp_lib_atomic_wai…

df29e3b

…t so a GCC 10 build spins as before #1057 (#1488)

giridhart pushed a commit to giridhart/Strata that referenced this pull request Oct 15, 2026

prefill: guard std::atomic::wait/notify_all with __cpp_lib_atomic_wai…

b6ca3a4

…t so a GCC 10 build spins as before Niko1221#1057 (Niko1221#1488)

Navigation Menu

Repository navigation

prefill: let Stager threads sleep instead of yield-spinning (Linux + --mmap-experts: prompts ~10x faster on DGX Spark, ~1 core instead of ~20) - #1057

prefill: let Stager threads sleep instead of yield-spinning (Linux + --mmap-experts: prompts ~10x faster on DGX Spark, ~1 core instead of ~20)#1057
uncle-daddy-jp wants to merge 1153 commits into
Niko1221:mainfrom
uncle-daddy-jp:prefill-stager-sleep

uncle-daddy-jp commented Oct 6, 2026

Uh oh!

Niko1221 commented Oct 6, 2026

Uh oh!

uncle-daddy-jp commented Oct 6, 2026

Uh oh!

Niko1221 commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

Conversation

uncle-daddy-jp commented Oct 6, 2026

Uh oh!

Niko1221 commented Oct 6, 2026

Uh oh!

uncle-daddy-jp commented Oct 6, 2026

Uh oh!

Niko1221 commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants