Repository navigation
Conversation
…solo l'input su rami del grafo (PR Niko1221#1889) Diff del PR applicato com'è, più due adattamenti: le due chiamate a native_mmvq_il_supported del ramo =2 passano n_in (API del Niko1221#1437), e con STRATA_SIDE_MIXER acceso STRATA_DF_BRANCH (acceso di default su Windows) si spegne già in init senza creare i suoi stream; se era chiesto esplicitamente lo dice una volta su stderr. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SIDE_MIXER=1/2, opt-in) STRATA_SIDE_MIXER=1 runs the GDN gates and the QSA indexer (key projection, tail snapshot, query chain, block scores, top-k) on a branch of the window's graph beside the attention's projections, joined before the recurrence and the attention; =2 also runs the GDN gate projection and the QSA queries' chain on a second branch. A branch that projects reads F4's interleaved copy, written on the main stream before the fork, and qkv is recorded before the branch so its grid is dispatched first. Every result is bitwise the same as in line; STRATA_DF_BRANCH is off while this is on. Unset, the window runs in line as before. The idea is from Eddoursul's fork (34a35db, b063c13).
602a812 to
38b0328
Compare
|
Rebased onto v0.1.43 ( Measured and drafted with Claude Code. |
|
Thanks. Ported as an opt-in (STRATA_SIDE_MIXER=2). Text is identical in our gate (10 prompts and 4K/20K dumps) on RTX 3060, A4000 and a P100. Gain on a single card was nil to +0.5% (30K on the 3060 +0.5%; A4000 short context -2.3..+0.2%). We did not run a layer split, so the #905 question stays open; it stays off by default. |
|
Thanks for the pull request. Your change is already in If part of your work did not make it in and you want it considered, open a new pull request on top of |
2 participants
Title
Issue: none (an opt-in speed change)
Summary
STRATA_SIDE_MIXER=1or=2(opt-in, off by default) records the mixer's work that reads only the layer's input onbranches of the verify window's CUDA graph, beside the projections that the recurrence and the attention wait for:
=1: the GDN gates (a/b) on one branch, joined before the recurrence; the QSA indexer's chain (key projection, tailsnapshot, key append, query projection with norm and RoPE, block scores, top-k) on it, joined before the attention.
=2: also the GDN gate projection (z) and the QSA queries' chain (projection, q/gate split, norm, RoPE, rotation) on asecond branch. A branch that projects reads F4's interleaved q8_1 copy, written on the main stream before the fork, and
qkv is recorded before the branch so that its grid is dispatched first.
Each step is the in-line code with its stream as a parameter, so the outputs are bitwise the same (exact7 with the switch
on, below).
STRATA_DF_BRANCHis the same idea for fewer of these steps; it is off whenever this switch is set. The ideais from Eddoursul's fork (34a35db, b063c13).
Measured 2026-10-10 on one PC: RTX 5080 16 GB + RTX 4060 Ti 8 GB (driver 610.57.04, CUDA 13.3), Ryzen 9 9950X3D (16 cores), 96 GB DDR5, Linux 7.0; Qwen3.8-Flash-Next GSQ-RCO IQ3_S at a 131,072-token context. Served, stock against stock + this patch only, A B B A with a fresh server per arm (the model files evicted from the page cache before each), both arms with
--expert-cache auto --prefill auto:16384 --spec 4 --spec-min-p 0.5 --mtp --max-context 131072 --kv int8 --kv-resident 32768 --pcie-frac 0 --ple-inflight 128 --vram-reserve-mib 713withSTRATA_IO_THREADS=64 CUDA_MODULE_LOADING=LAZY STRATA_GR_V3=1 STRATA_PF_FUSED=1 STRATA_MMVQ_IL=1 STRATA_ADAPT_LAG=2 STRATA_DISJOINT_ADAPT=1(the stock flags of my community benchmark report, #1681); the patch's arm addsSTRATA_SIDE_MIXER=2. Requests carry no sampling fields, so the server's configured defaults apply: temperature 1.0, top-p 0.95, top-k 20, min-p 0.05, reasoning effort high. Decode screen: long reasoning math and code (4,096 tokens each), five short questions, a 32K prompt with 768 tokens out; decode figures are the engine's own (accepted output tokens / decode time). Prompt-read screen: time to first token for prompts of 1K x3, 4K x3, 32K x2 and 120K x2 tokens (each a different text, 768 tokens out; its decode column comes from short sampled answers and varies more than the decode screen).In short (this patch against the same stock build, means of the two A B B A arms each):
STRATA_ROUTE_TAIL_SKIP=0The RTX 5080 alone, against v0.1.42
61b3fb5dwith upstream's defaults: v0.1.42 turns onSTRATA_ROUTE_TAIL_SKIP=7for single-GPU serve, so both arms skip low-ranked missed experts (what single-card users get).The RTX 5080 alone, against v0.1.42 with
STRATA_ROUTE_TAIL_SKIP=0in both arms (the tail skip off: the answers of a build without it).The RTX 5080 with the RTX 4060 Ti as the helper expert cache (
--expert-cache-device1 3300), measured on v0.1.41fb58e0dbwith this change on v0.1.41, before the rebase. The tail skip does not apply with a helper GPU.Against
STRATA_DF_BRANCH(on v0.1.41fb58e0dbbefore the rebase, the same day, the RTX 5080 alone, the same flags and decode screen; one run of six armsin mirrored order, two servers per arm): stock v0.1.41 113.8 tok/s and 17.44 ms per verify round;
STRATA_DF_BRANCH=1115.2(+1.2 %) and 17.21 ms;
STRATA_SIDE_MIXER=2118.5 (+4.1 %) and 16.89 ms, which is 1.9 % less per round thanSTRATA_DF_BRANCH.What changed
src/core/verify.cpp:g_side_mixer()reads the switch.Verifier::initcreates the two branch streams and theirfour events only when it is set. In
record_windowthe GDN gates and gate projection, and the QSA indexer and querysteps, are written as lambdas that take their stream (
idx_key,idx_snap,idx_append,idx_query,idx_select,q_chain). With the switch unset they run on the window's stream in the same order as before; with it set they run onthe branches.
STRATA_DF_BRANCH's branches are not used while it is set.include/strata/core/verify.hpp: the streams and events.Left alone: batch windows (each row reads its own slot's indexer, so they stay in line), the draft layer, the prompt
path,
STRATA_DF_BRANCHitself, and every path with the switch unset (no stream or event is created, the same launches inthe same order).
Extra Notes
How the default path was checked. exact7 (the CLI, seven fixed computations:
--adapt-swaps 0 --suffix-draft 0 --expert-cache 3500) on v0.1.4261b3fb5dand on this branch, both cards, the measurements' flags (above) plusSTRATA_PREFILL_CPU_SHARE=0on both (with the CPU prompt-read share on, short-prompt outputs depend on timing and the stock build does not repeat itself; with it off, stock v0.1.42 run twice gave the same seven hashes; the CLI does not use v0.1.42's tail skip, whose default is set only at serve start-up). With the switch unset: 7/7 identical to v0.1.42. WithSTRATA_SIDE_MIXER=2: 7/7 identical.61b3fb5d02deb3f102deb3f1(=)02deb3f1(=)6c9b2d106c9b2d10(=)6c9b2d10(=)ae3b089bae3b089b(=)ae3b089b(=)38e671a238e671a2(=)38e671a2(=)3e2193763e219376(=)3e219376(=)651f56c8651f56c8(=)651f56c8(=)a393ca0aa393ca0a(=)a393ca0a(=)The patch's own checks.
STRATA_SIDE_MIXER=2(table above): 7/7 identical to v0.1.42. Every step on a branch is the in-line kernel onanother stream, so the outputs are bitwise the same.
Upstream's tests (ctest, the whole suite, among them verify_parity, verify_batch_parity, spec_verify_parity, qsa_parity and spec_prob_test): v0.1.42 109/112 pass, this branch 109/112 pass; the same 3 fail on both (expert_parity, ple_parity, pool_test): they read model files from fixed paths that this PC does not have.
#905:
STRATA_DF_BRANCHis off by default on Linux because a stage's window with branches stopped inside its graph whenan NVML query came between requests (two RTX 3090s, two RTX 2080 Tis). This change uses the same fork and join inside the
captured graph, so the same risk may apply to layer-split stages. I have not seen it here:
STRATA_SIDE_MIXER=2has beenin my daily build since 2026-10-03 (Linux, open kernel modules, one stage on the RTX 5080 with the RTX 4060 Ti as the
helper expert cache), read by nvidia-smi every 15 minutes, and every second during 2-hour soaks.
Not tested:
verify.cppis compiled by the HIP builds, but this PC has no ROCm (installing it would be asystem change), so it is a draft. SYCL is its own tree and is not touched.
STRATA_SIDE_MIXER=1alone was not measured; the numbers are for=2.Measured and drafted with Claude Code.