Activity

1. xiake-1 commented on Oct 12, 2026

   

   xiake-1

   on Oct 12, 2026

   Author

   More actions

   Follow-up measurements on the same frozen 256K prompt and the same box, prompted by #1964 / #1975 / #1922. Neither of the two switches helps here, and 0.1.42 is already slow — so whatever I hit is older than the 0.1.42 -> 0.1.43 window those reports point at.

   All runs: --max-context 262144 --kv int8 --kv-resident 32768 --prefill auto --expert-cache auto --remote-expert-opt --layer-split auto, 2x T10 (sm_75), one cold prompt read (cache_n=0) + 5x512 decode.

   engine | env | cold prefill tok/s | decode tok/s -- | -- | -- | -- 0.1.42 (61b3fb5) | default | 804.1 | 40.76 0.1.43 (16374b3) | default | 802.1 | 40.78 0.1.43 | STRATA_ADAPT_OVERLAP=0 | 797.4 | 40.72 0.1.43 | STRATA_LOOKAHEAD=0 | 797.9 | 40.70 0.1.43 | STRATA_ADAPT_OVERLAP=0 STRATA_LOOKAHEAD=0 | 797.1 | 40.18 0.1.40.1 (82f46a8) | default | 1042.5-1048.0 | 43.9-44.2

   So on this machine:

   • 0.1.42 == 0.1.43 (804.1 vs 802.1, within noise). The 0.1.42 -> 0.1.43 changes in V0.1.43 Decode performance degradation on 4070ti + 14700 + 64 RAM, Linux IQ3_XXS #1964/[Regression] v0.1.43 adaptive overlap reduces IQ3_S decode ~26% on RTX 5090 #1975 are not my regression; the boundary stays at 0.1.40.1 -> 0.1.40.2 (242 commits, 37 with a speed keyword).
   • STRATA_ADAPT_OVERLAP=0 and STRATA_LOOKAHEAD=0 cost 0.5-0.9% each here (noise level) and do not recover anything.
   • CPU prefill share and STRATA_MMVQ_IL (fork F4) were also ruled out earlier in the issue; prompt chunk (8192 / 384-slot ring) and pcie_frac (0.17 / 0.30) are identical on both the slow and the fast engine.
2. mmbeanz commented on Oct 12, 2026

   

   mmbeanz

   on Oct 12, 2026

   More actions

   Another NVIDIA split data point for the tail-skip default, on IQ3_S: RTX 5060 Ti 16 GB (gen5 x8) + RTX 3060 12 GB (gen3 x4), engine 7695859a (v0.1.43), --layer-split 40, 512K ctx, Linux/nspawn.

   Skip ON (default =7) vs STRATA_ROUTE_TAIL_SKIP=0, same config, same evening, 40 requests each:

   • decode +5.2% aggregate (means 71.10 vs 67.58 tok/s; 95% CI −2.9%..+13.3%, p=0.20) — the banner's "+10..20%" does not hold on this pair;
   • the gain decays with context: +9.3% at 4K, +4.8% at 16K, +1.0% at 32K — expert work is a shrinking share of the window as attention/KV dominate, which matters for agent traffic that lives at long context;
   • prompt path unaffected (32K: 1,720.8 vs 1,722.7); draft acceptance identical (68.9% both arms);
   • verdict on this box: keep the default ON (never slower, ~5% at short context).

   Your −1.4% on the T10 split vs our +5.2% here: both far from the advertised +10-20%, and plausibly the sign turns on how much of the decode window the expert stage actually costs (16 GB-eq cards missing experts vs a pair where the 3060's x4 link already dominates).

   A measurement trap for anyone A/B-ing this flag: hit_rate is not comparable across it. From our two engine logs: with skip ON, lookups per output token are 619.9 including the 98,217 skipped misses, but the reported denominator is 564.3 — the skipped misses are excluded. On a common denominator the sign reverses: skip ON measures 0.752 vs 0.775. So the Monitor's "expert cache hit" and /metrics hit_rate silently change meaning with the flag, and "hit rate improves with skip ON" is an artifact. Two doc asks: a line for /metrics/Monitor consumers, and a log line when the flag is OFF — our OFF run's log carries no tail-skip lines at all, so the off state is only inferable from absence.

   Full artifacts (per-request /metrics?requests=all, both engine logs with the skip counters, configs): #2083.

   Drafted with AI assistance; I reviewed every line and ran the measurements myself.
3. Niko1221 commented on Oct 12, 2026

   

   Niko1221

   on Oct 12, 2026

   Owner

   More actions

   I tried to reproduce this on 2x RTX A4000 (sm_86, not sm_75), IQ3_XXS, --kv-resident 8192, 131072 context, 32K and 100K prompts, 3 interleaved pairs: v0.1.40.1 vs v0.1.40.2 prefill is 1981 vs 1977 tok/s at 32K (-0.2%) and 2317 vs 2304 at 100K (-0.5%). STRATA_PREFILL_STREAM_AHEAD=0 and STRATA_STAGER_SLEEP=0 (the two prefill defaults that changed in that range on Linux) make no difference either. So the cause is specific to Turing or to the 256K length. Reading the diff I found no Turing default change: the Volta attention fix is sm_70 only, the PR #540 decode attention kernel runs below sm_75 (and on HIP) only, and the FP32 block-score kernel is opt-in. Two cheap tests on your T10s would narrow it: STRATA_PREFILL_STREAM_AHEAD=0 and STRATA_STAGER_SLEEP=0, each on a 32K prompt (a 256K run is not needed to see a 23% change). If both are neutral, a bisect on 32K prompts over the commits touching src/prefill and src/core/verify.cpp in 0.1.40.1..0.1.40.2 is the next step, and I can suggest the order.
4. xiake-1 commented on Oct 13, 2026

   

   xiake-1

   on Oct 13, 2026

   Author

   More actions

   Root cause found: it is the layer split --layer-split auto picks, not the prefill defaults

   Both switches you suggested are neutral (measured at 32K on 0.1.40.2: 1122.2 / 1124.7 / 1126.8 tok/s for default / STRATA_PREFILL_STREAM_AHEAD=0 / STRATA_STAGER_SLEEP=0, i.e. all inside ±2%; re-checked at 256K on the slow engine: 795.2 / 795.2 / 794.4). So they are not the cause — but neither is the 0.1.40.1 → 0.1.40.2 range.

   On this box (2× Tesla T10 16 GB, sm_75, layer split, driver 580.178.04, CUDA 13.0, IQ3_S, --max-context 262144 --kv int8 --kv-resident 32768 --prefill auto --expert-cache auto --remote-expert-opt --layer-split auto --spec 4 --spec-min-p 0.5, frozen prompts, serve + timings, ABBA, engines rebuilt from source per step):

   engine | 256K cold prefill tok/s | vs 0.1.40.1 | 32K cold prefill tok/s | 256K decode tok/s -- | -- | -- | -- | -- 0.1.40.1 | 1034.8 | — | 1119.4 | 46.3 0.1.40.2 | 1034.6 | 100.0% | 1101.5 | 45.1 0.1.41 | 796.0 | 76.3% | 944.9 | 37.7 0.1.42 | 795.7 | 76.9% | — | 40.0 0.1.43 | 795.2 | 76.8% | 937.9 | 39.5 0.1.43 + --layer-split 23 | 1033.8 | 99.9% | 1119.0 | 46.8

   A 2-GPU layer-split prefill is a pipeline, so the slower stage sets the rate: 25 layers → 25/25 = 100%, 34 layers → 25/34 = 73.5%; measured 795.2 / 1034.8 = 76.8% (the difference is hand-off overhead and non-uniform layer cost). That is the whole 23% — and the ~8–15% decode loss comes with it.

   Forcing the old split restores everything

   0.1.43 with --layer-split 23 comes up exactly like 0.1.40.1 and runs exactly as fast:

   代码块

   # 0.1.43 default (slow)
   layer split auto: K=14 - predicted 55.8 ms per decode window; ...
   layer split: CUDA1 runs layers 14-47, expert cache 4399 slots, 4399 of its 17408 profiled pairs
   KV streaming: 32768 of 262144 cells per QSA layer in VRAM, the K/V in 0.77 GiB of pinned RAM
   0.1.43 --layer-split 23 (restored)
   layer split across 2 GPUs: CUDA0, then CUDA1 (split 23)
   
   layer split: CUDA1 runs layers 23-47, expert cache 4433 slots, 4433 of its 12800 profiled pairs
   
   KV streaming: 32768 of 262144 cells per QSA layer in VRAM, the K/V in 1.29 GiB of pinned RAM

   256K: 795.2 → 1033.8 tok/s (0.1.40.1: 1034.8). 32K: 937.9 → 1119.0 (0.1.40.1: 1119.4). Decode: 39.5 → 46.8.

   Suggested next steps

   1. Skip the 0.1.40.1..0.1.40.2 bisect on 32K — there is no signal in that range at all.
   2. Workaround for the reporter: "layer_split": "23" in the config, or --layer-split 23. (It is box-specific — 23 is the balanced split for 48 layers over 2 cards — so it is a workaround, not a constant.)
   3. Worth a look in code: the auto-split cost model that arrived in 0.1.41. It predicts 55.8 ms for its own winner while the old model predicted 49.8 ms for K=23, and on this box K=14 is slower for both prefill (−23%) and decode (−15%). That reads like a misprediction in the chooser rather than an intended trade-off, and fixing it would save every user from pinning a layer count by hand.
5. Niko1221

   added a commit that references this issue on Oct 13, 2026

   layer split auto (CUDA): a stage's link scales its miss cost only aga…

   ...

   9d41e18
6. Niko1221 commented on Oct 13, 2026

   

   Niko1221

   on Oct 13, 2026

   Owner

   More actions

   Thanks @xiake-1, that settles it, and the table shows it clearly: the split, not the prefill defaults.

   The change that moves auto is 0391e47 in 0.1.41. Since then the search scales each card's miss cost by 20 GB/s / its probed link (clamped to 1..64). That came from 2x 4090 D with one card on a physical x1 link, where it took the prompt from ~350 to ~1400 tok/s. On your T10s it moved K from 23 to 14, so CUDA1 runs 34 of the 48 layers. A layer split reads a prompt as a pipeline, so the slower stage sets the rate, as you computed.

   In 0.1.44 (out now) the scale is taken against the fastest card's link instead of 20 GB/s, and only from a 4x gap on, capped at 64. A symmetric pair, or a moderate gap, keeps the fitted formula; the x1 case still gets about x15. STRATA_SPLIT_LINK_ABS=1 gives the 0.1.41 rule back.

   Could you post the two layer split auto: CUDAn link X GB/s -> miss cost xY lines from your start? If your links differ by 4x or more, this change does not move your split and we need another look; if less, 0.1.44 should pick 23 again on its own. Until then, --layer-split 23 is the right workaround.
7. jds1-ai commented on Oct 14, 2026

   

   jds1-ai

   on Oct 14, 2026

   More actions

   Two symmetric NVIDIA pairs, Linux, CUDA 13.3, driver 595.91.07, IQ3_XXS, --layer-split auto --max-context 131072 --kv int8 --kv-resident 32768, MTP spec 4. Three starts each: v0.1.43.1, v0.1.44, v0.1.44 with STRATA_SPLIT_LINK_ABS=1. Fresh setup.sh --build per version.

   2x RTX 3060 12 GB, both on Gen3 x16 (probe 12.3 GB/s each):

   start	miss cost CUDA0 / CUDA1	K	predicted window	decode median, n=9
   0.1.43.1	x1.62 / x1.62	26	78.0 ms	61.6 tok/s
   0.1.44	x1.00 / x1.00	26	74.0 ms	58.9
   0.1.44 + LINK_ABS=1	x1.62 / x1.63	26	78.0 ms	63.4

   2x RTX 5060 Ti 16 GB, both at Gen3 x8 (probe 6.9 GB/s each):

   start	miss cost	K	predicted window	decode median, n=9
   0.1.43.1	x2.90 / x2.91	24	46.9 ms	84.6 tok/s
   0.1.44	x1.00 / x1.00	23	40.1 ms	83.7
   0.1.44 + LINK_ABS=1	x2.90 / x2.91	24	46.9 ms	84.3

   All six starts came up, 9/9 short requests plus a 1217-token prompt each, finish=stop, engine rc=0, no Xid.

   Observations:

   1. The 20 GB/s rule prices both Gen3 pairs as the commit describes (20/12.3 = 1.63, 20/6.9 = 2.90), and 0.1.44 prints x1.00 on both stages. STRATA_SPLIT_LINK_ABS=1 reproduces the 0.1.43.1 K and scale lines inside the 0.1.44 build on both boxes.
   2. On a symmetric pair the old rule does not move K. This follows from the window cost: layer_ms depends only on SM count and clock, so with identical cards the compute term is constant in K, and an equal scale on both stages multiplies the whole K-dependent cost without changing its argmin. The 3060 pair read K=26 on all three starts. The 5060 Ti pair moved one layer (24 vs 23) with scales of x2.90 and x2.91, a 0.3% probe difference breaking a near-tie.
   3. So the T10 shift from K=23 to K=14 under the same rule requires the two T10 links to have probed at different speeds. The two "link ... -> miss cost" lines from that rig would settle it; equal readings there would point to a different cause.
   4. Decode is flat across the three starts on both pairs. The 3060 pair ran the same placement (K=26) three times and read 58.9 to 63.4 tok/s, so differences under about 7% between single runs on that pair are run-to-run spread.
   5. The x1.00 lines here come from the symmetric branch and do not exercise the 4x threshold. A Gen3 x16 + Gen2 x4 pair is planned for that; it depends on hardware moves not yet made.

   Logs for all six starts are available on request.
8. xiake-1 commented on Oct 14, 2026

   

   xiake-1

   on Oct 14, 2026

   Author

   More actions

   Here are the two lines, from the 0.1.44 and 0.1.43 starts on this box (2× Tesla T10 16 GB, sm_75, one layer split; every engine below built from source here with -DCMAKE_CUDA_ARCHITECTURES=75):

   0.1.44 default (your new rule — relative to the fastest stage's link, 4× gap or more):

   代码块

   strata generate: layer split auto: CUDA0 link 6.2 GB/s -> miss cost x1.00
   strata generate: layer split auto: CUDA1 link 10.9 GB/s -> miss cost x1.00
   strata generate: layer split auto: K=23 - predicted 49.7 ms per decode window; ...

   0.1.44 with STRATA_SPLIT_LINK_ABS=1 (the 0.1.41 rule, for contrast):

   代码块

   strata generate: layer split auto: CUDA0 link 6.1 GB/s -> miss cost x3.25
   strata generate: layer split auto: CUDA1 link 10.8 GB/s -> miss cost x1.85
   strata generate: layer split auto: K=14 - predicted 55.8 ms per decode window; ...

   0.1.43 default (0.1.41 rule) gives the same x3.25 / x1.85 and the same K=14.

   Across every start I have logs for, the probes are stable: CUDA0 6.0–6.2 GB/s, CUDA1 10.8–10.9 GB/s — a gap of 1.76–1.79×, i.e. under your 4× threshold, so the fitted formula stays and 0.1.44 picks 23 again on its own. Your change does move my split, so no second look needed on that account.

   The STRATA_SPLIT_LINK_ABS=1 arm on the same 0.1.44 binary reproduces the 0.1.41 behaviour exactly (x3.25/x1.85 → K=14), which isolates the difference to that one change and rules out anything else in 0.1.44 moving the placement. I also checked v0.1.44.1's delta — Responses/Codex, setup and nvfp4 material, nothing that touches the split.

   0.1.44 restores the old path on its own

   Frozen prompts, one cold read (cache_n=0) + 5×512 warm, ABBA order, same box and hour, engines swapped one at a time:

   engine (256K prompt) | cold prefill tok/s (2 runs) | median | vs 0.1.40.1 | decode tok/s -- | -- | -- | -- | -- 0.1.40.1 (auto K=23) | 1042.6 / 1030.9 | 1036.8 | — | 45.2 0.1.43 (auto K=14) | 795.0 / 791.9 | 793.5 | 76.5% | 38.8 0.1.44 (auto K=23) | 1030.5 / 1029.9 | 1030.2 | 99.4% | 45.5 0.1.44 + --layer-split 24 | 1044.0 / 1043.7 | 1043.8 | 100.7% | 44.5

   So with the new rule auto lands on 23, the −23% prefill and the ~15% decode loss are both gone, and --layer-split is no longer needed as a workaround.

   One thing worth keeping in mind for this box: when I swept the split explicitly, an even 24/24 was a little faster than 23/25 — 256K: 1043.8 vs 1030.2; 32K: 1144.1 vs 1120.3. So auto landing on 23 is the right call and restores the old path, and a T10 user chasing the last 1–2% can still pass --layer-split 24.
9. MistVVK

   added a commit that references this issue on Oct 14, 2026

   層の分割の自動の配置: リンクの遅い段の外れの重みは、最速の段との比が 4 倍からだけ掛ける（upstream 9d41e18、Niko1…

   ...

   73e9bdd
10. giridhart

    added a commit that references this issue on Oct 15, 2026

    layer split auto (CUDA): a stage's link scales its miss cost only aga…

    ...

    ccce332
11. jds1-ai commented on Oct 17, 2026

    

    jds1-ai

    on Oct 17, 2026

    More actions

    The above-gate data promised in observation 5. One box, two matched pairs in turn, each with one card in the CPU slot and one in the chipset slot (Gen2 x4). Same engine build (0.1.44, ed4773e, source build), same pack.

    Box: MSI Z97 (i7-4790K, 4C/8T, 32 GB DDR3), Linux 6.8.0-147, NVIDIA open kernel module 580.178, CUDA 13.0. Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS native pack, --layer-split auto --max-context 32768 --kv int8 --kv-resident 32768 --resident-budget-gib 12 --spec 4 --mtp. Differences from our 10-14 runs: IQ2_XS instead of IQ3_XXS and a 32K context, both so the pack fits 32 GB of host RAM (with the default full RAM copy of the experts the engine is OOM-killed at load on this box; setup.py lists 48 GB for this pack, and --resident-budget-gib 12 is what makes it start).

    Pair 1: 2x RTX 5060 Ti 16 GB, Gen3 x8 + Gen2 x4 (sm_120 build; 3 auto starts plus 1 with STRATA_SPLIT_LINK_ABS=1) [n4-a-auto-s1/s2/s3-1015T23*, n4-a-abs-1015T233323]:

    layer split auto: CUDA0 36 SMs at 2.69 GHz -> 0.75 ms per layer
    layer split auto: CUDA1 36 SMs at 2.69 GHz -> 0.75 ms per layer
    layer split auto: CUDA0 link 6.5 GB/s -> miss cost x1.00
    layer split auto: CUDA1 link 1.6 GB/s -> miss cost x4.17        (x4.18, x4.19 on the other two starts)
    layer split auto: K=33 - predicted 38.8 ms per decode window; the caches hold 15876 of 24576 profiled pairs (~98.6% of the routed mass)
    layer split auto: NEAR TIE - K=36 is predicted 39.3 ms per decode window, within 1.3% of K=33
    

    With STRATA_SPLIT_LINK_ABS=1 on the same binary: x3.06 / x12.81, K=33 - predicted 44.7 ms, near tie K=36 at 3.3%.

    Pair 2: 2x RTX 3060 12 GB, Gen3 x16 + Gen2 x4 (sm_86 build of the same commit; auto starts) [n4d-d-auto-s10-1016T144018 and siblings]:

    layer split auto: CUDA0 28 SMs at 1.79 GHz -> 1.45 ms per layer
    layer split auto: CUDA1 28 SMs at 1.81 GHz -> 1.43 ms per layer
    layer split auto: CUDA0 link 10.7 GB/s -> miss cost x1.00
    layer split auto: CUDA1 link 1.5 GB/s -> miss cost x7.02
    layer split auto: K=39 - predicted 75.5 ms per decode window; the caches hold 10182 of 24576 profiled pairs (~96.7% of the routed mass)
    layer split auto: NEAR TIE - K=42 is predicted 76.4 ms per decode window, within 1.2% of K=39
    layer split: CUDA0 runs layers 0-38 / CUDA1 runs layers 39-47
    

    Across the 21 starts on this pair CUDA0 probed 10.5-10.7 GB/s and the scale read x6.86-7.02; K=39 and 75.5 ms on every start. No LINK_ABS start on this pair.

    Observations:

    1. Above the 4x gate the relative rule prices the slow stage at the probe ratio and nothing else: x4.17-4.19 = 6.5 / 1.56 on pair 1 and x6.86-7.02 on pair 2, the probe ratio of each start. The slow card received 15 of 48 layers on the 5060 Ti pair and 9 of 48 on the 3060 pair, fewer than the fast card in every start. Pair 1 sits 4-5% above the gate; the ratio was stable to 0.5% over four starts, so the branch did not flip, but a 4% drop in the ratio would read it as symmetric (x1.00 / x1.00) and change K. A margin or hysteresis at the threshold would remove that.

    2. On matched cards the two rules cannot disagree on K, and they did not: LINK_ABS=1 also picked K=33 on pair 1. In the window cost the compute term is constant in K when both cards have the same SM count and clock, and the miss term is weighted (1, r) under 0.1.44 and (20/b0)(1, r) under the old rule, so the argmin is the same. This is the same argument as our symmetric runs (K unchanged on both pairs there), now checked above the gate. So the K=23 to 14 shift on the T10s required the two links to probe differently, which the lines posted above (6.2 and 10.9 GB/s) show.

    3. The probe is host-sensitive, and the absolute rule's price moves with it. The x8 5060 Ti probes 6.5 GB/s in this Haswell/DDR3 box and 6.9 GB/s in a Xeon box with the same card model in a Gen3 x8 slot; the x16 3060 probes 10.5-10.7 here and 12.3 there. The probe is an end-to-end host-to-device copy, and a slow host lowers the readings more on the wider link (-6% at x8, -14% at x16), so the old rule's price for the same card model changes with the box (20/6.5 = x3.06 here against x2.90 there) while the relative rule's price moves less, since both links on a box share the host.

    4. Hand sweep around the chosen K on pair 1, nine ~105-token requests of 512 tokens per cell, two runs of most cells [n4pp-app-sweep-K*-1016T00*, n4q-q-*-1016T09*]. ms per verify pass at matched draft acceptance (tok/pass 2.0, fitted per cell), then pooled decode tok/s:

    K	ms per pass at 2.0 tok/pass	pooled decode tok/s	28.7K-token prompt, prefill tok/s
    24	27.2 / 26.8	73.5 / 75.2	229 / 283
    29	25.3 / 25.5	79.6 / 79.8	242 / 224
    33 (auto)	27.3 / 27.7 / 27.9	73.6 / 76.2 / 73.9	269* / 288
    36	33.8 / 31.7 (second fit unstable, intercept -18)	58.8 / 61.4	235 / 151
    41	44.9	44.3	not completed

    * The first auto K=33 start's long prompt was refused (HTTP 400, context), so 269 is the forced --layer-split 33 start and 288 the second auto start.

    Between-run spread at the same K is 2% or less on the matched-pass column at K=24, 29 and 33 (6% at K=36, where the second fit is unstable) and up to 24% on the prompt column (K=24: 229 against 283), so the prompt column does not separate K=24 from K=33 here. On the pass column K=29 is 7-9% faster than the auto choice in both runs, K=24 ties it, and K=36 and K=41 are 17-42% slower on decode tok/s: at K >= 36 the misses exceed the 12 GiB RAM budget and spill to the file tier (the RAM budget load line reads 11.54 GiB of GPU-missed experts at K=24 and 33, all held, against 13.31 GiB at K=36 and 16.80 GiB at K=41, so 1.3 and 4.8 GiB of them are read from the files). So on this pair the rule's K=33 is 4 layers from the measured best and on the side that keeps the misses in RAM; the near tie it printed (K=36) is the worse direction. The predicted window ranks K=33 ahead of K=36 correctly (38.8 against 39.3 ms) but the measured gap is 20%, since the prediction prices decode misses at link speed and does not know when the host runs out of RAM for them.

    5. Pair 2's sweep at --prefill auto (K=24, 35, 39, 42) measured the SSD, not the split: on this box the prompt path re-reads the lent experts from the drive on every 8192-token chunk (page cache too small for the chunk's set; filed as [Linux, 2x RTX 3060, 32 GB RAM, SATA SSD, 0.1.44] --prefill auto picks 8192 by VRAM alone; lent slots + file tier exceed the page cache, so a 28.7K prompt re-reads 28-45 GB (TTFT 164-287 s); --prefill 4096 fits: 0.9-1.4 GB read, TTFT 36-38 s #2824). The paired comparison was therefore rerun at a fitting seat, --prefill 4096 --ple-io mmap --resident-budget-gib 16: --layer-split 24 against auto (K=39), with a second auto start at the same seat earlier the same day as the between-run control [n4d-d-k24-b16-pf4096-mmap-reps-1016T223628, n4d-d-auto-b16-pf4096-mmap-reps-1016T224857, n4d-d-auto-s10-b16-pf4096-mmap-reps-1016T211559]. At K=24 the x4 card holds 4617 of its 12288 profiled pairs; at K=39 it holds 4586 of 4608. The drive share was the same on all three cells (0.8-0.9 GB read per short request), so it pairs out. Same 9 short requests and the same 28.7K-token prompt, three reps, as in observation 4.

    	K=24	auto K=39	auto K=39, second start
    28.7K-token prompt, reps 2-3: prefill tok/s (engine-side prompt time)	393 / 400 (73.0 / 71.8 s)	770 / 746 (37.2 / 38.5 s)	769 / 798 (37.3 / 36.0 s)
    short requests, pooled decode tok/s	35.6	36.5	39.6
    short requests, ms per pass at 2.0 tok/pass	57.4	55.8 (fit unstable, intercept -74)	50.6
    decode at 28K depth, reps 2-3, tok/s	52.9 / 59.7	34.6 / 43.4	49.9 / 48.4

    The rule's K=39 halves the long-prompt time (1.9-2x the prefill rate by rep, against both auto starts) and is 3-12% faster on short-prompt decode, winning 7-8 of the 9 prompt-by-rep pairs; one short request on the paired auto start ran at 19.9 tok/s directly after a 308 MiB of VRAM free warning (#2065), which is what unsettles that cell's fit. At 28K depth the 256-token decode went the other way: K=24 was faster on both reps against both auto starts, per pass as well as per token, so draft acceptance is not the cause. That line has two reps per cell and the two auto starts differ by 21% on it, so it gives a direction, not a size. We have not tested why. The rule prices expert misses and not the attention work of the layers each card runs, and at 28K that work is not small; a sweep at depth would separate the two. Rep 1 of the long prompt (cold fill) is in the logs and not scored: 317 / 453 / 486 prefill tok/s.

    6. Decode at the auto K, warm, pair 1: 93-102 tok/s at 28K depth after a warm-up request, 1.1K tok/s prompt on a warm cache [n4w-w-auto-1016T121634, n4g-g-auto-1016T122703]. Not a rule result; given for scale.

    Not tested: LINK_ABS=1 on the 3060 pair; gaps between 1.8x and 4.2x; the 4x threshold with unmatched cards (where the compute term is not constant in K and the rules can differ).

    Logs for every start (engine log, argv, per-request JSON) are available on request.

    ---

    Posted by jds1-ai. Runs by an agent (Fizz, Claude model); source reading, pre-registered scoring and this write-up by a second agent (Fable, Claude model); reviewed by the account owner before posting.
12. Niko1221

    added a commit that references this issue on Oct 17, 2026

    Layer split auto: STRATA_SPLIT_UNIFORM=1 restores the uniform miss cost

    ...

    b884451
13. jds1-ai commented on Oct 17, 2026

    

    jds1-ai

    on Oct 17, 2026

    More actions

    Follow-up to our comment of 2026-10-17 (#2007 (comment)), covering the item it listed as not tested: the 4x threshold with unmatched cards, where the compute term is not constant in K and the two rules can pick different K.

    Box: the same MSI Z97 (i7-4790K, 32 GB DDR3, Linux 6.8.0-147, NVIDIA open kernel module 580.178, CUDA 13.0), same engine build (0.1.44, ed4773e, source build with CMAKE_CUDA_ARCHITECTURES=86;120), same pack (Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS). Cards this time: CUDA0 = RTX 3060 12 GB in the CPU slot, CUDA1 = RTX 5060 Ti 16 GB in the chipset slot. LnkSta read under load during the LINK_ABS=1 cell's decode: 01:00.0 at 8 GT/s x16, 02:00.0 at 5 GT/s x4. Flags are the fitting seat from observation 5, --prefill 4096 --ple-io mmap --max-context 32768 --kv int8 --kv-resident 32768 --spec 4 --mtp, with --resident-budget-gib 15 (16 left 91 MB in swap after load on this box; 15 and vm.swappiness=10 were used for every cell here). One fresh engine start per cell, the same three prompts as before: a 7.7K-token warm-up, nine ~105-token requests of 512 tokens, then the 28.7K-token prompt three times with 256 completion tokens.

    Split lines, --layer-split auto [n4c-c-auto-b15-pf4096-mmap-reps-1017T101750], trailing clauses trimmed where marked:

    layer split auto: CUDA0 28 SMs at 1.79 GHz -> 1.45 ms per layer, …
    layer split auto: CUDA1 36 SMs at 2.60 GHz -> 0.77 ms per layer, …
    layer split auto: CUDA0 link 10.6 GB/s -> miss cost x1.00
    layer split auto: CUDA1 link 1.6 GB/s -> miss cost x6.80
    layer split auto: K=29 - predicted 63.6 ms per decode window; the caches hold 13212 of 24576 profiled pairs (~98.0% of the routed mass), …
    layer split auto: NEAR TIE - K=26 is predicted 63.8 ms per decode window, within 0.4% of K=29 (63.6 ms); the cards differ, so the prediction may not rank them (it prices decode, not the prompt). …
    

    With STRATA_SPLIT_LINK_ABS=1 on the same binary [n4c-c-linkabs-b15-pf4096-mmap-reps-1017T102815]: x1.88 / x12.87, K=33 - predicted 68.2 ms, near tie K=30 within 1.7%.

    Observations:

    1. On unmatched cards the two rules differ, as the argument in observation 2 said they could. The matched 3060 pair probed the same link ratio (x6.86-7.02 there, x6.80 here) and chose K=39; this pair chooses K=29 under the relative rule and K=33 under LINK_ABS=1. The ratio is the same within 2%, so the 10-layer move from 39 comes from the compute term (1.45 against 0.77 ms per layer), and the absolute scales, which raise both link costs about 1.9x against the fixed per-layer time, move K four layers toward the slow card.

    2. Hand sweep around the two picks, same prompts, one start per cell. CUDA1 slots is the engine's expert cache N slots ... N of its M profiled pairs line for the slow card; CUDA0 slots is its expert cache N slots line (the engine prints no CUDA0 pair count). Short requests: median over the nine, min and max in brackets. Long prompt: reps 2 and 3 (rep 1 is the cold fill and is not scored).

    cell	K	CUDA1 slots of its pairs	CUDA0 slots	short requests, decode tok/s	28.7K prompt, prefill tok/s (prompt time)	decode at 28K depth, tok/s
    --layer-split 25	25	7419 of 11776	5712	60.5 (53.3-64.9)	610 / 613 (47.0 / 46.9 s)	76.2 / 80.0
    auto	29	7433 of 9728	5659	53.6 (38.2-64.8)	849 / 825 (33.8 / 34.8 s)	59.0 / 69.1
    LINK_ABS=1	33	7373 of 7680	5632	53.0 (50.9-57.6)	921 / 963 (31.1 / 29.8 s) †	66.9 / 76.7 †
    --layer-split 33	33	7373 of 7680	5632	53.5 (50.8-54.4)	955 / 934 (30.0 / 30.7 s)	73.8 / 71.3
    --layer-split 37	37	5632 of 5632	5562	44.0 (30.9-48.4) ‡	819 / 847 (35.0 / 33.9 s) ‡	47.2 / 55.6 ‡

    † During this cell's long-prompt reps 2-3 the kernel wrote 663 MB of swap to the SSD that holds the model, against 1,178 MB the prompt read from it. The --layer-split 33 cell is the same split without that, so its prompt column is the one to read; the two cells' short-request columns agree within 1%.
    ‡ This cell paged during the scored phases: 19.8k swap-ins during the short requests against 3.2-3.3k in the K=25 and K=33 cells, and 1.9 GB of the engine's heap was written out during them. Its numbers are reported, not ranked.

    Spread inside this sweep: LINK_ABS=1 and --layer-split 33 are the same split. Their expert-cache hit and lookup counts are identical on the warm-up and the first two short requests and diverge from the third onward; their short-request medians differ by 1.0%, which is the between-run spread at a fixed split here. Rep 2 against rep 3 on the long prompt differs by 0.3-3.0% in the unflagged cells. On decode at depth the auto cell's two reps differ by 17%, so that column gives a direction, not a size, below that.

    Expert-cache lines by cell: CUDA0's slot count falls as K rises (5712 at K=25 to 5562 at K=37; the prompt path borrows about 1,530 CUDA0 slots, 2.03 GiB, in every cell). Over the nine short requests the engine's decode expert cache hit rate reads 94.9-97.3% at K=25, 93.4-96.0% at K=29 (median 95.7), 90.6-95.6% at K=33 (91.1-95.5% in the LINK_ABS=1 cell) and 84.9-90.9% at K=37 (median 88.2). Its share of routed experts read over PCIe is 0.2-0.6%, 0.4-1.1% (median 0.5), 0.5-1.6% and 1.1-2.3% (median 1.7) in the same order.

    3. On this pair the relative rule's K=29 is not the best cell on any column. --layer-split 33, the LINK_ABS=1 pick, matches it on short-request decode (within 1%) and is 11-13% faster on prompt time, prefill rate and decode at depth, on all four long-prompt reps, outside the 3% rep-to-rep spread. The near-tie line's own caveat applied: the rule's two candidates were a decode tie and the prompt path separated them.

    4. --layer-split 25 is 13% faster than auto on short-request decode and 22% faster at 28K depth (its two reps are both above both of auto's), and 37% slower on the prompt (610 against 849 tok/s). This is the same shape as pair 2 (observation 5, K=24 against K=39): the lower K wins decode, the higher K wins the prompt, now on a second pair with different cards and an auto pick 10 layers lower.

    5. --layer-split 37, the K the link term alone would choose on this ratio (the matched 3060 pair chose 39), is 18% slower than auto on short-request decode and 20% slower at depth, and ties on the prompt. It is also the cell with the fewest CUDA0 slots, a median hit rate 7.5 points under auto's, about 3.4 times auto's over-PCIe share on medians, and the paging in ‡. So its penalty includes CUDA0 misses served from the file tier, and this sweep does not separate that from the compute term the rule prices.

    6. The auto cell's short-request spread (38.2-64.8) comes from three requests: two ~105-token prompts that the engine took 5.3 and 7.2 s to read (5.6 and 7.3 s to first token at the client), and one that decoded at 38.2 tok/s. No VRAM warning precedes any of them; the load-time lines (446 MiB of VRAM free, CUDA1: 330 MiB ... LOW) are the same in every cell except K=37 (2,894 MiB free on CUDA1). The auto cell's short-request phase had 9.8k swap-ins against 3.2-3.3k in the K=25 and K=33 cells; it was the first cell after swap was cleared. Medians are quoted for that reason.

    Not tested: K between 29 and 33 or below 25; more than one start per hand cell (the K=33 pair is the only replicate); the reverse arrangement, 16 GB card in the CPU slot and 12 GB card on the x4 link. Seat differs from pair 2 by --resident-budget-gib 15 against 16.

    Logs for every start (engine log, argv, per-request JSON, /proc/vmstat and /proc/<pid>/status samples per phase) are available on request [n4c-c-{auto,linkabs,k25,k33,k37}-b15-pf4096-mmap-reps-1017T10*].

    ---

    Posted by jds1-ai. Runs by an agent (Fizz, Claude model); source reading, pre-registered scoring and this write-up by a second agent (Fable, Claude model); reviewed by the account owner before posting.
14. Niko1221 commented on Oct 19, 2026

    

    Niko1221

    on Oct 19, 2026

    Owner

    More actions

    Thanks for the report. This is fixed in 0.2.0-beta (or an earlier release), and we're closing it as part of the tracker reset for 0.2.0-beta.

    If it still happens on 0.2.0-beta, add the area prefix to the title ([Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other]), put the Strata version and your hardware in the issue form fields, and comment /reopen. A check runs and reopens it.
15. Niko1221

    closed this as completedon Oct 19, 2026
16. DmitriyW9

    added a commit that references this issue on Oct 19, 2026

    Layer split auto: STRATA_SPLIT_UNIFORM=1 restores the uniform miss cost

    ...

    6629cdf