Skip to content

Prefill regression on sm_86 (2x RTX A5000) in 0.1.30: ~-35% on ~2K-token prompts, not recovered by STRATA_PREFILL_STREAM_MIN / GDN_PIPELINE=0 / SCORES_MULTI=0 #340

Closed
Closed
Prefill regression on sm_86 (2x RTX A5000) in 0.1.30: ~-35% on ~2K-token prompts, not recovered by STRATA_PREFILL_STREAM_MIN / GDN_PIPELINE=0 / SCORES_MULTI=0#340

Description

@riverhh76
riverhh76
opened on Oct 1, 2026

Summary

Upgrading 0.1.28 → 0.1.30 regressed prefill throughput on 2× RTX A5000 (sm_86)
by ~35% on ~2K-token prompts, ~12% on ~16K, and ~2% on ~48K. Decode is unaffected
(long-gen 4096 tok: 71.5 → 71.6 tok/s; TTFT actually improved 1.3 s → 0.73 s).

All knobs I could find were tried; none recovers 0.1.28's small-prompt prefill.
Suspect a prefill-kernel regression specific to sm_86 + AVX-2 hosts (0.1.29's
"sampled answers up to 40% faster" / prompt-kernel changes measured on RTX 5070 / sm_120?).

Environment

  • 2× RTX A5000 24 GB (sm_86), layer-split auto, CUDA 13.0, driver 580
  • Host CPU: no AVX-512 (AVX-2 only), 20 logical cores, 124 GB RAM
  • Ubuntu 24.04, engine built from source (bbaaabb = 0.1.28 vs 30ec18e = 0.1.30)
  • Model: Qwen3.8-Flash-Next Coder IQ1_M native pack + PLE shard + MTP draft (rt/),
    expert-cache auto, expert profile: shipped coder profile (3584 entries) and a
    full-coverage 48x256 profile (12288 entries, regenerated via make_profile.py) —
    both give the same numbers

Config (identical for both versions)

--pack ... --native shard1 --ple-gguf shard2 --expert-profile <profile>
--expert-cache auto --prefill auto   (also tried: --prefill 8192 explicit)
--spec 4 --spec-min-p 0.70 --pcie-frac 0.00   (0.1.28-calibrated; 0.1.30 re-calibrated to the same values)
--mtp .../rt --max-context 262144 --kv int8 --kv-resident 32768 --layer-split auto
(0.1.30 additionally tested with: --vision + strata-vision, --ple-io ram)

Measurements (non-stream, timings.prompt_per_second, 3-run medians, salted prompts vs LCP cache)

prompt tokens 0.1.28 0.1.30 delta
1,840 1487 tok/s 883-973 tok/s -35~-38%
14,216 2143 tok/s 1860-1906 tok/s -11~-13%
42,495 2239 tok/s 2157-2225 tok/s -2%

Decode cross-check (4096-token long-gen, stream): 0.1.28 = 71.0-71.5 tok/s,
0.1.30 = 70.6-77.8 tok/s — unchanged or better. TTFT on the same long generations:
0.1.28 = 1302-1378 ms, 0.1.30 = 601-730 ms. So this is prefill-only.

Things ruled out

  1. --prefill 8192 explicit (vs auto) — no change
  2. STRATA_PREFILL_STREAM_MIN=8192 (restore pre-0.1.30 chunk floor) — no change
  3. STRATA_GDN_PIPELINE=0 + STRATA_SCORES_MULTI=0 (pre-0.1.29 kernels) —
    recovers ~16K slightly (1873), 2K still ~917
  4. Vision: measured with --vision + strata-vision (GPU/CPU modes) and fully
    disabled — no change
  5. Expert cache capacity: 0.1.28 filled 5120 slots, 0.1.30 auto-fills 3584 (draft
    head VRAM reserve); regenerated a full-coverage profile → CUDA0 3584 + CUDA1
    ~8550 ≈ all 12288 (layer,expert) pairs resident, hit rate 99.8-100% — still slow.
    Also tried --vram-reserve-mib 350 (crashes with a concurrent vision encoder).
  6. --ple-io ram — +3-4% prefill (kept), does not close the gap
  7. Re-calibrated on 0.1.30: best is still --pcie-frac 0.00 --spec-min-p 0.70
    (83.3 tok/s on the calibration prompts, same as 0.1.28's 83.1)

Guess

The 0.1.29/0.1.30 prompt-kernel changes (#187, #188, #207, GDN recurrence
pipelining ac6aad4, QSA block-scores multi a20f3b5, routed-only staging
cf0d12c) were measured on newer GPUs (5070 / sm_120 per the commit notes).
On sm_86 + AVX-2 the new path seems slower for small chunks. Happy to run
A/Bs or provide traces (--dump-routing, --stage-timing) if useful.

Repro

Same model pack (Coder IQ1_M), flags above, measure timings.prompt_per_second
non-stream on ~1.8K-token prompts, 0.1.28 vs 0.1.30.

Activity

  1. Niko1221 commented on Oct 1, 2026

    @Niko1221
    Niko1221
    on Oct 1, 2026
    Owner

    Thanks for the careful numbers. It isn't the streaming threshold (your STRATA_PREFILL_STREAM_MIN=8192 test shows that); the extra time is a fixed ~0.4-0.9 s per request, which fits the per-stage prompt loans 0.1.30 added for splits: each card lends cache slots and refills them one after another. We're reproducing it on a two-GPU box. Meanwhile, could you run 0.1.31 with --no-prefill-borrow --prefill 2048 (0.1.28's split behaviour) and post a log with STRATA_TRACE=1?

  2. Niko1221 commented on Oct 1, 2026

    @Niko1221
    Niko1221
    on Oct 1, 2026
    Owner

    Update: we reproduced it on a two-GPU box. Short prompts on a layer split lose ~24% on 0.1.30 and ~37% on 0.1.31 against 0.1.29 (16K prompts only 3-4%); one GPU is not affected. Causes: 0.1.30's per-card prompt loans (every request lends cache slots on each card and refills them afterwards) and, in 0.1.31, larger prompt buffers on a split. Correction to my earlier ask: please don't try --no-prefill-borrow on 0.1.31 - on a split it can stop the engine at start (the draft head no longer fits). A STRATA_TRACE=1 log of one 2K request is still useful. The fix is in progress for 0.1.32.

  3. Efs-O commented on Oct 1, 2026

    @Efs-O
    Efs-O
    on Oct 1, 2026
    Contributor

    On our box the fixed per-request prompt cost shows up on one GPU as well, and it is bigger there. That may be useful next to your two-GPU repro. Both runs are 0.1.30 prebuilt, STRATA_TRACE=1, and Qwen3.8-Flash-Next IQ3_S.

    Box: i9-9900KF, 128 GB DDR4-3000 dual channel. 2x RTX 5060 Ti 16 GB (sm_120) on PCIe gen3 x8/x8, Windows 10. Run with --expert-cache auto --prefill auto --spec 4 --kv int8 --kv-resident 32768 and the default --short-read 64.

    Loans the engine reports at start

    • 1 GPU: the prompt path borrows 2405 CUDA0 cache slots (4.56 GiB)
    • 2 GPUs (layer split auto): CUDA0 borrows 2149 slots (3.87 GiB), CUDA1 borrows 1927 of its 4307 (3.87 GiB)

    Test: a 23.8K-token prefix, then appended tails like an agent loop (tool result plus next turn). These are strata trace: read N tokens lines from the trace:

    tail tokens path 1 GPU 2 GPUs
    53-60 windows 1.03-1.42 s 0.84 s
    131-132 batched 2.99-3.03 s 2.09 s
    320-333 batched 4.43 s 3.34 s
    883-885 batched 6.34-6.40 s 5.2 s

    A linear fit over the batched rows:

    • 2 GPUs: about 1.55 s + 4.1 ms/token
    • 1 GPU: about 2.4 s + 4.4 ms/token

    So a 130-token tool result costs 2-3 s before the first token, on every agent round.

    My guess: the loan refill is bound by PCIe bandwidth. On gen3 x8, 4.56 GiB on one card takes longer than 3.87 GiB per card on two cards refilled in parallel. That would explain why your 1-GPU run looks unaffected on a faster link while ours does not. Prebuilt 0.1.30 has no --no-prefill-borrow, so I couldn't A/B it. I'm happy to rerun on 0.1.32 or a test build.

    --short-read sweep on 2 GPUs (switches the path to windows):

    • 256 gains only 0.2-0.5 s on ~130-token tails.
    • 1024 is much worse. Windows reads cost about 14.6 ms/token on 2 GPUs and about 20 ms/token on 1.
  4. Niko1221
    added 8 commits that reference this issue on Oct 1, 2026

    #340 prototype: a layer split refills every stage's prompt loan concu…

    f63817b

    #340 prototype: a split stage with room for its prompt buffers keeps …

    86d0acf

    #340 prototype: a split's streamed ring is 96 slots when most experts…

    0e24b61

    #340 prototype: STRATA_SPLIT_SMALL_MAX=M - a split request of at most…

    b3ae458

    #340: a layer split keeps 0.1.29's own prompt buffers (2048-token chu…

    bc65469

    #340: with the split's own prompt buffers (its rule, not --no-prefill…

    970bcbf

    #340: a split's own prompt buffers are opt-in (STRATA_SPLIT_OWN=1, or…

    f44f342

    Engine 0.1.32: split prompts faster with the same output (#340), AMD …

    7440789

  5. Niko1221 commented on Oct 1, 2026

    @Niko1221
    Niko1221
    on Oct 1, 2026
    Owner

    Fixed in 0.1.32: a layer split refills every card's prompt loan at once, uses a smaller streaming ring, and a card with free VRAM keeps its own prompt buffers - the same output as 0.1.31. On an R9700 + RX 9070 XT: 2K prompts 993 -> 1,270 tok/s, 16K 1,852 -> 1,980. STRATA_SPLIT_OWN=1 (opt-in) gives every card its own buffers (2K 1,470 there, slightly different output). Please reopen if your A5000 pair doesn't get back to 0.1.28's speed.

  6. Niko1221
    closed this as completedon Oct 1, 2026

  7. Efs-O commented on Oct 2, 2026

    @Efs-O
    Efs-O
    on Oct 2, 2026
    Contributor

    Retested on 0.1.34 (prebuilt, same box: i9-9900KF, 128 GB DDR4-3200, 2x RTX 5060 Ti 16 GB on PCIe gen3 x8/x8, IQ3_S, layer split auto). The short-tail cost on a layer split is unchanged from 0.1.30 on our gen3 link, and the prompt loans don't seem to be the cause here.

    Same replay as before: a ~20.9K-token prefix, then appended tails like an agent loop. Time to first token from the server's prompt_ms:

    new tokens 0.1.30 0.1.34 default 0.1.34 STRATA_SPLIT_OWN=1
    ~66 0.84 s 0.79-1.12 s 1.14-1.24 s
    ~130 2.09 s 1.72-2.58 s 1.87-2.70 s
    ~355 3.34 s 3.10-3.53 s 3.19-3.61 s
    ~850 5.2 s 4.94-5.06 s 4.77-4.85 s
    • The default run still borrows (CUDA0 2149 slots, CUDA1 1927 of 4307, 3.87 GiB each).
    • With STRATA_SPLIT_OWN=1 the log says the prompt path allocates its own buffers (too few cache slots to borrow), so nothing is borrowed. The tails cost the same, and a cold 20.9K prompt takes 38 s instead of 19 s.

    So on gen3 x8 the ~1.5 s fixed cost per batched read is still there with no loan to refill. My guess now is the per-run expert streaming over the slower link rather than the loan refill. Decode is fine: 60 tok/s short, 53 tok/s at 21K. I'm happy to run a STRATA_TRACE=1 log or a test build if that helps.

  8. rwkeyes
    added a commit that references this issue on Oct 3, 2026

    Merge tag v0.1.32 into avx1-floor

    47a246a

  9. 24 remaining items

  10. anon761
    added 5 commits that reference this issue on Oct 6, 2026

    Niko1221#340: a layer split keeps 0.1.29's own prompt buffers (2048-t…

    82c5559

    Niko1221#340: with the split's own prompt buffers (its rule, not --no…

    c3d77ee

    Niko1221#340: a split's own prompt buffers are opt-in (STRATA_SPLIT_O…

    481494c

    Engine 0.1.32: split prompts faster with the same output (Niko1221#340)…

    e8d66d3

    Niko1221#583: the byte-budget ring only where it gains; short prompts…

    0f7cea2
  11. fragtion
    added a commit that references this issue on Oct 7, 2026

    from upstream PR Niko1221#1190: generate: size every part's lend regi…

    29a8fcc
  12. evaanp
    added a commit that references this issue on Oct 9, 2026

    generate: size every part's lend region for a layer split by the serv…

    6d8f179
  13. Niko1221
    added a commit that references this issue on Oct 10, 2026

    generate: size every part's lend region for a layer split by the serv…

    51d55a9
  14. MistVVK
    added a commit that references this issue on Oct 12, 2026

    層の分割: 各段が prompt の経路に貸すスロットのエキスパートも RAM の写しに残す（upstream aea9dc3、51d55…

    354c71e
  15. lask3802
    added a commit that references this issue on Oct 12, 2026

    Remote stage: a bounded, cancellable reconnect; a worker's chunk; a r…

    4bbcb16
  16. k0valik
    added 4 commits that reference this issue on Oct 15, 2026

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): include/strata from ac922

    a23bcef

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): src/program from ac922

    1e26c8d

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): include/strata from ac922

    55297b8

    replay(hub.lumenfield.work-eelgaev-Strata-AC922-ac922): src/program from ac922

    17dda1e
  17. giridhart
    added a commit that references this issue on Oct 15, 2026

    generate: size every part's lend region for a layer split by the serv…

    755d19d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions