Layer split with a small second GPU caps prompt chunk at 512: 6.2x slower prompt reads, same decode #448

Environment

Engine 0.1.31, repo HEAD 9259cad (2026-10-01). Measured 2026-10-01 on:

• RTX PRO 4500 Blackwell 32 GB (82 SM, CUDA arch 120) + RTX 3080 10 GB (68 SM, arch 86)
• Threadripper PRO 5975WX, 499 GB RAM, Linux
• model qwen3.8-flash-next-ud-q4_k_xl (Unsloth UD-Q4_K_XL split GGUF, 71.73 GiB routed experts
  loaded into the RAM arena), --max-context 262144 --kv int8 --spec 4 --mtp --expert-cache auto

a small second GPU in a layer split collapses prompt chunk auto to 512 tokens (~6x slower prompt reads)

Symptom. Same model, same weights, same flags; the only difference is whether the 3080 takes a
slice of the layer split:

Decode is unchanged (±1 %). Prompt reads differ by 6.2x. On an agent workload — long prefixes,
modest generation, most requests re-reading 100-200K tokens — the pair arm is much worse overall,
so the small card's residency gain is not merely neutral, it hides a large prompt regression.

Cause. src/program/generate.cpp:3663-3682: the chunk is the largest candidate every
participant can fund, and a participant is each stage's own expert cache:

auto fits = [&](int64_t c, bool cap) -> bool {
    for (const PfPart& p : pf_parts) {
        const int64_t k = part_slots(p, c);
        if (k <= 0 || k + 128 > p.cache->slots()) return false;
        if (cap && k * 100 > kAutoLendPct * p.cache->slots()) return false;
    }
    return true;
};

CUDA1's 512-slot cache must keep the 128-slot floor, so it can fund at most ~384 slots of prompt
staging — the binding constraint for every candidate above 512 tokens. CUDA0's 8,566 slots never
get a say. The comment above fits() states the assumption this exposes:

the chunk is the largest one EVERY participant can lend (a smaller chunk only reads slower)

which is right for comparable cards and wrong when a participant is a small helper card.

Worth stating precisely, because it is easy to misdiagnose: an explicit --prefill is not
saved by the percentage cap (fits(c, false) — the auto scan alone applies kAutoLendPct, 85 % or
90 % depending on Prefill::pinned_share()). --prefill 4096 on the pair arm halves down to 512
through the same slot floor and says so:

strata serve: prompt chunk 4096 -> 512 tokens so its buffers fit in every expert cache

So there is currently no operator setting that keeps a large chunk while the small card is in the
split — the only lever is taking it out of the split, which is a bigger change than tuning.

Suggested fixes, roughly in order of how little they disturb the current behaviour:

1. When the candidates are scanned, weight each participant by its share of the loan rather than
   demanding that the smallest cache fund the whole chunk — the large stage can carry most of the
   staging.
2. Or treat a participant whose cache is far below the others (say under 15 % of the largest) as
   unable to lend, letting it stream during the prompt instead of capping everyone.
3. At minimum, name the constraint when the fallback bites, e.g. prompt chunk auto: 512 tokens (CUDA1's 512-slot expert cache cannot fund more), so a 6x prompt regression is traceable in one
   log line instead of needing an A/B against a single-card arm. docs/SECOND_GPU.md and
   docs/DETAILS.md describe split residency and prompt lending separately; neither notes that the
   two interact through this floor.

Not a bug, for the record: --expert-cache-device1 on that same small card is the good use of
it — one stage plus a 1,800-slot tier left prompt chunk auto: 8192 intact and raised decode from
84.8 to 91.1 tok/s at 167K context (hit rate 69.6 % -> 73.6 %). The tier's own cost is the
per-request synchronisation the docs warn about (host 74 ms staging+launching, 124 ms waiting for it), and it grows with size. 1,800 vs 2,600 slots, requests of similar size:

Residency rose 4.4 points and decode did not (87.7 tok/s at 182K context vs 91.1 at 167K — the
contexts differ, so the wait column is the trustworthy comparison, and it moved 2.4x on 1.5x the
launches). Later, larger requests on the 2,600 tier waited 416-597 ms. The tier has a sweet spot
well below "as much as fits", which is worth a sentence in docs/SECOND_GPU.md.