[gopinath87607]

…ranks' prompt rows

Three things a layer split was paying for and not using.

• THE SESSION CARVE. A stage carved state for ALL 48 layers whatever slice it ran - every QSA KV pool at max_cells, every GDN recurrence - the whole-model-sized-buffer disease the chunked-QSA-prefill PR cured upstream. session_bytes/session_init take [layer_lo, layer_hi), and a session now owns only that range: qsa_states/gdn_state keep GLOBAL ordinal indexing and simply hold fewer rows (qsa_ord0/qsa_alloc, gdn_ord0/gdn_alloc), per-layer consumers subtract gdn_ord0, and the model-level uses - the shared RoPE table, the prefill staging identity, the MTP rope borrow - read qsa_states[qsa_primary()]. A range with no QSA layer still carves ONE primary state so those uses stay valid, and the default full range is byte-identical to the old carve. On the 2x3060 + 2x5060 rig, CUDA1's [30, 48) session costs 0.93 GiB: 7.83 GiB free before the carve, 6.90 GiB after (session [30, 48) and the head).

• PER-STAGE PROMPT LOANS. no_prefill_borrow = true for split stages said "each stage's prompt path has its own buffers" - true, but that is a reason to give each stage its own LOAN, not to make every stage withhold a chunk-sized reserve from its cache for the whole session, and forced on it also collapsed --prefill auto to
  2048. Each prompt path now borrows the tail of ITS OWN stage's cache (PfPart), so outside the prompt the whole cache is expert cache: --prefill auto is the largest chunk every participant can lend (8192 tokens here), a participant refills its loan into its own slots and marks only its own layers (a slot refilled into another cache, or another stage's rows marked resident, is silent and produces plausible tokens), and a cache that cannot lend even a 256-token chunk is refused with both numbers rather than over-subscribing a card whose cache has already filled its VRAM. The flat 1 GiB withheld from every stage after the first becomes what is actually held back after the weights, session and drafter exist: 96 MiB of verify windows, plus 1000 MiB of drafter and head on the ONE stage that carries them. CUDA0 lends 2162 slots (3.98 GiB), CUDA1 1993 of its 2653 (3.97 GiB).

• THE HELPER RANKS' PROMPT ROWS - SHIPPED OFF, AND MEASURED AS A LOSS. RemoteExperts::has/begin_rows/ prompt_input/prompt_output and kBatchRows let a helper rank (CUDA2/3) compute prompt rows for the experts it already holds, so those blobs are never staged to a stage; Prefill::set_remote drives it behind STRATA_PREFILL_REMOTE (STRATA_PREFILL_REMOTE_SHARE gradients it by dropping a hash-scattered share), the staging is one 4096-row batch per rank (CAP was MAXT*k for decode, now one prompt batch: ~42 MiB pinned and ~54 MiB of device scratch per rank), and generate.cpp hands the ranks to exactly ONE prompt path - a rank owns one set of buffers and one stream, and a split runs the next stage on its own thread.

  IT LOSES. 23,420-token prompt, measured on the 0.1.24 tree this was developed against: 38.1 -> 83.4 s of wall, 2.2x slower. Two causes, both measured: the 3.5M rows batch into 858 fully serialized 4096-row calls at ~52 ms each where the transfer is ~7 ms (1.5 GB/s against an 11.0 GB/s probe), and STRATA_REMOTE_ZEROCOPY (default on) makes the helper quantize from, and write into, BAR1-mapped host memory 41.9 MB at a time; turning it off alone recovers 26 s of timeline (76.9 -> 51.1 s) without closing the gap. It is therefore OFF by default and nothing in the shipped path changes.

  IT ALSO CORRUPTS UNDER MMQ, SO THAT COMBINATION IS NOW REFUSED. MMQ computes MMQ_GROUP=16 experts per launch and flushes the group's one GEMM from the compute call of the group's LAST member, so an expert this stage skips because a helper holds it silently takes its whole group's output with it - a third to a half of a layer's experts missing reads as fluent garbage (! x 64). With the per-expert path (STRATA_PREFILL_MMQ=0) the two arms are BIT-IDENTICAL (sha256(reasoning) equal, 542.1 vs 549.3 tok/s), which is what proves the row staging itself is exact. Prefill::init now prints the reason and drops the ranks instead of corrupting.

• REPORTING. expert_slots/expert_cache_mib sum every tier: a 4-GPU 256K run reported CUDA0's cache alone (3327 experts) while the four cards held 13310, so the Monitor tab was wrong by 4x for every multi-GPU config. expert_slots_primary/expert_cache_primary_mib keep the old figure. The helper ranks' ms_begin/ms_wait are cumulative since boot and were printed next to per-request deltas; they take deltas now. KV-streaming stats iterate the session's OWNED ordinals.

• REFUSALS. --gpu-only-full and --gpu-stages replay every layer through CUDA0's session, which a split carves to CUDA0's range, so they are refused with a layer split rather than answering about another stage's state; the token-graph hit path is disabled under a split (such a graph cannot span stages); and the sessions are allocated AFTER the split search, because the search prices each placement by what that range's session will cost on that device (session_bytes is pure arithmetic).

updates

Multi-GPU: the session carve and the per-stage prompt loans

Two things a layer split was paying for and not using.

• THE SESSION CARVE. A stage carved state for ALL 48 layers whatever slice it ran - every QSA KV pool at
  max_cells, every GDN recurrence - the whole-model-sized-buffer disease the chunked-QSA-prefill PR cured
  upstream. session_bytes/session_init take [layer_lo, layer_hi), and a session now owns only that range:
  qsa_states/gdn_state keep GLOBAL ordinal indexing and simply hold fewer rows (qsa_ord0/qsa_alloc,
  gdn_ord0/gdn_alloc), per-layer consumers subtract gdn_ord0, and the model-level uses - the shared RoPE
  table, the prefill staging identity, the MTP rope borrow - read qsa_states[qsa_primary()]. A range with no
  QSA layer still carves ONE primary state so those uses stay valid, and the default full range is
  byte-identical to the old carve. On the 2x3060 + 2x5060 rig, CUDA1's [30, 48) session costs 0.93 GiB:
  7.83 GiB free before the carve, 6.90 GiB after (session [30, 48) and the head).

• PER-STAGE PROMPT LOANS. no_prefill_borrow = true for split stages said "each stage's prompt path has its
  own buffers" - true, but that is a reason to give each stage its own LOAN, not to make every stage withhold a
  chunk-sized reserve from its cache for the whole session, and forced on it also collapsed --prefill auto to
  2048. Each prompt path now borrows the tail of ITS OWN stage's cache (PfPart), so outside the prompt the
  whole cache is expert cache: --prefill auto is the largest chunk every participant can lend (8192 tokens
  here), a participant refills its loan into its own slots and marks only its own layers (a slot refilled into
  another cache, or another stage's rows marked resident, is silent and produces plausible tokens), and the
  chunk is the largest one EVERY participant can lend rather than one card's answer imposed on the others.
  A participant that cannot lend the buffers falls back to the prompt path's own, as a lone one always could;
  with a single participant the plan reduces to plan_lend exactly, so the single-GPU loan is unchanged from
  main - the percentage cap stays an AUTO-chunk rule (an explicit --prefill is the operator's number and a
  loan of it only has to leave the 128-slot floor). The flat 1 GiB withheld from every stage after the first
  becomes what is actually held back after the weights, session and drafter exist: 96 MiB of verify windows,
  plus 1000 MiB of drafter and head on the ONE stage that carries them. CUDA0 lends 2162 slots (3.98 GiB),
  CUDA1 1993 of its 2653 (3.97 GiB).

• REPORTING. expert_slots/expert_cache_mib sum every tier: a 4-GPU 256K run reported CUDA0's cache alone
  (3327 experts) while the four cards held 13310, so the Monitor tab was wrong by 4x for every multi-GPU
  config. expert_slots_primary/expert_cache_primary_mib keep the old figure. The helper ranks'
  ms_begin/ms_wait are cumulative since boot and were printed next to per-request deltas; they take deltas now.
  KV-streaming stats iterate the session's OWNED ordinals.

• REFUSALS. --gpu-only-full and --gpu-stages replay every layer through CUDA0's session, which a split
  carves to CUDA0's range, so they are refused with a layer split rather than answering about another stage's
  state; the token-graph hit path is disabled under a split (such a graph cannot span stages); and the sessions
  are allocated AFTER the split search, because the search prices each placement by what that range's session
  will cost on that device (session_bytes is pure arithmetic).

🤖 Generated with Claude Code

[Niko1221]

The carve and the per-stage loan are exactly what the split needed. Please split this: keep the single-GPU loan identical to main's plan_lend (explicit chunks ignore the percentage cap; when nothing is lendable, fall back to own buffers instead of exiting), and drop the helper-rank offload. It's a measured loss, conflicts with MMQ, and ro_collect/begin_rows reuse the pinned output while copies on m.cs may still be reading it.

[gopinath87607]

The carve and the per-stage loan are exactly what the split needed. Please split this: keep the single-GPU loan identical to main's plan_lend (explicit chunks ignore the percentage cap; when nothing is lendable, fall back to own buffers instead of exiting), and drop the helper-rank offload. It's a measured loss, conflicts with MMQ, and ro_collect/begin_rows reuse the pinned output while copies on m.cs may still be reading it.

Both done. The offload is gone entirely — you were right about the race: ro_collect issues the H2D from the pinned output on m.cs, and the next ro_issue/begin_rows on that rank overwrites the same buffer with nothing ordering the two. The loan is plan_lend again: the cap is an auto-chunk rule only, and a participant that can't lend falls back to its own buffers. With one participant it reduces to main exactly — verified with STRATA_PREFILL_LEND_PCT=50: auto demotes 8192→4096 (47%), explicit holds 8192 at 80.8%. Carve and loans unchanged on the rig (8192 / 2154 / 1994 of 2597).

btw
i am using ai if anythink i need to correct it kindly let me know it.

[gopinath87607]

fixing the conflict now

[gopinath87607]

testing the quality. seems like there is quality issues above 100k

updated tested like this

cd /home/gopi/Strata
/usr/bin/numactl --membind=0 /home/gopi/Strata/.venv/bin/python /home/gopi/Strata/serve/server.py
--engine strata
--config /home/gopi/Strata/strata-iq3_s-pfauto-vision.json
--port 8080 --open

json
{
"exe": "/home/gopi/Strata/engine/strata",
"args": [
"--pack",
"/home/gopi/Strata-data/packs/iq3_s",
"--native",
"/mnt/nvme/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
"--ple-gguf",
"/mnt/nvme/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
"--expert-profile",
"/home/gopi/Strata/data/expert-profile.bin",
"--expert-cache",
"auto",
"--prefill",
"auto",
"--spec",
"4",
"--spec-min-p",
"0.5",
"--mtp",
"/home/gopi/Strata-data/mtp/rt",
"--max-context",
"262144",
"--kv",
"int8",
"--kv-resident",
"32768",
"--expert-cache-device1",
"3500",
"--expert-cache-device2",
"3500",
"--vision"
],
"cwd": "/home/gopi/Strata",
"tokenizer": "/home/gopi/Strata-data/packs/iq3_s/tokenizer",
"model_name": "qwen3.8-flash-next-iq3_s-pfauto-vision",
"log": "/home/gopi/Strata/strata-iq3_s-pfauto-vision.log",
"lib_dirs": [
"/usr/local/cuda-13.2/bin",
"/usr/local/cuda-13.2/lib64"
],
"port": 8080,
"gpu": [
0,
1,
2,
3
],
"gpus_asked": true,
"layer_split": "30",
"vision": {
"exe": "/home/gopi/Strata/engine/strata-vision",
"mmproj": "/mnt/nvme/mmproj-F16.gguf",
"model": "/mnt/nvme/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
"gpu": true,
"max_tokens": 1024
}
}

the response is really smooth and accurate above 100k & currently its ready to merge

[Niko1221]

Thanks @gopinath87607, all our asks are in and it merges cleanly. Next: our single-GPU byte gate on 4 quants plus Windows and HIP builds. Two things from you: a split vs single-GPU greedy identity run at a fixed cache, and a word on the '100K quality' remark (is there an open problem above 100K or not?).

[gopinath87607]

Thanks @gopinath87607, all our asks are in and it merges cleanly. Next: our single-GPU byte gate on 4 quants plus Windows and HIP builds. Two things from you: a split vs single-GPU greedy identity run at a fixed cache, and a word on the '100K quality' remark (is there an open problem above 100K or not?).

hi niko i am currently focusing on adding Q8_0 quant support and above 100k its working well but still i am not satisfied with quality. give me a day i will update everything.

[Adamyno]

Independent datapoint on 3x RTX 2080 Ti 22GB (the 2x16 + x8 topology, PCIe P2P, no NVLink) —
#216's tree (0.1.28-base) compiled for sm_75, and this is the split-vs-single identity run you
asked for, with one methodological caveat at the end.

Single card (GPU0, fixed cache): --expert-cache 10044 --adapt-swaps 0 --pcie-frac 0,
STRATA_MMQ_NO_STREAMK=1, greedy, 64 generated tokens:

• 13,501-token prompt: 18.71 s
• 30,131-token prompt: 40.54 s
• 97,641-token prompt: 119.63 s

Split + cache-only third card: --layer-split 26 (explicit 2-stage — the auto planner gives all
three visible cards stages and then no card is left for --expert-cache-remote, which aborts
with "needs a GPU that runs no stage"; worth a docs line), stages on GPU0+GPU1, remote cache
3851 pairs on the x8 card:

• 13k: 16.76 s (−10%)
• 30k: 30.04 s (−26%)
• 100k: 78.31 s (−35%)

The remote cache turns the second stage's host-computed misses into GPU hits over PCIe, and the
x8 link carries it — at 100k that is 2.6x prefill throughput on this box.

Identity: run-to-run on the single card is identical (A1 vs A2, 285 chars), so the gate is
meaningful here — but split vs single is NOT byte-identical at 13k and 30k (one token flips
around char ~130 in each; at 100k the outputs matched exactly). The cause looks set-borne, not
kernel-borne: the remote-cache admission is per-stage ranked (stripe), so 6193+3851 does not
reproduce a single card's global top-10044 — some pairs cached on the GPU in one arm are
host-computed in the other, and the documented GPU-vs-CPU rounding does the rest. So the
"fixed cache" identity condition can't currently be expressed from the CLI for a split — a
global-rank split-fill mode, or a split-vs-split (with/without #216) gate, would make it
testable; happy to run either on this rig.

On the 100K-quality question: at 97.6k tokens the split+cache output matched the single card
exactly (271 chars), and the single card was run-to-run stable — no anomaly seen in this arm.

[gopinath87607]

i have pushed the support for unsloth Q8_0 . fixing the conflicts now
#255

[Niko1221]

@gopinath87607 your carve and per-stage loans are in the 0.1.30 release candidate, branch rc/0.1.30, together with #189 (the multi-conversation cache, whose checkpoints now follow your carve: a stage saves and restores only the layers it owns). On one GPU everything is byte-identical to 0.1.29 on our four quants, but we have no multi-GPU box right now. Could you run your split test on that branch before we release? Most useful:

1. greedy split vs single-GPU at a fixed cache (--expert-cache 1500 --adapt-every 100000 --pcie-frac 0 --suffix-draft 0), same text expected;
2. a prompt longer than 16K on the split (a mid-prompt checkpoint is saved from the stages' parts), then the same conversation plus a new turn: RESUME > 0 and the same tokens as a fresh read;
3. --conversation-cache-mib 8192 --layer-split auto should refuse at start ('does not yet support --layer-split').

And a word on the earlier 100K remark would help. Thanks!

[gopinath87607]

@gopinath87607 your carve and per-stage loans are in the 0.1.30 release candidate, branch rc/0.1.30, together with #189 (the multi-conversation cache, whose checkpoints now follow your carve: a stage saves and restores only the layers it owns). On one GPU everything is byte-identical to 0.1.29 on our four quants, but we have no multi-GPU box right now. Could you run your split test on that branch before we release? Most useful:

1. greedy split vs single-GPU at a fixed cache (`--expert-cache 1500 --adapt-every 100000 --pcie-frac 0 --suffix-draft 0`), same text expected;

2. a prompt longer than 16K on the split (a mid-prompt checkpoint is saved from the stages' parts), then the same conversation plus a new turn: `RESUME` > 0 and the same tokens as a fresh read;

3. `--conversation-cache-mib 8192 --layer-split auto` should refuse at start ('does not yet support --layer-split').

And a word on the earlier 100K remark would help. Thanks!

Tested on rc/0.1.30 (a74b1cd), IQ3_S, greedy. The split is my daily rig: 4 cards — CUDA0+CUDA1 hold the layers (--layer-split 30), CUDA2+CUDA3 are the two 3670-slot remote expert caches. Single-GPU arms are one 3060.

1. Split vs single at the fixed cache — not identical (214 vs 205 tokens, first diff char 241). It's the cache set, not the carve. The split's two stages hold 2005 and 2849 slots filled from their own layers' pairs, with 3670 more on each 5060; the single card holds the global top-1934. Different experts on the GPU, and a GPU-cached expert doesn't round like a CPU one. Isolating it:

• both stages on one GPU (--split-device 0, so one shared 1934-slot set) vs single: byte-identical — same 205 tokens, same 110/147 drafts. The stage partition, the per-window hand-off and the split drive add nothing when the set matches.
• single vs single, --expert-cache 700 vs 1500: differ, char 225 — the set alone moves the text.
• split run twice, fresh boots, plus test 1's arm: byte-identical — not run-to-run noise. Adamyno's 3x2080Ti result is the same shape.

Limit: --split-device 0 shares the full session, so it does not exercise the per-stage carve — only the confounded 2-GPU run does. Make --expert-cache 0 mean "profile for the search, no resident slots" under a split and this becomes an exact gate.

2. >16K prompt + new turn — RESUME passes, the fresh-read identity does not. 19343 = 19116 reused + 227 read in 4297 ms vs 40420 ms cold. But turn 1 alone already differs between the arms (char 63, both cold reads, caches/loans/chunk 6144 identical), so it is --prompt-cache itself, not the checkpoint. Mechanically: with --prompt-cache > 0, generate.cpp:4613 splits a chat prompt at the last <|im_start|> and :4642 feeds each part as one sp.run, so the last chunk is 785 wide instead of 764+21 — and that position is the first generated token. Control: --prompt-cache 6 --prompt-cache-every 1000000 (ring on, no checkpoint ever written) is byte-identical to the cache-6 arm and different from cache-0. That read path is 23f6b15b (Mirtraxxx, the conversation-cache read) — not the carve.

3. --conversation-cache-mib 8192 --layer-split auto — refuses at start, exit 2: does not yet support --layer-split. Control without --layer-split never prints it.

100K — reran on rc/0.1.30 and I can't reproduce it: the output is identical. I'd treat my earlier note as a quality observation, not a regression on this branch.

One thing I have not isolated, for what it's worth: 2 of 4 engine runs ignored SIGINT and sat in futex_do_wait holding their VRAM until SIGKILL. Both were --max-context 262144 --kv-resident 32768 runs; the 8192-context ones exited cleanly every time. Could be my harness rather than the engine.

[Niko1221]

Released in 0.1.30, with the snapshot/checkpoint follow-up so a session saves and restores only the layers it owns (needed together with #189). Single-GPU output is byte-identical to 0.1.29.

We also ran it on two AMD RDNA4 cards (R9700 32 GB + RX 9070 XT 16 GB): with the same experts on the GPUs the split gives exactly the tokens of one card, and checkpoints on the split (second turn, rewind, a shared prefix, a cancelled prompt retried) equal a fresh read. We don't have an NVIDIA multi-GPU rig, so please run 0.1.30 on your 2x3060 + 2x5060 and tell us if anything differs from your branch. Merged under your name. Thank you!