--batch 2 --batch-groups auto on a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s

Summary

Windows, 2x RTX 3080 20 GB layer split, engine 0.1.42, "parallel": 2 with --batch-groups auto (2 groups of 1 slot, i.e. the pipelined path), Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS + MTP. A real agent client keeps two conversations alive (48–55K tokens each) and issues two tool-call turns concurrently (skill_manage + execute_code).

In one 23-minute window the engine logged 33 prompt reads; 11 of them with reused 0 — the whole prompt read from token 0. Two of those 11 are negligible (one 293-token probe and one synthetic request of mine, marked below); the 9 real ones are 38K–180K tokens and cost 608.6 s of reading (24–272 s each). Seconds earlier the same conversation had been re-admitted as a copy: slot 0 takes 47688 tokens (copied in 456.9 ms). While a read runs, the other group's decode falls to 3.7–13 tok/s (29–37 tok/s when both groups decode).

No hang and no disconnect in this window: /metrics showed running: 2, queued: 0, admitting: 0 throughout, finish: stop. This is a performance report, not the #1603 hang.

Rig, config

Windows 11 / WDDM, 2x RTX 3080 20 GB, layer split auto -> CUDA0 layers 0-24, CUDA1 25-47 (K=25, predicted 33.1 ms/window)
Xeon E5-2696 v3 (18C/36T), no AVX-512, 128 GB RAM
engine 0.1.42 from the release zip (strata.exe 165,004,288 B), strata-vision.exe 108,172,288 B
args: --pack <iq3_xxs> --native <shard> --ple-gguf <shard> --expert-profile <bin> --expert-cache auto --prefill auto --batch-groups auto --spec 4 --mtp <rt> --max-context 262144 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700 --control-vector-scaled <esp.gguf>:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer --remote-expert-opt --spec-min-p 0.5 --layer-split auto --batch 2
"parallel": 2, no --conversation-cache-* configured
start-up: --batch-groups auto: 2 groups of 1 slot; expert cache 8261 slots (12.04 GiB) + CUDA1 6566 slots (11.90 GiB) = 14,827; slot sessions take 1.03 GiB on CUDA0; prompt path borrows 2502 CUDA0 + 2006 CUDA1 slots
PCIe probe 12.4 GB/s -> pcie_frac 0.34, miss cost x1.61
Note for Fewer missed experts in decode: STRATA_ROUTE_TAIL_SKIP=R and STRATA_ROUTE_PRIOR=lambda (decode x1.33 on a 12 GB card, opt-in, change the output) #1668: STRATA_ROUTE_TAIL_SKIP's default is off here because o.batch != 0 (src/program/generate.cpp:8076), so this run does not have that 0.1.42 default either

Evidence 1 — the re-reads (all from one session, 08:34:59–08:58)

prompt	read	read speed	generated	decode
151,335	272.3 s	556 tok/s	1542	80.4 tok/s
180,159	112.5 s	1602 tok/s	9501	65.3
38,350	23.6 s	1622 tok/s	1	38.1
42,855	21.8 s	1962 tok/s	385	68.6
48,273	24.1 s	2000 tok/s	499	63.3
[40,687	17.3 s	2351 tok/s	1	43.2]
51,068	24.4 s	2090 tok/s	1	45.1
44,528	22.1 s	2010 tok/s	1	21.1
54,301	38.8 s	1400 tok/s	1	13.5
54,569	68.8 s	793 tok/s	1	40.3

9 real re-reads, 608.6 s of reading. The other 22 admissions in that session reused: e.g. 48157 = 47697 reused + 1 read, or 47550 = 47137 reused + 413 read.

Give back 8x, slot N takes 15x in the same window. Every 1-token admission reports drafts accepted 0 of 0 (the pipelined path, as documented in docs/BATCHING.md), while the longer generations had working drafts (drafts accepted 6030 of 8284, suffix drafts: 343 windows, 1022 of 1029 accepted).

Evidence 2 — the same conversation, copy vs read, seconds apart

strata batch: slot 0 gave back 45097 tokens of this conversation (all it holds) in 419.4 ms
strata batch: slot 0 takes 47137 tokens (copied in 475.3 ms)
strata serve: prompt 47137 tokens = 45097 reused + 2040 read in 5695 ms (358.2 tok/s), 1 generated ...
...
strata batch: slot 0 takes 47688 tokens (copied in 456.9 ms)
strata serve: prompt 47688 tokens = 47687 reused + 1 read in 3 ms (340.8 tok/s), 1 generated ...
...
strata batch: slot 1 takes 54301 tokens (copied in 832.1 ms)
strata serve: prompt 54301 tokens = 0 reused + 54301 read in 38786 ms (1400.0 tok/s), 1 generated ...

The copy path is 0.46 s; the read path for a conversation of the same size is 38.8 s. Both happen with the same two conversations in the same minute.

Evidence 3 — what the sibling group does while a read runs

Console, two requests interleaved (server's own progress lines), during a 54.5K read:

[strata] writing a tool call: execute_code: 152 of max 213875 tokens, 13.6 tok/s, 19 s
[strata] writing a tool call: skill_manage:  193 of max 207835 tokens,  9.1 tok/s, 61 s
[strata] reading the prompt: 16,384 of 54,569 tokens, 23 s so far
[strata] writing a tool call: execute_code: 162 of max 213875 tokens,  4.8 tok/s, 41 s
[strata] reading the prompt: 24,576 of 54,569 tokens, 32 s so far
[strata] writing a tool call: execute_code: 162 of max 213875 tokens,  3.7 tok/s, 51 s

/metrics sampled every 12 s in the same window:

time	slot 0	slot 1	hw total	GPU	PCIe rx
08:53:20	decoding 28.8 tok/s	decoding 30.0	62.7	76 %	477 MB/s
08:53:45	decoding (mean 114.7)	reading 51,068	23.6	98 %	35.4 GB/s
08:53:57	gen frozen at 792 for 12 s	still reading	17.2	99 %	3.9 GB/s
08:54:09	31.5	30.5	61.3	70 %	397 MB/s

So two groups decoding = ~29–37 tok/s each (total ~60–75); one group reading = the other drops to 3.7–13 tok/s or pauses for 12 s and then resumes.

Questions

What decides takes ... (copied in 0.4–1.0 s) (parked, reuse) vs prompt N = 0 reused + N read in 24–69 s for a conversation of the same size, in the same minute, with only two conversations and two slots? If the slot assignment moves the conversations between slots (the pick_slot / group choice), could the slot be picked by the prompt's prefix instead of by turn order?
On this engine, would --batch-groups 1 show the same re-read rate? I can run the same 10-minute workload twice and send both tables (per-admission reused/read ms and the sibling's rate) if that helps.
batch_read_piece() returns 0 for piped, so the STRATA_BATCH_DECODE_SHARE / STRATA_BATCH_YIELD_CHUNKS interleave never runs on the pipelined path (this is what the second follow-up on parallel: 2 + two long-context sessions: the request hangs with running: 0 / queued: 0 and the #1317 frozen-engine watchdog never sees it #1603 points at). Is that intended? If it is, is there any supported way to bound how long the other group waits for a read — measured here at 12–39 s per read, on a workload where a turn is ~10 s long?

parallel: 2 + two long-context sessions: the request hangs with running: 0 / queued: 0 and the #1317 frozen-engine watchdog never sees it #1603 — the hang (running: 0); not seen in this window. My two follow-ups on it are the 55.9 % hit-rate dump and the --batch-groups auto read-splitting analysis.
--batch-groups disables the per-slot conversation cache even when the group size is 1 (no pad row is ever written) → alternating long conversations re-prefill from token 0 every turn #1604 — said "could not reproduce in v0.1.42" with 2x R9700 and three rounds of ~3.2K-token conversations. This report is the same area with 47–54K conversations and ~30 turns, and it does reproduce the re-read.
--batch-groups G>1 + --batch-mtp is silently inert: the pipelined path never calls batch_step, but still builds per-slot drafters #1413, and the new include/strata/program/batch_groups.hpp note that auto gives way to --batch-mtp / --adapt-async (not asked for here).
Fewer missed experts in decode: STRATA_ROUTE_TAIL_SKIP=R and STRATA_ROUTE_PRIOR=lambda (decode x1.33 on a 12 GB card, opt-in, change the output) #1668 — STRATA_ROUTE_TAIL_SKIP=7 is off with --batch slots, so this configuration forgoes that 0.1.42 gain.
[Bug]: Two-GPU pipelined decode (--pipeline-windows 2) stalls on Windows: the serving thread blocks in cudaSetDevice while both stages wait on it #1740 — a real --pipeline-windows 2 / cudaSetDevice stall; not this (the engine stayed alive here).

Correction to the range in the title and the body: the sibling does not only fall to 4–13 tok/s, it stops.

I measured the 4–13 tok/s from /metrics sampled every 12 s. Two things are lower than that in the same window:

The client's own console line reports the rate as a mean from the request's first token, and a turn that sits through a read shows 0.1 tok/s (e.g. a writing a tool call line frozen at 162 of 213875 tokens while the other group read 54,569 tokens for 39 s). The engine's per-DONE line never shows this, because it only counts the segment that produced the token — its minimum in this session is 7.2 tok/s (1 token in 140 ms). Same window, two different means.
/metrics' aggregate tok_s history (1 s samples) has 8 zero samples, including three consecutive ones — i.e. neither group produced a token for that stretch, not just a slow one.

So the honest range is 0.1–13 tok/s (aggregate 0 for up to ~3 s at a time), not 4–13. The rest of the report stands: 9 full re-reads (608 s of reading) in 23 minutes, no hang, running: 2 throughout, and every admission's drafts accepted 0 of 0 on the pipelined path.

Same workload, ~1h50m later: 38 full re-reads (2,303 s of reading), 12 consecutive seconds with no token at all, and the client now aborts mid-read

Two hours on the same machine and config as the report above (0.1.42, 2x RTX 3080 20 GB layer split, "parallel": 2, --batch-groups auto, two agent conversations). The prompts have grown from 48–55K to 100–107K, and everything I posted above got bigger, not smaller.

1. Prompt reads, this session (engine start 08:34:59 -> 10:24, `strata-swift-iq3_xxs.log`)

189 prompt-read lines, 38 of them with reused 0; 35 of those took >= 20 s: 2,303 s of reading = 38 min of a 110-minute session (~35 %)
the cost is trending up with the context: 48–55K reads cost 21–69 s, the last hour's 80–107K reads cost 80–116 s

prompt	reused	read	speed	generated
107,051	0	99.4 s	1077 tok/s	1
104,770	0	(in flight at 10:24)	—	—
100,559	0	116.3 s	865 tok/s	1
91,031	0	104.6 s	870 tok/s	1
90,418	0	102.4 s	883 tok/s	1
89,673	sum of two attempts	10.6 s + 30.8 s	776 / 594 tok/s	1
87,681	0	96.2 s	912 tok/s	1
85,893	0	96.4 s	891 tok/s	1
80,621	0	94.2 s	856 tok/s	1
54,569	0	68.8 s	793 tok/s	1

Note the rate column: the same kind of read ran at 2,300–2,700 tok/s earlier in the session (e.g. 80247 = 0 reused + 80247 read in 29.7 s (2703.3 tok/s)) and is now 855–1,157 tok/s — the other group was decoding during these. The cost is mutual: the read pays for the sibling's windows, and the sibling pays for the read.

2. The sibling does not just slow down, it stops

/metrics sampled 12 s apart, 10:23:43 / 10:23:56 / 10:24:09:

10:23:43  hw 0.0 tok/s  GPU 93.0 %  PCIe 5380 MB/s  slot0 decoding p=107052 gen=3 0.0 t/s ; slot1 reading p=104770
10:23:56  hw 0.0 tok/s  GPU 76.5 %  PCIe 2592 MB/s  slot0 decoding p=107052 gen=3 0.0 t/s ; slot1 reading p=104770
10:24:09  hw 46.7 tok/s GPU 73.5 %  PCIe  750 MB/s  slot0 decoding p=107052 gen=28 0.2 t/s ; slot1 decoding p=104770 gen=27 23.3 t/s
tok_s history (1 s samples, the 60 s window at 10:23:43): [0.0 x 12, 4.0, 0.8, 46.7]

Twelve consecutive 1-second samples at 0.0 — neither group produced a token for at least those 12 s — with slot 0 frozen at gen 3 for the whole span while slot 1 read 104,770 tokens. Then it recovers (46.7 tok/s aggregate) without any intervention. (This is why the title's 0.1–13 tok/s is itself the optimistic end of the range: 0.0 for tens of seconds is on the same line.)

3. The client now gives up, and the work is thrown away

strata serve: prompt 89663 tokens = 0 reused + 8192 of 89663 read in 10559 ms (775.8 tok/s), 0 generated ... (cancelled)
strata serve: prompt 89689 tokens = 0 reused + 71485 of 89689 read in 77704 ms (920.0 tok/s), 0 generated ... (cancelled)
finish=disconnect,  duration 85.2 s, prompt 71485, reused 0, output_tokens 0
strata batch: slot 1 takes 89773 tokens (copied in 1445.8 ms)
strata serve: prompt 89773 tokens = 71485 reused + 18288 read in 30777 ms (594.2 tok/s), 1 generated ...

Both attempts at the same ~90K conversation were abandoned by the client (one after 10.6 s, one after 77.7 s with 71,485 of 89,689 read), a third request disconnected after 85.2 s having produced zero output tokens, and then a fourth continued from the 71,485 tokens the engine had already read (71485 reused + 18288 read in 30.8 s). So the read work is paid for twice, and the user-visible turn produced nothing.

4. Questions (same three as above, plus one)

Would a "read in pieces" mode on the piped path be acceptable, or is the stage-pipeline overlap meant to cover this? batch_read_piece() returns 0 for piped, so STRATA_BATCH_DECODE_SHARE / STRATA_BATCH_YIELD_CHUNKS never run here. A read now costs 95–116 s while a client timeout is 60–90 s, which is what produces the aborts above.
Is there any supported way to bound the sibling's wait? Measured here: 0.0 tok/s for 12+ s, up to 0.1–13 tok/s over a whole read.
What decides takes ... (copied in 0.4-1.4 s) (reuse) vs 0 reused (24–116 s) for the same conversation minutes apart? 38 of 189 reads took the second path.
If --batch-groups 1 is the better behaviour for this workload, I can run the same two hours on it and post the same tables for comparison — say the word and I will.

Follow-up 2: 'no overlap at all' is still 17.8 tok/s — the cost is per-turn give-back/take plus a pipeline-fill read, not the concurrency

Same machine/config/engine as above. The client here alternates between two ~113–115K conversations, so at any instant /metrics shows running: 1 (one request at a time) — and it is still slow. Three things account for it, all from one 11-minute window (10:27–10:38).

1. Every turn gives the whole conversation back and takes it again

strata batch: slot 0 gave back 113649 tokens of this conversation (all it holds) in 1588.8 ms
strata batch: slot 0 takes 113732 tokens (copied in 1635.6 ms)
strata serve: prompt 113732 tokens = 113649 reused + 83 read in 3756 ms (22.1 tok/s), 1 generated in 20 ms, drafts accepted 0 of 0
strata serve: KV streaming: 58.18% of 15402 block reads hit VRAM, 25.9 MiB read from RAM
strata batch: slot 0 gave back 113887 tokens of this conversation (all it holds) in 1598.0 ms
strata batch: slot 0 takes 113918 tokens (copied in 1659.1 ms)
strata serve: prompt 113918 tokens = 113887 reused + 31 read in 2288 ms (13.5 tok/s), 1 generated in 18 ms, drafts accepted 0 of 0

In that window there are 108 gave back / takes events across both slots (slot 0 and slot 1 alternate turns). So the cost per turn is:

step	measured
`gave back ... (all it holds)`	1.59–1.63 s
`takes ... (copied in ...)`	1.64–1.67 s
incremental read of 31–124 new tokens	2.29–4.26 s (13.5–29.1 tok/s for the read)
total before the first token	~5.6–7.5 s per turn

A turn then produces 30–120 tokens. Reading 31–124 tokens takes seconds because a layer split reads its stages as a pipeline and a piece costs a pipeline fill (include/strata/program/batch_read.hpp says exactly this), and each of these small reads still moves 25–78 MiB of K/V from RAM (58–86 % of block reads hit VRAM on these, against 98–99 % on decode-only segments).

2. The engine's own summary for that window: one row per step, 17.8 tok/s

strata batch (pipelined, 2 groups of 1): 7008 group-steps, 7008 rows in 394698 ms = 17.8 rows/s (admissions included)

7008 rows / 7008 steps = 1.00 row per step, i.e. the two groups never produce a row in the same window here, and the whole stretch averages 17.8 tok/s while it alternates.

3. The speculation is not the problem (so the loss is elsewhere)

Across this session, of the 73 requests that generated >= 200 tokens, all 73 ran drafts, median decode 87.5 tok/s (best 99617 tokens -> 3548 generated in 28947 ms (122.6 tok/s), drafts accepted 2505 of 2954). The drafts accepted 0 of 0 lines above are only the 1-token admission segments. So when a turn does decode, it decodes at the expected speed; what hurts is everything around it.

4. And the full re-reads got worse again

strata serve: prompt 113038 tokens = 0 reused + 113038 read in 140687 ms (803.5 tok/s), 179 generated in 4511 ms, drafts accepted 127 of 146

Second report's headline was 9 re-reads / 608 s; by the time of the previous comment it was 38 re-reads / 2,303 s with the last ones at 855–1,157 tok/s; this one is 113K in 140.7 s at 803 tok/s.

Questions

Can a slot keep its conversation across consecutive turns of the same conversation, instead of gave back ... (all it holds) + takes ... (copied in 1.6 s) every turn? At running: 1 nothing else needed that VRAM in this window.
Is 2 rows per step reachable in the pipelined path at all (the summary above is 1.00), or is one row per group per stage-pass inherent?
Same piped-read question as before: batch_read_piece() returns 0 when piped, so the decode-share/yield-chunks interleave never runs; with pieces the read would pay a pipeline fill per piece (that is the 2.3–4.3 s seen here for 31–124 tokens), so is the intended answer for this workload simply --batch-groups 1 (or "parallel": 1)? If so I will run the same window on both and post the same tables.

batch A/B/C: SIMT scorer adopted, the rest measured and left off; sto…

Cross-reference: same root cause, different trigger path.

--batch 2 --batch-groups auto on a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s #1906 (parallel:2, hand-off path): full re-read happens when a conversation is handed between slots -- 9 real re-reads, 608 s of reading in 23 min. The log there still showed "N reused + M read", so the "reused 0" was visible.
parallel:1 / 0.1.42 -- whole prompt re-read 22x in one session; any total-length change kills the prefix #1930 (parallel:1, single-slot path): full re-read happens on almost every turn, trigger is even simpler -- any change in prompt total length kills the prefix. 22 whole-prompt re-reads in one session (~190K tokens each, ~195 s per read, ~71 min of reading).

Both are the same thing: the KV prefix is discarded whenever the prompt head is not byte-for-byte identical to the previous turn. Under parallel:2 the client hand-off breaks the head; under parallel:1 the client's per-turn trim/append breaks it. Same engine behavior, two ways to trigger it.

Also note: the 0.1.42 single-slot log line dropped the "reused N" field, so on parallel:1 you cannot see a re-read from the log anymore (only by spotting a "reading the prompt: 8,192 of ..." sequence). The expert cache hit % printed next to each turn is the MoE expert-weight number, not the KV prefix -- it reads 92-96% even on a 216 s full re-read.

batch-groups: decode and give way between the pieces of a prompt read…

@ylzbj1-stack

@ylzbj1-stack I ran into the same stall with agents on a 2x RTX 5090 split and opened #2050 (opt-in,
STRATA_PIPED_READ_PIECE). It's the answer to your question 3: with the variable set, the pipelined read goes in
whole-chunk pieces, the other group decodes between them, and a waiting shorter request can take over at a piece boundary.

I reproduced your shape (--batch 2 --batch-groups 2, one agent decoding while the other's cold prompt is read; my
prompts came out a bit larger than yours), median of 2:

cold prompt	release: sibling's longest pause	32K pieces	16K pieces
66K	8.0 s	4.3 s	2.5 s
127K	14.0 s	4.8 s	3.0 s
189K	20.7 s	4.8 s	3.0 s

With pieces the pause stays at about one piece's read time however long the prompt is. During a 189K read the sibling
gets 7.2 tok/s (32K) or 10.5 tok/s (16K) instead of 2.8; the price is the joining prompt's TTFT, 20.3 s -> 26.5 s / 33.8 s.
It doesn't touch the full re-reads you also describe.

On your cards a piece takes longer to read (your log: 54,569 tokens in 39 s, about 1,400 tok/s), so the pause would be
about 12 s with 16K pieces and 6 s with 8K. I'd start with STRATA_PIPED_READ_PIECE=8192 STRATA_PIPED_DECODE_MAX_MS=400;
tools/piped_read_join_bench.py --scenarios two_joiners in the PR measures it. A run on your 2x RTX 3080 would be very
welcome.

@SuperDistance

@SuperDistance thanks for #2050 -- that is exactly the answer to question 3, and the pause table is what I could not produce (I have the cost, not the pauses).
Two questions before I try it, because my rig differs from yours in the two places that matter:

Does the piece size still pay off when the expert path is CPU-bound?
You are on 2x RTX 5090; here it is 2x RTX 3080 20 GB on a layer split, Xeon E5-2696 v3 (18C/36T, no AVX-512 -> AVX-2 expert kernels). The reason I care: with pieces, the decode runs between the pieces, and decode here is the part that reads experts out of host RAM. If the piece boundaries just hand the host back to a decoder that is already saturating the same memory path, the sibling's tok/s may not recover the way it does on your 5090s. I can measure it and post the table.
Does 16K vs 32K still win with a layer split?
Each piece has to cross the CUDA0 -> CUDA1 hand-off, and you measured 8.0 -> 4.3 s (32K) vs 2.5 s (16K) on a single-GPU pair with no split. On a split, the extra hand-offs may eat into that margin; if 16K still wins there, that is the default I would pick.
I will run both piece sizes on a cold-prompt-vs-decoding-sibling shape (--batch 2 --batch-groups 2, one agent decoding while the other's first request is read) and post the numbers either way, including if they come out worse.
For reference on what the read costs when it does go wrong, on 0.1.42 (single slot, IQ3_S): one session had 8 of 38 prompt reads with reused 0, and those 8 were 77% of all prompt-read wall clock, worst single read 308.6 s for 219,597 tokens (see parallel:1 / 0.1.42 -- whole prompt re-read 22x in one session; any total-length change kills the prefix #1930). That is the ceiling this change is up against.

@ylzbj1-stack

@SuperDistance thanks for #2050 -- that is exactly the answer to question 3, and the pause table is what I could not produce (I have the cost, not the pauses). Two questions before I try it, because my rig differs from yours in the two places that matter:

Does the piece size still pay off when the expert path is CPU-bound?
You are on 2x RTX 5090; here it is 2x RTX 3080 20 GB on a layer split, Xeon E5-2696 v3 (18C/36T, no AVX-512 -> AVX-2 expert kernels). The reason I care: with pieces, the decode runs between the pieces, and decode here is the part that reads experts out of host RAM. If the piece boundaries just hand the host back to a decoder that is already saturating the same memory path, the sibling's tok/s may not recover the way it does on your 5090s. I can measure it and post the table.

Does 16K vs 32K still win with a layer split?
Each piece has to cross the CUDA0 -> CUDA1 hand-off, and you measured 8.0 -> 4.3 s (32K) vs 2.5 s (16K) on a single-GPU pair with no split. On a split, the extra hand-offs may eat into that margin; if 16K still wins there, that is the default I would pick.
I will run both piece sizes on a cold-prompt-vs-decoding-sibling shape (--batch 2 --batch-groups 2, one agent decoding while the other's first request is read) and post the numbers either way, including if they come out worse.
For reference on what the read costs when it does go wrong, on 0.1.42 (single slot, IQ3_S): one session had 8 of 38 prompt reads with reused 0, and those 8 were 77% of all prompt-read wall clock, worst single read 308.6 s for 219,597 tokens (see parallel:1 / 0.1.42 -- whole prompt re-read 22x in one session; any total-length change kills the prefix #1930). That is the ceiling this change is up against.

@ylzbj1-stack For the question 2, yes my setup is similar. I have --layer-split 25 with --batch 2 --batch-groups 2. a 45k prompt has +12% with 32k pieces and +25% with 16k.

For the question 1, I don't have similar hardware to reproduce. But pieces handling the host to a decoder that's already saturating it mostly won't happen with #2050 because reading and decoding take turns, a piece runs and drain. a decode window runs and then start the next piece of prompt. So they don't compete for host memory.

I recommend start with STRATA_PIPED_READ_PIECE=8192 and no cap or STRATA_PIPED_DECODE_MAX_MS=2000 if the joiner's TTFT is important. Thanks for volunteering on testing on your rig!

batch-groups: read a prompt beside the decoding slots on a thread of …

docs/BATCHING.md: reading beside the decoding slots, and what it meas…

batch-groups: decode and give way between the pieces of a prompt read…

Server: a slot whose tokens match a prompt up to its last turn bounda…

Niko1221#1906: a slot's turn checkpoints move with its conversation i…

@davidbhoward

Posted by Claude (an AI assistant) at @davidbhoward's request.

I could reproduce this on 2x 3090 with two clients that have thinking on: 10 of 12 turns read their whole history again. Two causes: a request that goes slot -> solo -> slot loses the slot's turn checkpoints, and pick_slot ignores what a slot holds whenever groups are pipelined, even groups of one. With both fixed it drops to the 2 first turns only (95 s of reading down to 27 s). PR: #2384

batch-groups: read a prompt beside the decoding slots on a thread of …

docs/BATCHING.md: reading beside the decoding slots, and what it meas…

sycl: port the Niko1221#1906 slot-checkpoint fix to the SYCL engine

sycl: message-boundary checkpoints reach past a long final turn

sycl: mirror each read's checkpoints into its batch slot as they are …

sycl: an admission carries every checkpoint that prefixes the convers…

Thanks for taking the time to report this.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

Re-test on 0.2.0-beta. It may already be fixed. Write down what you see.
Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
Fill in the parts of the issue form: what happened, the Strata version, your graphics card, memory and system.
Comment /reopen. A check runs and either reopens the issue or lists what is still missing.

The rules are in CONTRIBUTING.md.

Navigation Menu

Repository navigation

`--batch 2 --batch-groups auto` on a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s #1906

Summary

Rig, config

Evidence 1 — the re-reads (all from one session, 08:34:59–08:58)

Evidence 2 — the same conversation, copy vs read, seconds apart

Evidence 3 — what the sibling group does while a read runs

Questions

Related

Same workload, ~1h50m later: 38 full re-reads (2,303 s of reading), 12 consecutive seconds with no token at all, and the client now aborts mid-read

1. Prompt reads, this session (engine start 08:34:59 -> 10:24, `strata-swift-iq3_xxs.log`)

2. The sibling does not just slow down, it stops

3. The client now gives up, and the work is thrown away

4. Questions (same three as above, plus one)

Follow-up 2: 'no overlap at all' is still 17.8 tok/s — the cost is per-turn give-back/take plus a pipeline-fill read, not the concurrency

1. Every turn gives the whole conversation back and takes it again

2. The engine's own summary for that window: one row per step, 17.8 tok/s

3. The speculation is not the problem (so the loss is elsewhere)

4. And the full re-reads got worse again

Questions

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

--batch 2 --batch-groups auto on a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s #1906

Description

Summary

Rig, config

Evidence 1 — the re-reads (all from one session, 08:34:59–08:58)

Evidence 2 — the same conversation, copy vs read, seconds apart

Evidence 3 — what the sibling group does while a read runs

Questions

Related

Activity

ylzbj1-stack commented on Oct 11, 2026

ylzbj1-stack commented on Oct 11, 2026

Same workload, ~1h50m later: 38 full re-reads (2,303 s of reading), 12 consecutive seconds with no token at all, and the client now aborts mid-read

1. Prompt reads, this session (engine start 08:34:59 -> 10:24, strata-swift-iq3_xxs.log)

2. The sibling does not just slow down, it stops

3. The client now gives up, and the work is thrown away

4. Questions (same three as above, plus one)

ylzbj1-stack commented on Oct 11, 2026

Follow-up 2: 'no overlap at all' is still 17.8 tok/s — the cost is per-turn give-back/take plus a pipeline-fill read, not the concurrency

1. Every turn gives the whole conversation back and takes it again

2. The engine's own summary for that window: one row per step, 17.8 tok/s

3. The speculation is not the problem (so the loss is elsewhere)

4. And the full re-reads got worse again

Questions

ylzbj1-stack commented on Oct 11, 2026

SuperDistance commented on Oct 12, 2026

ylzbj1-stack commented on Oct 12, 2026

SuperDistance commented on Oct 12, 2026

davidbhoward commented on Oct 14, 2026

Niko1221 commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`--batch 2 --batch-groups auto` on a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s #1906

1. Prompt reads, this session (engine start 08:34:59 -> 10:24, `strata-swift-iq3_xxs.log`)