[sanastasiou]

What

On a layer split under serve --batch, a prompt read that starts while a batch slot is decoding is cut into single chunks with a decode window between each, because the stage pipeline only overlaps within one sp.run. The read kept that shape to its last chunk, even when the slots finished their short turns a few seconds in. Long reads in a busy server ran at about half the speed of the same read on an idle server.

read_part now decides at every chunk boundary how much to read next (prompt_read_piece / prompt_read_run_end in include/strata/program/read_piece.hpp):

• a slot is decoding, or the engine is not a layer split: one chunk, as before;
• layer split and no slot decoding: STRATA_SPLIT_PIECE tokens, rounded down to whole chunks (at least one);
• STRATA_SPLIT_PIECE unset or 0: the whole remaining prompt in one run, as before.

The piece size bounds how long a shorter request's BYIELD waits behind a long read.

Scope: this applies with --batch-groups 1 (the default). With --batch-groups > 1 on a split (piped), read_part still reads the whole prompt in one sp.run; this PR does not change that path.

Default behaviour does not change. With STRATA_SPLIT_PIECE unset, an idle split reads the whole prompt in one run, the same as today. The only thing that differs without the variable is that a read which started beside a decoding slot goes back to one run once no slot decodes, where today it stayed on single chunks.

Measured

2× RTX 3090 layer-split pair (PCIe 3.0, no NVLink), Qwen3.8-Flash-Next IQ3_S, --batch 2 --prefill auto (8K chunks), STRATA_SPLIT_PIECE=65536. Load: 4 parallel Claude Code compaction benchmarks, each growing to ~400K tokens, through one router onto two such pairs.

read	before	after
prompt reads that began beside a decoding slot	1,240–1,740 tok/s	1,800–2,450 tok/s
e.g. a 124K read	1,716 tok/s	2,456 tok/s
e.g. a 99.5K read at long context	1,311–1,347 tok/s	1,793 tok/s
idle 249K read (unchanged path)		3,323 tok/s, 75 s

Over the whole 4×400K run: time to first byte p95 123 s → 76–80 s, mean wall time per conversation 1,260 s → 1,178 s. Recall 5/5 in all four conversations.

Tests

• read_piece_test (new, header-only, 15 checks): piece rules, rounding to whole chunks, run end clamped at the prompt end.
• CUDA build: generate.cpp compiles; ran in production on both pairs.
• sycl/src/program/generate.cpp: same edit, not compiled (no oneAPI here).

[blange48]

Tested on a 4-stage split (4× RTX 5080 16 GB (PCIe Gen3), Qwen3.8-Flash-Next IQ3_S, 262K, int8 KV, --layer-split 12,24,36 --trim-stage-weights --batch 8 --batch-groups 4, MTP spec 4, --conversation-cache-mib 16384 --conversation-cache-slots 8, on v0.1.40.3 plus a few PRs). Scenario: an agent conversation is held, a fresh 45K-token prompt starts reading, and the agent's short follow-up arrives 2 s later. Follow-up's first token, 2 rounds each:

	follow-up during the 45K read	45K read alone	fresh ~325 / ~585-token prompt
STRATA_SPLIT_PIECE unset	10.35 / 10.11 s	11.03 / 11.28 s	520 / 596 ms
=16384	10.27 / 10.09 s	11.09 / 11.20 s	517 / 594 ms
=8192	10.27 / 10.08 s	11.08 / 11.13 s	522 / 593 ms

The follow-up alone answers in 0.15 s, so on this setup the read still doesn't give way between pieces. My guess is the case your PR doesn't target: here no slot is decoding when the read starts, so the read runs as one sp.run, and the waiting follow-up is a new admission rather than a decoding slot. #1288's yield rule and --prefill 8192 didn't change it either. Nothing regressed: batch_test 8/8 and 3/3, and 1-8 clients 147 / 170 / 238 / 269 / 394 tok/s. If a BYIELD trace on this setup would help, tell me what to log.

[sanastasiou]

Thanks, that table answers it. With --batch-groups 4 the read never reaches this code: piped is true (generate.cpp, const bool piped = o.batch > 0 && o.batch_groups > 1 && n_pipe > 1), and read_part returns early with one whole sp.run before it looks at STRATA_SPLIT_PIECE. So the env var has no effect on your setup, which is what you measured. I've added that to the description.

Our numbers are from --batch 2 with the default --batch-groups 1 on a 2-stage split. There the change does two things: a read that started next to a decoding slot goes back to multi-chunk runs once the slot is done (the +30-40 %), and an idle read is cut into STRATA_SPLIT_PIECE runs with the BYIELD check between them.

Your case (an idle read and a waiting new admission, with pipelined groups) needs the pipelined path to read in pieces too. That isn't a one-line change, because the groups hand rows between stages, and I haven't measured it, so I'd keep it out of this PR. If you want to try it, the first thing to log would be when the server sends BYIELD for the follow-up and when read_part next checks for it.

[blange48]

Follow-up on v0.1.41, same rig and same scenario (4x RTX 5080 PCIe Gen3, IQ3_S, --layer-split 12,24,36 --batch 8 --batch-groups 4, MTP spec 4). An agent's short follow-up arrives while a fresh 45K-token prompt is being read:

	follow-up alone	follow-up during the 45K read	the 45K read finishes
0.1.40.3 + PRs	0.12 s	~9.5 s	~12 s
0.1.41 + PRs, run 1	0.12 s	9.67 s	12.2 s
0.1.41 + PRs, run 2	0.12 s	9.49 s	12.0 s

As you explained, the pipelined read is still one sp.run, so the follow-up waits for almost the whole read. The 0.1.41 fix for "one long request next to short ones slowed by 39-47%" is about decode; it does not change this case. If a split of the pipelined read into pieces is on the table, I can measure it on this rig.

[sanastasiou]

Thanks for re-checking on 0.1.41; that matches the code. With --batch-groups > 1 the read never reaches the piece logic, so the numbers can't move with this PR.

Splitting the pipelined read into pieces isn't something I plan to do in this PR. The groups hand rows between stages inside one sp.run, so breaking it up means changing how the pipeline drains between pieces, and I don't run --batch-groups > 1 myself, so I couldn't measure it properly. If someone takes it on, your rig and this scenario would be the right test: follow-up TTFT during a 45K read with groups 4.

[sanastasiou]

One more measurement would settle whether the PR is useful on your rig. Both cases need --batch-groups 1 (the default), because with groups > 1 the read never reaches this code:

1. A long read that starts next to a decoding slot (the main gain on our side, +30-40 %)

• Your --layer-split 12,24,36, --batch 2, --batch-groups 1.
• Send request A: a short prompt with max_tokens ~600, so it decodes for a few seconds.
• About 1 s after A's first token, send request B: a fresh prompt of 100K+ tokens (no cache hit), max_tokens 1.
• Record B's time to first token and the engine's "the prompt was read in N parts" line.
• Three variants, ≥3 runs each: base build without the PR; PR build with STRATA_SPLIT_PIECE unset (should equal base); PR build with STRATA_SPLIT_PIECE=65536.

Without the PR, B stays in single chunks for the whole read even after A has finished. With it, B goes back to multi-chunk pipelined runs once A is done. With 4 stages instead of our 2, the difference should be larger.

2. Your follow-up scenario, with groups 1
The same 45K read plus the short follow-up 2 s later, but --batch-groups 1 and STRATA_SPLIT_PIECE=16384. The follow-up's BYIELD is checked between pieces, so it should wait about one piece instead of ~9.5 s.

If either case shows no gain on your rig, that's worth knowing too.

[blange48]

Both measurements, as asked, on 4x RTX 5080 (PCIe Gen3), IQ3_S, --layer-split 12,24,36, MTP spec 4, int8 KV. The base is v0.1.41 plus our other patches; "without PR" is that build with this PR's commit reverted.

1. Fresh ~100K read starting ~1 s after A's first token (--batch 2 --batch-groups 1, A = 600 tokens at temperature 0.7, B = max_tokens 1, 3 runs each)

variant	B time to first token	A finished in	engine line
without the PR	42.4 / 41.9 / 41.7 s	35.1-35.4 s	"read in 10 parts, the slots decoding 10.4-10.7 s between them"
PR, STRATA_SPLIT_PIECE unset	39.5 / 39.1 / 39.2 s	34.8-35.4 s	"read in 10 parts", 10.3-10.4 s
PR, STRATA_SPLIT_PIECE=65536	39.8 / 39.2 / 39.4 s	35.1-35.6 s	"read in 10 parts", 10.3-10.6 s

On our rig, then:

• The gain is about 7% (42 s to 39 s), not +30-40%.
• The gain is already there with the variable unset. You expected that case to equal the base, so something else in the commit may be helping.
• All three variants report 10 parts. A finishes around 35 s, well before B ends, but I did not see B go back to longer runs.

2. Our follow-up scenario with groups 1 (--batch 8 --batch-groups 1: 45K fresh read, short agent follow-up 2 s later, 2 rounds)

variant	follow-up alone	follow-up during the 45K read	45K read, first token
without the PR	0.13-0.16 s	9.94 / 9.99 s	10.7 / 10.8 s
PR, STRATA_SPLIT_PIECE=16384	0.14-0.15 s	4.02 / 4.07 s	17.7 / 17.8 s

So the follow-up waits 2.5x less (10 s down to 4 s), and the long read pays 7 s for it (+65%). The engine says "the prompt was read in 2 parts, the slots decoding 550 ms between them". I expected more parts with 16384-token pieces on a 45K prompt.

For us, the trade-off loses: production runs --batch-groups 4 for throughput (about 390 tok/s in total at 8 clients; the release notes measure +32-68% for groups over one group), and this path is not taken there. Thanks for the detailed protocol.

[sanastasiou]

Thanks — that's the measurement we needed, and it doesn't support the PR. Case 1 gives ~7 % instead of the +30-40 % we saw, the gain is there with the variable unset, and the read never returns to longer runs, so our explanation of where the gain comes from doesn't hold on your rig. Case 2 trades a 2.5x shorter follow-up wait for a 65 % longer read, and the "2 parts" on a 45K prompt with 16K pieces says the piece logic doesn't behave as described either.

On 0.1.41 we've also measured that --batch-groups auto beats groups 1 on our 2-stage split (+31-44 % two-row decode, 2.6x faster reads next to a decoding slot), so this path stops mattering for us as well. I'm closing this PR. We'll re-measure on our side once we run groups auto; if piecing the read still helps there, we'll come back with data. If the follow-up latency case comes up again, the right place is the pipelined read, and your rig and protocol would be the test. Thanks for the careful runs.