[Jackwwg83]

#656 lets a long prompt read give way at a chunk boundary to a waiting request whose prompt is under half as long. It compares prompt lengths only. An agent's requests all carry a long system prompt and the history, so a waiting follow-up turn is almost never under half of a fresh read: it waits for the whole read, though it needs only a few hundred tokens. In a replay of a multi-user agent workload (a 27K-token system prompt, 5-15 users, RTX PRO 5000), 2,086 prompt reads gave way 1 time.

What changes (serve/server.py only)

The rule compares tokens left to read on both sides:

• a waiting request: its prompt less the start the engine holds for sure - all a slot holds, or a slot's or an earlier read's prompt up to its last turn token (<|im_start|>, where the engine checkpoints a prompt and the next turn resumes), while a slot still holds it or the conversation cache parked it (only its slots' worth of the latest reads);
• the read on the control lines: what its PP lines say is left (reset when a read is sent; a request that arrives no longer clears it).

A read gives way only while the waiting request gets a slot at once (an admission: one free slot; a solo read: two, one for its parked part), so a request never waits on the control lines for a slot that a parked part needs to go on. Only prompts read to their end are noted. Each slot's held list is packed once; whole prefixes compare with startswith (8 slots of 50K tokens and 16 waiting prompts: under 50 ms a check).

A wrong guess (an evicted conversation, for example) costs one give-way, and a request gives way at most YIELDS_MAX (2) times; outputs do not change (batch_interleave_test covers a read that gives way).

Behaviour change: a fresh short prompt no longer makes a long read give way when the read has less than twice that left (the old rule compared full lengths).

Tests

• test_a_long_read_gives_way_to_a_follow_up_its_slot_holds: a follow-up whose conversation a slot holds is answered before a fresh read of about the same prompt length. On main it fails (2.25 s against 1.61 s).
• GiveWayByWhatIsLeft: the rule case by case (partial prefixes, reads noted only at their end, the turn-token resume point, the free-slot condition, the read's own progress) and its cost.
• The server suite passes (457).

Measured (RTX PRO 5000 48 GB, the 0.1.40 engine, only server.py differs)

"parallel": 8 with the conversation cache on. Public text (the repository's own code and docs). B's first turn (~20K tokens) is answered; a fresh ~30K-token prompt A starts its read; 1 s later B's second turn arrives (about 24 new tokens). Median of 3 rounds:

	0.1.40	with this change
B's next turn, time to first token	6.65 s	3.54 s
A (fresh), time to first token	6.99 s	7.80 s
reads that gave way	0 of 3	2 of 3

The engine takes the give-way at the chunk boundary after the one where the server asks (8,192-token chunks here), and not when less than a chunk is left (the third round).

In our own 8-user replay (95% of prompt tokens reused, so 276 of 325 reads needed under 2K new tokens), the new rule gave way 1 time; the time to first token was 1.54 s median / 4.43 s p90 with the old rule and 1.35 s / 4.63 s with the new one (the p90 within noise). The long reads there were mostly two chunks (12-16K new tokens): after the first boundary one chunk or less is left, and the engine does not give way then. So the change matters where a read of three or more chunks meets a cached follow-up.

🤖 Generated with Claude Code

[blange48]

Tested on a layer split. Rig: 4× RTX 5080 16 GB (PCIe Gen3), i9-10920X, Qwen3.8-Flash-Next IQ3_S, 262K, int8 KV, --layer-split 12,24,36 --trim-stage-weights, MTP spec 4, --batch 8 --batch-groups 4, conversation cache 8 GiB, on v0.1.40.1 plus our serve patches; HTTP server, T=0.7, 300 tokens.

Scenario (giveway-test): an agent conversation (~12K-token system prompt + one turn, its state held), then a fresh 45K-token prompt starts reading, and 2 s later the agent sends a short follow-up. Time to first token of the follow-up, 2 rounds each:

	follow-up alone	follow-up during the 45K read	the 45K read ends at
without #1288	0.15 / 0.14 s	10.41 / 10.19 s	13.0 / 12.7 s
with #1288	0.12 / 0.12 s	9.98 / 10.05 s	12.5 / 12.5 s

So on this setup the read did not give way: the follow-up still waited for (almost) the whole read. It may be that --prefill auto picks a chunk here that covers most of the 45K, leaving no boundary to yield at, or that the batch path does not take the yield. I'm happy to rerun with a fixed --prefill chunk or extra logging if you tell me what to look for. No regressions otherwise: batch vs solo 8/8 and 3/3 identical, throughput unchanged, and it's in our production.

[blange48]

Follow-up, now on 0.1.40.2 with this PR on top, to rule out the chunk size: same scenario (agent conversation held, a fresh 45K-token read starts, the agent's short follow-up arrives 2 s later). Same rig (4× RTX 5080 split, --batch 8 --batch-groups 4), 2 rounds each:

--prefill	follow-up's first token during the read	45K read alone
auto	10.33 / 10.24 s	11.20 / 11.04 s
16384	11.63 / 11.66 s	12.55 / 12.37 s
8192	10.43 / 10.16 s	11.15 / 11.07 s

The follow-up alone answers in 0.15 s. Even with 8K chunks, so several chunk boundaries inside the read, it waits for the whole read. My reading is that the give-way decision isn't reached on the --batch admission path at all, rather than the chunks being too large. If you'd like, I can run with extra logging around the yield check on this setup.

[Niko1221]

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.